> ## Documentation Index
> Fetch the complete documentation index at: https://docs.streamnative.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Pulsar Schema Evolution and Compatibility

> How the Pulsar Schema Registry versions schemas, which compatibility strategies it supports, and the order in which to upgrade producers and consumers.

Schemas change as the business changes. The Pulsar Schema Registry versions every schema on a topic
and checks each new version against the old ones, so that a schema change doesn't silently break the
applications already reading that topic.

<Note title="Note">
  Schema evolution applies to Avro, JSON, Protobuf, and ProtobufNative schemas. Other schema types use
  a default checker that disables evolution entirely.
</Note>

Two mechanisms govern evolution: the [compatibility check](#schema-compatibility-check), which
decides whether a new schema is allowed at all, and [AutoUpdate](#schema-autoupdate), which decides
whether a passing schema is registered automatically.

## Schema versioning

Every `SchemaInfo` stored with a topic carries a version, and versions are what make schema change
manageable within a topic.

Messages produced with a `SchemaInfo` are tagged with its schema version. When a client consumes one,
it uses that version to retrieve the matching `SchemaInfo` and deserialize the data with the right
schema. Once a producer is assigned a version, every subsequent message it sends carries that
version.

Suppose you create a producer like this:

```java theme={null}
PulsarClient client = PulsarClient.builder()
        .serviceUrl("pulsar://localhost:6650")
        .build();

Producer<SensorReading> producer = client.newProducer(JSONSchema.of(SensorReading.class))
        .topic("sensor-data")
        .sendTimeout(3, TimeUnit.SECONDS)
        .create();
```

Three things can happen when that producer connects:

| Scenario                                                              | Result                                                                                                                                                          |
| --------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| No schema exists for the topic.                                       | The producer is created with the given schema, the schema is sent to the broker and stored, and any consumer using the same schema can read from `sensor-data`. |
| A schema exists and the producer connects with the same one.          | The schema is sent to the broker, the broker finds it compatible, and—on discovering it's already stored—reuses it to tag produced messages.                    |
| A schema exists and the producer connects with a new, compatible one. | The schema is sent to the broker, which finds it compatible and stores it as the current version under a new version number.                                    |

## Schema compatibility check

A compatibility check exists to guarantee that consumers already running can still process the
messages a new schema introduces.

When a broker receives a `SchemaInfo` from a producer, it identifies the schema type and runs that
type's compatibility checker against the topic's existing schema, applying the configured strategy.
Avro, JSON, and Protobuf each have their own checker; every other schema type falls back to a default
checker that disables evolution.

### Compatibility strategies

Suppose a topic holds three schema versions—V1 is the oldest, V3 the newest.

| Strategy              | Definition                                                                                                                                              | Changes allowed                    | Checked against       |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | --------------------- |
| `ALWAYS_COMPATIBLE`   | Disables the compatibility check.                                                                                                                       | All changes                        | All previous versions |
| `ALWAYS_INCOMPATIBLE` | Disables schema evolution—every change is rejected.                                                                                                     | None                               | N/A                   |
| `BACKWARD`            | Consumers on V3 can read data written by producers on the **last** version, V2.                                                                         | Add optional fields; delete fields | Latest version        |
| `BACKWARD_TRANSITIVE` | Consumers on V3 can read data written by producers on **all** previous versions, V2 and V1.                                                             | Add optional fields; delete fields | All previous versions |
| `FORWARD`             | Consumers on the **last** version V2 can read data written by producers on the new schema V3, though they may not use everything the new schema offers. | Add fields; delete optional fields | Latest version        |
| `FORWARD_TRANSITIVE`  | Consumers on **any** previous version, V2 or V1, can read data written by producers on V3.                                                              | Add fields; delete optional fields | All previous versions |
| `FULL`                | Both backward and forward compatible between V2 and V3—consumers on either can read data written under the other.                                       | Modify optional fields             | Latest version        |
| `FULL_TRANSITIVE`     | Both backward and forward compatible across V3, V2, and V1.                                                                                             | Modify optional fields             | All previous versions |

<Tip>
  The default strategy depends on the schema type: **`FULL` for Avro and JSON**, and
  **`ALWAYS_INCOMPATIBLE` for everything else**. To change it, see
  [Set the compatibility strategy](/cloud/governance/sr/pulsar/manage-schemas#set-the-compatibility-strategy).
</Tip>

<Warning title="Pulsar and Kafka defaults differ">
  A Pulsar cluster with the Kafka protocol enabled runs both registries, and they do not share a
  default. Pulsar defaults to `FULL` for Avro and JSON; the
  [Kafka Schema Registry](/cloud/governance/sr/kafka-schema-registry) defaults to `BACKWARD` for every
  format. If you operate both on the same cluster, set the strategy explicitly on each rather than
  relying on either default.
</Warning>

For deeper background on what constitutes a compatible change in each format, see the
[Avro schema resolution rules](https://avro.apache.org/docs/1.10.2/spec.html#Schema+Resolution) and
the [Protobuf field rules](https://developers.google.com/protocol-buffers/docs/proto#optional).

## Schema validation enforcement

Schema validation enforcement lets brokers reject producers and consumers that connect without a
schema.

It's **disabled for producers by default** (`isSchemaValidationEnforced` is `false`), which means:

* A producer with no schema can write anything to a topic that has one, so a misconfigured
  application can put garbage on a schematized topic.
* Clients with no schema support can produce to a topic that has a schema.

Turn it on when a topic's schema is a contract you need enforced. See
[Manage schema validation](/cloud/governance/sr/pulsar/manage-schemas#manage-schema-validation).

## Schema AutoUpdate

AutoUpdate is **enabled by default**. When a schema passes the compatibility check, the producer
registers it against the topic automatically.

### On the producer side

* If the **topic has no schema**—the data is raw bytes—Pulsar registers the schema automatically.
* If the **topic has a schema but the producer carries none**, it produces raw bytes. Whether it's
  allowed depends on [schema validation enforcement](#schema-validation-enforcement): with
  enforcement off the producer connects and writes; with it on the producer is rejected.
* If the **topic has a schema and the producer carries one**, the broker follows the
  [producer-side workflow](/cloud/governance/sr/pulsar/overview#producer-side).

### On the consumer side

* A consumer connecting to a topic **without a schema** consumes raw bytes and connects with no
  compatibility check.
* A consumer connecting to a topic **with a schema** follows the
  [consumer-side workflow](/cloud/governance/sr/pulsar/overview#consumer-side).

## Order of upgrading clients

Which side you upgrade first is determined by the compatibility strategy—get this backwards and you
break consumers in production even though every schema passed its check.

| Strategy                          | Upgrade order       | Why                                                                                                                                                                                  |
| --------------------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `ALWAYS_COMPATIBLE`               | Any order           | The compatibility check is disabled.                                                                                                                                                 |
| `ALWAYS_INCOMPATIBLE`             | N/A                 | Schema evolution is disabled.                                                                                                                                                        |
| `BACKWARD`, `BACKWARD_TRANSITIVE` | **Consumers first** | Nothing guarantees that a consumer on the old schema can read data written under the new one. Upgrade every consumer, then start producing new data.                                 |
| `FORWARD`, `FORWARD_TRANSITIVE`   | **Producers first** | Nothing guarantees that a consumer on the new schema can read data written under the old one. Upgrade every producer, let the old data age out of reach, then upgrade the consumers. |
| `FULL`, `FULL_TRANSITIVE`         | Any order           | Consumers on either schema can read data written under the other.                                                                                                                    |

## What's next

<CardGroup cols={2}>
  <Card title="Manage schemas" icon="sliders" href="/cloud/governance/sr/pulsar/manage-schemas">
    Set the compatibility strategy and manage schema versions.
  </Card>

  <Card title="Schema types" icon="shapes" href="/cloud/governance/sr/pulsar/schema-types">
    Which types support evolution, and how they're defined.
  </Card>
</CardGroup>

<Note title="Attribution">
  Parts of this page are adapted from the [Apache Pulsar documentation](https://pulsar.apache.org/docs/schema-understand/),
  licensed under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).
</Note>