Kafka Serializers and Deserializers

Your application doesn’t call the Schema Registry directly. A serializer does it on the producer side and a deserializer does it on the consumer side, both configured through ordinary Kafka client properties.

Choose a serializer

Format	Serializer	Deserializer
Avro	`io.confluent.kafka.serializers.KafkaAvroSerializer`	`KafkaAvroDeserializer`
JSON Schema	`io.confluent.kafka.serializers.json.KafkaJsonSchemaSerializer`	`KafkaJsonSchemaDeserializer`
Protobuf	`io.confluent.kafka.serializers.protobuf.KafkaProtobufSerializer`	`KafkaProtobufDeserializer`

Configure them like any other serializer, plus the registry URL and credentials:

props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class);
props.put(KafkaAvroSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl);
props.put(KafkaAvroSerializerConfig.BASIC_AUTH_CREDENTIALS_SOURCE, "USER_INFO");
props.put(KafkaAvroSerializerConfig.USER_INFO_CONFIG, String.format("%s:%s", "any-user", apiKey));

See Connect for the URL and authentication options.

Subject name strategies

The serializer derives the subject name from the record. Three strategies ship with the Confluent clients:

Strategy	Subject for topic `orders`	Use when
`TopicNameStrategy` (default)	`orders-value`, `orders-key`	One record type per topic.
`RecordNameStrategy`	The record’s fully qualified name, for example `com.acme.Order`	The same record type appears on several topics and should evolve as one unit.
`TopicRecordNameStrategy`	`orders-com.acme.Order`	Several record types share a topic but should evolve independently.

Set it per producer:

props.put(AbstractKafkaSchemaSerDeConfig.VALUE_SUBJECT_NAME_STRATEGY,
    io.confluent.kafka.serializers.subject.RecordNameStrategy.class.getName());

RecordNameStrategy and TopicRecordNameStrategy produce subjects containing dots, and StreamNative Cloud parses those into tenant, namespace, and topic. com.acme.Order becomes tenant com, namespace acme, topic Order. Confirm the tenant and namespace exist and that your credentials cover them. See Subject names and Pulsar coordinates.

If you also enable schema ID validation, set the matching strategy on the topic so the broker derives the same subject the producer did.

Multiple event types on one topic

To put several record types on a single topic, use RecordNameStrategy or TopicRecordNameStrategy so each type gets its own subject and evolves on its own schedule. With the default TopicNameStrategy all types share one subject, and the compatibility check compares unrelated schemas against each other. Protobuf handles this most naturally through a oneof wrapper message; in Avro you’d use a union at the top level.

Auto-registration

By default a producer registers its schema the first time it sends a record under a subject that doesn’t have it. That’s convenient in development and risky in production—any application can create a subject and set its first version. Turn it off, and register schemas deliberately instead:

props.put(AbstractKafkaSchemaSerDeConfig.AUTO_REGISTER_SCHEMAS, false);
props.put(AbstractKafkaSchemaSerDeConfig.USE_LATEST_VERSION, true);

With auto.register.schemas off, the serializer looks the schema up and fails if it isn’t registered. With use.latest.version on, it uses the subject’s latest registered version rather than the schema derived from your class. Restrict who can register schemas with the schema-writer and schema-reader roles.

Normalization

Two schemas that differ only in field ordering or whitespace are semantically identical but produce different schema IDs. Normalization canonicalizes a schema before it’s stored or looked up, so cosmetic differences stop creating new versions. Request it with ?normalize=true on registration and lookup, or set AbstractKafkaSchemaSerDeConfig.NORMALIZE_SCHEMAS on the client.

?normalize=true performs real normalization for Avro and Protobuf. For JSON Schema it’s a no-op—the schema is stored exactly as submitted, so reordering properties still yields a new schema ID.On the /compatibility/* endpoints, normalize is accepted and then ignored for every format.

The wire format

The serializer prefixes every record:

[ 1 byte magic = 0x00 ][ 4 bytes schema ID, big-endian ][ serialized payload ]

That’s why a consumer needs registry access to decode a record, and why reading the payload with a non-schema-aware tool produces garbage. See The wire format.

Client caching

Deserializers cache schemas by ID, so a consumer hits the registry once per distinct schema rather than once per record. Producers cache the subject-to-ID mapping the same way. Because of that cache, a schema registered moments ago may not be visible to a client that already cached a lookup miss. Restart the client, or configure a cache expiry, if you’re chasing an unexpected Schema not found.

Introduction

Get Started

Kafka Clusters

Process

Lakehouse

Governance

Migrate

Kafka Clients

Choose a serializer

Subject name strategies

Multiple event types on one topic

Auto-registration

Normalization

The wire format

Client caching

Next steps

Schema references

Evolution and compatibility

​Choose a serializer

​Subject name strategies

​Multiple event types on one topic

​Auto-registration

​Normalization

​The wire format

​Client caching

​Next steps

Schema references

Evolution and compatibility

Choose a serializer

Subject name strategies

Multiple event types on one topic

Auto-registration

Normalization

The wire format

Client caching

Next steps