Pulsar Schema Types

A Pulsar schema is defined by a data structure called SchemaInfo. It’s stored and enforced per topic—you can’t define one at the namespace or tenant level. Here’s a SchemaInfo for a string schema:

{
    "name": "test-string-schema",
    "type": "STRING",
    "schema": "",
    "properties": {}
}

Field	Description
`name`	The schema name.
`type`	The schema type, which determines how the data is serialized and deserialized.
`schema`	The schema data: a sequence of 8-bit unsigned bytes whose meaning depends on the schema type.
`properties`	A user-defined string-to-string map. Applications can use it to carry their own logic.

Schema types

Pulsar schema types fall into three categories: primitive, complex, and auto.

Primitive types

Primitive type	Description	Java	Python	Go	C++	C#
`BOOLEAN`	A binary value.	boolean	bool	bool	bool	bool
`INT8`	An 8-bit signed integer.	int	int	int8	int8_t	byte
`INT16`	A 16-bit signed integer.	int	int	int16	int16_t	short
`INT32`	A 32-bit signed integer.	int	int	int32	int32_t	int
`INT64`	A 64-bit signed integer.	int	int	int64	int64_t	long
`FLOAT`	A single-precision (32-bit) IEEE 754 floating-point number.	float	float	float32	float	float
`DOUBLE`	A double-precision (64-bit) IEEE 754 floating-point number.	double	double	float64	double	double
`BYTES`	A sequence of 8-bit unsigned bytes.	byte[], ByteBuffer, ByteBuf	bytes	[]byte	void *	byte[], ReadOnlySequence
`STRING`	A Unicode character sequence.	string	str	string	std::string	string
`TIMESTAMP` (`DATE`, `TIME`)	A logical type representing an instant in time with millisecond precision, stored as milliseconds since `January 1, 1970, 00:00:00 GMT` in an `INT64`.	`java.sql.Timestamp` (`java.sql.Time`, `java.util.Date`)	N/A	N/A	N/A	DateTime, TimeSpan
`INSTANT`	A single instantaneous point on the timeline, with nanosecond precision.	`java.time.Instant`	N/A	N/A	N/A	N/A
`LOCAL_DATE`	An immutable date-time object representing a date, usually viewed as year-month-day.	`java.time.LocalDate`	N/A	N/A	N/A	N/A
`LOCAL_TIME`	An immutable date-time object representing a time to nanosecond precision, usually viewed as hour-minute-second.	`java.time.LocalTime`	N/A	N/A	N/A	N/A
`LOCAL_DATE_TIME`	An immutable date-time object representing a date and a time.	`java.time.LocalDateTime`	N/A	N/A	N/A	N/A

Pulsar stores no schema data in SchemaInfo for primitive types. Some primitive schema implementations use properties to hold implementation-specific settings—a string schema, for example, can store the character encoding it uses to serialize and deserialize strings.

Complex types

Complex type	Description
`KeyValue`	A key/value pair.
`Struct`	Structured data. Covers `AvroBaseStructSchema`, `ProtobufNativeSchema`, and `NativeAvroBytesSchema`.

KeyValue schema

A KeyValue schema lets an application define a schema for the key and a schema for the value. Pulsar stores both SchemaInfo records together. There are two ways to encode a single key/value pair in a message:

INLINE—the key and value are encoded together in the message payload.
SEPARATED—the key is stored as the message key and the value as the message payload.

Struct schema

Type	Description
`AvroBaseStructSchema`	Uses the Avro specification to declare the schema definition. Covers `AvroSchema`, `JsonSchema`, and `ProtobufSchema`. One set of tools manages the schema definitions while different serialization methods handle the data.
`ProtobufNativeSchema`	Based on the Protobuf native descriptor. Uses native protobuf-v3 to serialize and deserialize data, and supports `AUTO_CONSUME`.
`NativeAvroBytesSchema`	Wraps a native `org.apache.avro.Schema`. The resulting schema instance accepts a serialized Avro payload without validating it against the wrapped schema.

NativeAvroBytesSchema exists for migration. When you ingest events from an external system such as Kafka or Cassandra, the data is often already serialized as Avro and has already been validated against a schema—including compatibility checks—by the system that produced it. In that case the Pulsar producer doesn’t need to repeat the validation. It passes each message through with its schema.

To use Kafka Avro, JSON Schema, or Protobuf schemas from a Pulsar client—with schemas resolved from the Kafka Schema Registry rather than Pulsar’s—see Use External Avro Schema with Pulsar clients.

There are three ways to construct a struct schema.

Static
Generic
SchemaDefinition

Predefine the struct as a POJO in Java, a struct in Go, or a class generated by Avro or Protobuf tooling. Pulsar reads the schema definition from it using an Avro library, and that definition becomes the schema data inside SchemaInfo.

Define the class for the messages you send:

// With Lombok
@Builder
@AllArgsConstructor
@NoArgsConstructor
public static class User {
    public String name;
    public int age;
}

// Without Lombok, add the constructors yourself:
// public static class User {
//     String name;
//     int age;
//     public User() { }
//     public User(String name, int age) { this.name = name; this.age = age; }
// }

Create a producer with the struct schema and send a message:

Producer<User> producer = client.newProducer(Schema.AVRO(User.class)).create();
producer.newMessage().value(new User("pulsar-user", 1)).send();

Create a consumer with the struct schema and receive it:

Consumer<User> consumer = client.newConsumer(Schema.AVRO(User.class)).subscribe();
User user = consumer.receive().getValue();

When your application has no predefined struct, define the schema with GenericSchemaBuilder, build records with GenericRecordBuilder, and consume into GenericRecord.

Build a schema with RecordSchemaBuilder:

RecordSchemaBuilder recordSchemaBuilder = SchemaBuilder.record("schemaName");
recordSchemaBuilder.field("intField").type(SchemaType.INT32);
SchemaInfo schemaInfo = recordSchemaBuilder.build(SchemaType.AVRO);

Consumer<GenericRecord> consumer = client.newConsumer(Schema.generic(schemaInfo))
     .topic(topicName)
     .subscriptionName(subscriptionName)
     .subscribe();
Producer<GenericRecord> producer = client.newProducer(Schema.generic(schemaInfo))
     .topic(topicName)
     .create();

Build struct records with RecordBuilder:

GenericSchemaImpl schema = GenericAvroSchema.of(schemaInfo);

// Send a message
GenericRecord record = schema.newRecordBuilder().set("intField", 32).build();
producer.newMessage().value(record).send();

// Receive a message
Message<GenericRecord> msg = consumer.receive();
Assert.assertEquals(msg.getValue().getField("intField"), 32);

Define a SchemaDefinition and generate a struct schema from it.

Define the class for the messages you send:

public static class User {
    public String name;
    public int age;

    public User(String name, int age) {
        this.name = name;
        this.age = age;
    }

    public User() {}
}

Create a producer with a SchemaDefinition and send a message:

SchemaDefinition<User> schemaDefinition = SchemaDefinition.<User>builder().withPojo(User.class).build();
Producer<User> producer = client.newProducer(Schema.AVRO(schemaDefinition)).create();
producer.newMessage().value(new User("pulsar-user", 1)).send();

Create a consumer with the same SchemaDefinition and receive it:

SchemaDefinition<User> schemaDefinition = SchemaDefinition.<User>builder().withPojo(User.class).build();
Consumer<User> consumer = client.newConsumer(Schema.AVRO(schemaDefinition)).subscribe();
User user = consumer.receive().getValue();

Auto schemas

When you can’t know a topic’s schema type in advance, use an auto schema to produce or consume generic records.

AUTO_PRODUCE sends data to a topic that already has a schema, and validates that the outbound bytes are compatible with it.
AUTO_CONSUME reads from a topic that has a schema and deserializes each message into a language-specific GenericRecord, using the SchemaInfo it retrieves from the broker.

What’s next

Compatibility

Compatibility strategies, versioning, and safe evolution.

Use with clients

Construct each schema type in your application.

Parts of this page are adapted from the Apache Pulsar documentation, licensed under the Apache License 2.0.

Get Started

Clusters

Data Streams

Process

Connect

Lakehouse

Governance

Pulsar Clients

MQTT Clients

Tools

Pulsar Changelogs

Schema types

Primitive types

Complex types

KeyValue schema

Struct schema

Auto schemas

What’s next

Compatibility

Use with clients

​Schema types

​Primitive types

​Complex types

​KeyValue schema

​Struct schema

​Auto schemas

​What’s next

Compatibility

Use with clients

Schema types

Primitive types

Complex types

KeyValue schema

Struct schema

Auto schemas

What’s next