Kafka Protocol and Features

Supported Kafka Requests

The current implementation of the Apache Kafka protocol in StreamNative Cloud supports all the functionalities in Classic Engine and limited functionalities in the Ursa Engine. Specifically, the following Apache Kafka requests are currently supported:

API Key	Message	Classic Engine	Ursa Engine
0	Produce	Y	Y
1	Fetch	Y	Y
2	ListOffsets	Y	Y
3	Metadata	Y	Y
8	OffsetCommit	Y	Y
9	OffsetFetch	Y	Y
10	FindCoordinator	Y	Y
11	JoinGroup	Y	Y
12	Heartbeat	Y	Y
13	LeaveGroup	Y	Y
14	SyncGroup	Y	Y
15	DescribeGroups	Y	Y
16	ListGroups	Y	Y
18	ApiVersions	Y	Y
19	CreateTopics	Y	Y
20	DeleteTopics	Y	Y
21	DeleteRecords	Y	Y
22	InitProducerId	Y	Y (*)
24	AddPartitionsToTxn	Y	N
25	AddOffsetsToTxn	Y	N
26	EndTxn	Y	N
28	TxnOffsetCommit	Y	Y
32	AlterConfigs	Y	Y
32	DescribeConfigs	Y	Y
37	CreatePartitions	Y	Y
44	IncrementalAlterConfigs	Y	Y
47	OffsetDelete	Y	Y
51	DeleteGroups	Y	Y
60	DescribeCluster	Y	Y
61	DescribeProducers	Y	N
65	DescribeTransactions	Y	N
66	ListTransactions	Y	N

Notes

InitProducerId in Ursa Engine is partially supported when the transactional.id is NOT specified.

We're actively working on adding support for more Kafka features and more Kafka requests to both Classic Engine and Ursa Engine. Please contact us for specific feature requests or if you notice any discrepancies.

Unsupported Kafka Requests

Because the Kafka protocol implementation in StreamNative Cloud is based on Pulsar for Classic Engine and Ursa for Ursa Engine, the following Kafka requests are irrelevant and not supports. For example, requests like:

These requests are irrelevant to StreamNative Cloud since data storage and replication are handled differently - either in BookKeeper or in the underlying object storage. Therefore, these requests are unlikely to be supported.

API Key	Request Name
4	LeaderAndIsr
5	StopReplica
6	UpdateMetadata
7	ControlledShutdown
23	OffsetForLeaderEpoch
34	AlterReplicaLogDirs
35	DescribeLogDirs
43	ElectLeaders
45	AlterPartitionReassignments
46	ListPartitionReassignments
55	DescribeQuorum
56	AlterPartition
57	UpdateFeatures
58	Envelope
64	UnregisterBroker
67	AllocateProducerIds
80	AddRaftVoter
81	RemoveRaftVoter

These security and ACL-related requests are not supported because StreamNative Cloud provides more comprehensive security mechanisms, including OAuth2 and Role-Based Access Control (RBAC). For information about using Kafka ACLs in StreamNative Cloud, see Manage Kafka ACLs to learn how to map Kafka ACLs to Pulsar ACLs.

API Key	Request Name
29	DescribeAcls
30	CreateAcls
31	DeleteAcls
38	CreateDelegationToken
39	RenewDelegationToken
40	ExpireDelegationToken
41	DescribeDelegationToken
50	DescribeUserScramCredentials
51	AlterUserScramCredentials

Kafka's quota feature is not supported. Please use Pulsar's quota management instead.

API Key	Request Name
48	DescribeClientQuotas
49	AlterClientQuotas

API from latest KIPs

The API requests introduced from the latest KIPs are not yet supported. We are actively evaluating the feasibility of supporting them. If you have any specific feature requests, please contact us.

API Key	Request Name	KIP
68	ConsumerGroupHeartbeat	KIP-848: The Next Generation of the Consumer Rebalance Protocol
69	ConsumerGroupDescribe	KIP-848: The Next Generation of the Consumer Rebalance Protocol
71	GetTelemetrySubscriptions	KIP-714: Client metrics and observability
72	PushTelemetry	KIP-714: Client metrics and observability
74	ListClientMetricsResources	KIP-1000: List Client Metrics Configuration Resources
75	DescribeTopicPartitions	KIP-966: Eligible Leader Replicas

Supported topic configs

StreamNative Cloud supports the following topic configurations:

cleanup.policy: This configuration can be set to either compact or compact,delete. The policy controls topic compaction and is only effective when compact is included in the setting. Due to StreamNative Cloud's storage architecture, which preserves both raw and compacted segments rather than replacing individual log segments with compacted ones, we currently only support the values compact and compact,delete.

Supported Schema Registry APIs

StreamNative's Kafka Schema Registry is compatible with Confluent Schema Registry.

Following are the supported APIs:

Schemas

GET /schemas/ids/{int: id}
GET /schemas/ids/{int: id}/subjects
GET /schemas/ids/{int: id}/schema
GET /schemas/ids/{int: id}/versions
GET /schemas/types

Subjects

GET /subjects
GET /subjects/(string: subject)/versions
GET /subjects/(string: subject)/versions/(versionId: version)
GET /subjects/(string: subject)/versions/(versionId: version)/schema
GET /subjects/(string: subject)/versions/(string: versionId)/referencedby
POST /subjects/(string: subject)
POST /subjects/(string: subject)/versions?normalize=(boolean: normalize)
DELETE /subjects/(string: subject)?permanent=(boolean: permanent)
DELETE /subjects/(string: subject)/versions/(string: versionId)

Mode

GET /mode

Compatibility

GET /compatibility/subjects/${subject}/versions/latest

Config

GET /config
GET /config/${subject}
PUT /config/${subject}