Kafka Connect SMTs(Single Message Transformations)

Transformations

StreamNative cloud supports Kafka Connect transformations. You can use the following transformations in your Kafka Connect configurations:

• InsertField - Add a field using either static data or record metadata
• ReplaceField - Filter or rename fields
• MaskField - Replace field with valid null value for the type (0, empty string, etc) or custom replacement (non-empty string or numeric value only)
• ValueToKey - Replace the record key with a new key formed from a subset of fields in the record value
• HoistField - Wrap the entire event as a single field inside a Struct or a Map
• ExtractField - Extract a specific field from Struct and Map and include only this field in results
• SetSchemaMetadata - modify the schema name or version
• TimestampRouter - Modify the topic of a record based on original topic and timestamp. Useful when using a sink that needs to write to different tables or indexes based on timestamps
• RegexRouter - modify the topic of a record based on original topic, replacement string and a regular expression
• Flatten: Flatten a nested data structure, generating names for each field by concatenating the field names at each level with a configurable delimiter character.
• Cast: Cast fields or the entire key or value to a specific type, e.g. to force an integer field to a smaller width.
• TimestampConverter: Convert timestamps between different formats such as Unix epoch, strings, and Connect Date/Timestamp types.
• Filter - Removes messages from all further processing. This is used with a predicate to selectively filter certain messages.
• InsertHeader - Add a header using static data
• HeadersFrom - Copy or move fields in the key or value to the record headers
• DropHeaders - Remove headers by name

Some built-in connectors also has specified transformations:

YugabyteDB CDC Source

• YBExtractNewRecordState SMT
• ExtractTopic
• PGCompatible SMT

Iceberg Sink

• CopyValue
• DmsTransform
• DebeziumTransform
• JsonToMapTransform
• KafkaMetadataTransform
• MongoDebeziumTransform

Predicates

StreamNative cloud supports Kafka Connect predicates. You can use the following predicates in your Kafka Connect configurations:

• TopicNameMatches: matches records in a topic with a name matching a particular Java regular expression.
• HasHeaderKey: matches records which have a header with the given key.
• RecordIsTombstone: matches tombstone records, that is records with a null value.

What’s next?

• Discover kafka Connect Ecosystem on StreamNative Hub.