​
Transformations

StreamNative cloud supports Kafka Connect transformations. You can use the following transformations in your Kafka Connect configurations:

  • InsertField - Add a field using either static data or record metadata
  • ReplaceField - Filter or rename fields
  • MaskField - Replace field with valid null value for the type (0, empty string, etc) or custom replacement (non-empty string or numeric value only)
  • ValueToKey - Replace the record key with a new key formed from a subset of fields in the record value
  • HoistField - Wrap the entire event as a single field inside a Struct or a Map
  • ExtractField - Extract a specific field from Struct and Map and include only this field in results
  • SetSchemaMetadata - modify the schema name or version
  • TimestampRouter - Modify the topic of a record based on original topic and timestamp. Useful when using a sink that needs to write to different tables or indexes based on timestamps
  • RegexRouter - modify the topic of a record based on original topic, replacement string and a regular expression
  • Flatten: Flatten a nested data structure, generating names for each field by concatenating the field names at each level with a configurable delimiter character.
  • Cast: Cast fields or the entire key or value to a specific type, e.g. to force an integer field to a smaller width.
  • TimestampConverter: Convert timestamps between different formats such as Unix epoch, strings, and Connect Date/Timestamp types.
  • Filter - Removes messages from all further processing. This is used with a predicate to selectively filter certain messages.
  • InsertHeader - Add a header using static data
  • HeadersFrom - Copy or move fields in the key or value to the record headers
  • DropHeaders - Remove headers by name

Some built-in connectors also has specified transformations:

​
YugabyteDB CDC Source

​
Iceberg Sink

​
Predicates

StreamNative cloud supports Kafka Connect predicates. You can use the following predicates in your Kafka Connect configurations:

  • TopicNameMatches: matches records in a topic with a name matching a particular Java regular expression.
  • HasHeaderKey: matches records which have a header with the given key.
  • RecordIsTombstone: matches tombstone records, that is records with a null value.

​
What’s next?