The official Apache Iceberg Kafka Connect Sink connector.
kc-test-iceberg-table
as an example.
my_s3_namespace
as an example.
eu-north-1
, account-id
, kc-test-iceberg-table
and my_s3_namespace
with your real region
, account-id
, table-bucket
and database
respectively.:
Access Key ID
and Secret Access Key
later.
region
, account-id
, table_bucket
, access_key_id
and secret_access_key
with your real values:
events
topic, you can use the following command to produce some test data:
events
table in the my_s3_namespace
database, and you can run a query like below to see the data:
Parameter | Description |
---|---|
topics | Comma-separated list of the Kafka topics you want to replicate. (You can define either the topics or the topics.regex setting, but not both.) |
topics.regex | Java regular expression of topics to replicate. (You can define either the topics or the topics.regex setting, but not both.) |
iceberg.control.topic | The name of the control topic. It cannot be used by other Iceberg connectors. |
iceberg.catalog.type | The type of Iceberg catalog. Allowed options are: REST , HIVE , HADOOP . |
iceberg.tables | Comma-separated list of Iceberg table names, which are specified using the format {namespace}.{table} . |
Parameter | Description |
---|---|
iceberg.control.commit.timeout-ms | Commit timeout interval in ms. The default is 30000 (30 sec). |
iceberg.tables.route-field | For multi-table fan-out, the name of the field used to route records to tables. Required when iceberg.tables.dynamic-enabled is set to true . |
iceberg.tables.cdc-field | Name of the field containing the CDC operation, I , U , or D , default is none |