Sources
Kafka connect mongodb source
The official MongoDB Kafka Connect Source connector.
The MongoDB Kafka source connector is a Kafka Connect connector that reads data from MongoDB and writes data to Kafka topics.
Prerequisites
- The
connection.uri
is in form ofmongodb+srv://username:password@cluster0.xxx.mongodb.net
- Valid credentials with the
read
role on the database. For more granular access control, you can specify a custom role that allowsfind
, andchangeStream
actions on the databases or collections.
Quick Start
-
Setup the kcctl client: doc
-
Create a MongoDB Cluster, you can create one in k8s cluster with below yaml file:
-
Initialize the local MongoDB cluster:
-
Create a JSON file like the following:
-
Run the following command to create the connector:
Configuration
The MongoDB Kafka source connector is configured using the following Required properties:
Parameter | Description |
---|---|
connection.uri | The connection URI for the MongoDB server. |
database | The MongoDb database from which the connector imports data into Redpanda topics. The connector monitors changes in this database. Leave the field empty to watch all databases. |
collection | The collection in the MongoDB database to watch. If not set, then all collections are watched. |
topic.prefix | The prefix for the Kafka topics that the connector creates. The connector appends a database name and collection name to this prefix to create the topic name. |
The full properties are also available from the offical MongoDB Kafka Source Connector documentation.