Skip to main content
The Google Cloud Pub/Sub Group Kafka Connector library provides Google Cloud Platform (GCP) first-party connectors for Pub/Sub products with Kafka Connect. You can use the library to transmit data from Apache Kafka to Cloud Pub/Sub or Pub/Sub Lite and vice versa.

​
Prerequisites

You must have a GCP project in order to use Cloud Pub/Sub or Pub/Sub Lite. Follow these setup steps for Pub/Sub before doing the quickstart. Follow these setup steps for Pub/Sub Lite before doing the quickstart. For general information on how to authenticate with GCP when using the Google Cloud Pub/Sub Group Kafka Connector library, please visit Provide credentials for Application Default Credentials.

​
Quick Start

  1. Setup the kcctl client: doc
  2. Create a Pub/Sub topic in your GCP project.
  3. Create a JSON file like the following:
{
    "name": "pubsub-sink",
    "config": {
        "connector.class": "com.google.pubsublite.kafka.sink.PubSubLiteSinkConnector",
        "pubsublite.project": "${GCP_PROJECT_ID}",
        "pubsublite.topic": "${PUBSUB_TOPIC_NAME}",
        "pubsublite.location": "${PUBSUB_LOCATION}",
        "gcp.credentials.json": "${GCP_CREDENTIALS_JSON}",
        "topics": "${KAFKA_TOPIC_NAME}"
    }
}
  1. Run the following command to create the connector:
kcctl create -f <filename>.json

​
Configuration

The Google Pub/Sub Lite sink connector is configured using the following properties:
ConfigValue RangeDefaultDescription
pubsublite.topicStringREQUIRED (No default)The Pub/Sub Lite topic ID, e.g. “foo” for topic “/projects/bar/locations/europe-south7-q/topics/foo”.
pubsublite.projectStringREQUIRED (No default)The project containing the Pub/Sub Lite topic, e.g. “bar” from above.
pubsublite.locationStringREQUIRED (No default)The location of the Pub/Sub Lite topic, e.g. “europe-south7-q” from above.
gcp.credentials.file.pathStringOptionalThe filepath, which stores GCP credentials. If not defined, the environment variable GOOGLE_APPLICATION_CREDENTIALS is used. If specified, use the explicitly handed credentials. Consider using the externalized secrets feature in Kafka Connect for passing the value.
gcp.credentials.jsonStringOptionalGCP credentials JSON blob. If specified, use the explicitly handed credentials. Consider using the externalized secrets feature in Kafka Connect for passing the value.
The full properties are also available from the Official Google Pub/Sub Lite Kafka Sink Connector documentation.