Skip to main content
The Google Cloud Pub/Sub Group Kafka Connector library provides Google Cloud Platform (GCP) first-party connectors for Pub/Sub products with Kafka Connect. You can use the library to transmit data from Apache Kafka to Cloud Pub/Sub or Pub/Sub Lite and vice versa.

​
Prerequisites

You must have a GCP project in order to use Cloud Pub/Sub or Pub/Sub Lite. Follow these setup steps for Pub/Sub before doing the quickstart. Follow these setup steps for Pub/Sub Lite before doing the quickstart. For general information on how to authenticate with GCP when using the Google Cloud Pub/Sub Group Kafka Connector library, please visit Provide credentials for Application Default Credentials.

​
Quick Start

  1. Setup the kcctl client: doc
  2. Create a Pub/Sub topic in your GCP project.
  3. Create a JSON file like the following:
{
    "name": "pubsub-source",
    "config": {
        "connector.class": "com.google.pubsublite.kafka.source.PubSubLiteSourceConnector",
        "pubsublite.project": "${GCP_PROJECT_ID}",
        "pubsublite.topic": "${PUBSUB_TOPIC_NAME}",
        "pubsublite.subscription": "${PUBSUB_SUBSCRIPTION_NAME}",
        "pubsublite.location": "${PUBSUB_LOCATION}",
        "gcp.credentials.json": "${GCP_CREDENTIALS_JSON}",
        "kafka.topic": "${KAFKA_TOPIC_NAME}"
    }
}
  1. Run the following command to create the connector:
kcctl create -f <filename>.json

​
Configuration

The Google Pub/Sub Lite source connector is configured using the following properties:
ConfigValue RangeDefaultDescription
pubsublite.subscriptionStringREQUIRED (No default)The Pub/Sub Lite subscription ID, e.g. “baz” for the subscription “/projects/bar/locations/europe-south7-q/subscriptions/baz”.
pubsublite.projectStringREQUIRED (No default)The project containing the Pub/Sub Lite subscription, e.g. “bar” from above.
pubsublite.locationStringREQUIRED (No default)The location of the Pub/Sub Lite subscription, e.g. “europe-south7-q” from above.
kafka.topicStringREQUIRED (No default)The Kafka topic which will receive messages from Pub/Sub Lite.
pubsublite.partition_flow_control.messagesLongLong.MAX_VALUEThe maximum number of outstanding messages per Pub/Sub Lite partition.
pubsublite.partition_flow_control.bytesLong20,000,000The maximum number of outstanding bytes per Pub/Sub Lite partition.
gcp.credentials.file.pathStringOptionalThe filepath, which stores GCP credentials. If not defined, the environment variable GOOGLE_APPLICATION_CREDENTIALS is used. If specified, use the explicitly handed credentials. Consider using the externalized secrets feature in Kafka Connect for passing the value.
gcp.credentials.jsonStringOptionalGCP credentials JSON blob. If specified, use the explicitly handed credentials. Consider using the externalized secrets feature in Kafka Connect for passing the value.
The full properties are also available from the Official Google Pub/Sub Lite Kafka Source Connector documentation.