Kafka connect google pubsub lite source - StreamNative Documentation

The Google Cloud Pub/Sub Group Kafka Connector library provides Google Cloud Platform (GCP) first-party connectors for Pub/Sub products with Kafka Connect. You can use the library to transmit data from Apache Kafka to Cloud Pub/Sub or Pub/Sub Lite and vice versa.

Prerequisites

You must have a GCP project in order to use Cloud Pub/Sub or Pub/Sub Lite. Follow these setup steps for Pub/Sub before doing the quickstart. Follow these setup steps for Pub/Sub Lite before doing the quickstart. For general information on how to authenticate with GCP when using the Google Cloud Pub/Sub Group Kafka Connector library, please visit Provide credentials for Application Default Credentials.

Quick Start

Setup the kcctl client: doc
Create a Pub/Sub topic in your GCP project.
Create a JSON file like the following:

{
    "name": "pubsub-source",
    "config": {
        "connector.class": "com.google.pubsublite.kafka.source.PubSubLiteSourceConnector",
        "pubsublite.project": "${GCP_PROJECT_ID}",
        "pubsublite.topic": "${PUBSUB_TOPIC_NAME}",
        "pubsublite.subscription": "${PUBSUB_SUBSCRIPTION_NAME}",
        "pubsublite.location": "${PUBSUB_LOCATION}",
        "gcp.credentials.json": "${GCP_CREDENTIALS_JSON}",
        "kafka.topic": "${KAFKA_TOPIC_NAME}"
    }
}

Run the following command to create the connector:

kcctl apply -f <filename>.json

Configuration

The Google Pub/Sub Lite source connector is configured using the following properties:

Config	Value Range	Default	Description
pubsublite.subscription	String	REQUIRED (No default)	The Pub/Sub Lite subscription ID, e.g. “baz” for the subscription “/projects/bar/locations/europe-south7-q/subscriptions/baz”.
pubsublite.project	String	REQUIRED (No default)	The project containing the Pub/Sub Lite subscription, e.g. “bar” from above.
pubsublite.location	String	REQUIRED (No default)	The location of the Pub/Sub Lite subscription, e.g. “europe-south7-q” from above.
kafka.topic	String	REQUIRED (No default)	The Kafka topic which will receive messages from Pub/Sub Lite.
pubsublite.partition_flow_control.messages	Long	Long.MAX_VALUE	The maximum number of outstanding messages per Pub/Sub Lite partition.
pubsublite.partition_flow_control.bytes	Long	20,000,000	The maximum number of outstanding bytes per Pub/Sub Lite partition.
gcp.credentials.file.path	String	Optional	The filepath, which stores GCP credentials. If not defined, the environment variable GOOGLE_APPLICATION_CREDENTIALS is used. If specified, use the explicitly handed credentials. Consider using the externalized secrets feature in Kafka Connect for passing the value.
gcp.credentials.json	String	Optional	GCP credentials JSON blob. If specified, use the explicitly handed credentials. Consider using the externalized secrets feature in Kafka Connect for passing the value.

The full properties are also available from the Official Google Pub/Sub Lite Kafka Source Connector documentation.

Aiven JDBC Kafka Connect Source

⌘I

​Prerequisites

​Quick Start

​Configuration

Prerequisites

Quick Start

Configuration