Skip to main content

​
Prerequisites

Before deploying a connector to StreamNative Cloud, make sure the following prerequisites have been met:

​
Create a built-in connector

Before creating a connector, it’s highly recommended to do the following:
  1. Check connector availability to ensure the version number of the connector you want to create is supported on StreamNative Cloud.
  2. Go to StreamNative Hub and find the connector-specific docs of your version for configuration reference.
The following example shows how to create a data generator source connector named test on Streamnative Cloud using different tools. The builtin:// is followed by the name of the built-in connector, such as builtin://data-generator.
  • snctl
  • pulsarctl
  • pulsar-admin
  • Terraform
  • Rest API
To create a data generator source connector named test, run the following command.
snctl pulsar admin sources create --archive builtin://data-generator --destination-topic-name public/default/dg-test --source-config '{"sleepBetweenMessages": 60}' --name test --use-service-account
You should see the following output:
Created test successfully
If you want to verify whether the data generator source connector has been created successfully, run the following command:
snctl pulsar admin sources list
You should see the following output:
+---------------------+
| PULSAR SOURCES NAME |
+---------------------+
| test                |
+---------------------+
If you want to create a sink connector, use the snctl pulsar admin sinks create command.
For all the common configurations of built-in connectors, see Configuration reference.

​
Pass sensitive configs to connector

Some connectors require sensitive information, such as passwords, token, to be passed to the connector. And you may not want to expose these sensitive information in the connector configuration. To solve this problem, you can use the following methods to pass sensitive information to the connector:
  • Create a secret For example, the AWS lambda sink connector requires the AWS access key and secret key to be passed to the connector. You can create a secret in the console UI and pass the secret name to the connector configuration. screenshot of creating secret for AWS lambda sink
The location should be the same as the region of your Pulsar cluster.The awsAccessKey and awsSecretKey is the field name, and the lambda-sink-secret can be any unique name you want to give to the secret.
Only “sensitive” fields are able to load from secrets. You can get the list of sensitive fields from the connector configuration reference. E.g. AWS lambda sink configurations
  • Pass secrets to the connector configuration The following example shows how to create an AWS lambda sink connector named test on Streamnative Cloud using different tools. The builtin:// is followed by the name of the built-in connector, such as builtin://data-generator.
  • snctl
  • pulsarctl
  • pulsar-admin
  • Terraform
  • Rest API
To create an AWS lambda sink connector named test, run the following command.
snctl pulsar admin sinks create --archive builtin://aws-lambda --inputs public/default/lambda-sink-test --sink-config '{"awsRegion": "us-west-2","lambdaFunctionName": "test-hello","payloadFormat": "V2"}' --secrets '{"awsAccessKey":{"path":"lambda-sink-secret","key":"awsAccessKey"},"awsSecretKey":{"path":"lambda-sink-secret","key":"awsSecretKey"}}' --name test --use-service-account
You should see the following output:
Created test successfully
The awsAccessKey and awsSecretKey is the field name, and the lambda-sink-secret is the secret name you created in UI.
If you want to verify whether the AWS lambda sink connector has been created successfully, run the following command:
snctl pulsar admin sinks list
You should see the following output:
+---------------------+
| PULSAR SINKS NAME |
+---------------------+
| test                |
+---------------------+
For all the common configurations of built-in connectors, see Configuration reference.

​
Create a custom connector

Before creating a connector, it’s highly recommended to do the following:
  1. Check connector availability to ensure the version number of the connector you want to create is supported on StreamNative Cloud.
  2. Go to StreamNative Hub and find the connector-specific docs of your version for configuration reference.
To create a custom Pulsar Connector, you need to upload the connector jar/nar file to the StreamNative Cloud Package service first. Below are the steps:

​
Upload your connector file to Pulsar

  • snctl
  • Pulsarctl
  • Pulsar-admin
Upload packages
snctl pulsar admin packages upload sink://public/default/custom-connect@v1 \
--path /tmp/your-connector.jar \
--description "custom connector" \
--properties fileName=your-connector.jar
You should see the following output:
The package 'sink://public/default/custom-connect@v1' uploaded from path '/tmp/your-connector.jar' successfully
You can also upload your package to source://${tenant}/${namespace}/${name}@{$version}, currently Pulsar Package Service supports below protocols:
  • source://
  • sink://
  • function://
To create a custom connector, just replace the archive argument to the package URL(like sink://public/default/custom-connect@v1) you uploaded.