Prerequisites
Before deploying a connector to StreamNative Cloud, make sure the following prerequisites have been met:- A running external data system service.
- A running Pulsar Cluster on StreamNative Cloud and the required environment has been set up.
-
At least one of the required tools: snctl, pulsarctl,
pulsar-admin
, or the Terraform module. For a quickstart of setting uppulsarctl
andpulsar-admin
, see set up client tools.
Create a built-in connector
Before creating a connector, it’s highly recommended to do the following:
- Check connector availability to ensure the version number of the connector you want to create is supported on StreamNative Cloud.
- Go to StreamNative Hub and find the connector-specific docs of your version for configuration reference.
test
on Streamnative Cloud using different tools.
The builtin://
is followed by the name of the built-in connector, such as builtin://data-generator
.
To create a data generator source connector named You should see the following output:If you want to verify whether the data generator source connector has been created successfully, run the following command:You should see the following output:
test
, run the following command.If you want to create a sink connector, use the
snctl pulsar admin sinks create
command.Pass sensitive configs to connector
Some connectors require sensitive information, such as passwords, token, to be passed to the connector. And you may not want to expose these sensitive information in the connector configuration. To solve this problem, you can use the following methods to pass sensitive information to the connector:-
Create a secret
For example, the AWS lambda sink connector requires the AWS access key and secret key to be passed to the connector.
You can create a secret in the console UI and pass the secret name to the connector configuration.
The
location
should be the same as the region of your Pulsar cluster.The awsAccessKey
and awsSecretKey
is the field name, and the lambda-sink-secret
can be any unique name you want to give to the secret.Only “sensitive” fields are able to load from secrets. You can get the list of sensitive fields from the connector configuration reference.
E.g. AWS lambda sink configurations
-
Pass secrets to the connector configuration
The following example shows how to create an AWS lambda sink connector named
test
on Streamnative Cloud using different tools. Thebuiltin://
is followed by the name of the built-in connector, such asbuiltin://data-generator
.
To create an AWS lambda sink connector named You should see the following output:If you want to verify whether the AWS lambda sink connector has been created successfully, run the following command:You should see the following output:
test
, run the following command.The
awsAccessKey
and awsSecretKey
is the field name, and the lambda-sink-secret
is the secret name you created in UI.Create a custom connector
Before creating a connector, it’s highly recommended to do the following:
- Check connector availability to ensure the version number of the connector you want to create is supported on StreamNative Cloud.
- Go to StreamNative Hub and find the connector-specific docs of your version for configuration reference.
Upload your connector file to Pulsar
Upload packagesYou should see the following output:
You can also upload your package to
source://${tenant}/${namespace}/${name}@{$version}
, currently Pulsar Package Service supports below protocols:source://
sink://
function://
archive
argument to the package URL(like sink://public/default/custom-connect@v1
) you uploaded.