The SQS source connector is used to consume messages from Amazon SQS and publish them to Pulsar.
The AWS Simple Queue Service (SQS) source connector feeds data from Amazon AWS SQS and writes data to Pulsar topics.
The prerequisites for connecting an AWS SQS source connector to external systems include:
AccessKey
(Please record AccessKey
and SecretAccessKey
).The following command shows how to use pulsarctl to create a builtin
connector. If you want to create a non-builtin
connector,
you need to replace --source-type sqs
with --archive /path/to/pulsar-io-sqs.nar
. You can find the button to download the nar
package at the beginning of the document.
If you are a StreamNative Cloud user, you need set up your environment first.
The --source-config
is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.
If you want to configure more parameters, see Configuration Properties for reference.
You can also choose to use a variety of other tools to create a connector:
pulsar-admin
are similar to those of pulsarctl
. You can find an example for StreamNative Cloud Doc.You can use the following simple code to send messages to AWS SQS.
If your connector is created on StreamNative Cloud, you need to authenticate your clients. See Build applications using Pulsar clients for more information.
Before using the AWS SQS source connector, you need to configure it. This table outlines the properties and the Descriptions of an AWS SQS source connector.
Name | Type | Required | Sensitive | Default | Description |
---|---|---|---|---|---|
awsRegion | String | true | false | ” ” (empty string) | Supported AWS region. For example, us-west-1, us-west-2. |
queueName | String | true | false | ” ” (empty string) | The name of the SQS queue that messages should be read from or written to. |
awsCredentialPluginName | String | false | false | ” ” (empty string) | The fully-qualified class name of implementation of AwsCredentialProviderPlugin. For more information, see [Configure AwsCredentialProviderPlugin](###Configure AwsCredentialProviderPlugin). |
awsCredentialPluginParam | String | false | true | ” ” (empty string) | The JSON parameter to initialize awsCredentialsProviderPlugin . For more information, see [Configure AwsCredentialProviderPlugin](###Configure AwsCredentialProviderPlugin). |
awsEndpoint | String | false | false | ” ” (empty string) | AWS SQS end-point URL. You can find it at AWS SQS Service endpoints. |
batchSizeOfOnceReceive | int | false | false | 1 | The maximum number of messages that are pulled from SQS at one time. By default, it is set to 1. The value ranges from 1 to 10. |
numberOfConsumers | int | false | false | 1 | The expected numbers of consumers. You can scale consumers horizontally to achieve high throughput. By default, it is set to 1. The value ranges from 1 to 50. |
sourceQueueSize | int | false | false | 10000 | The size of the queue that holds the messages received from SQS. By default, it is set to 10000. The value cannot smaller than 1. |
The batchSizeOfOnceReceive
and numberOfConsumers
options are available for SQS source 2.8.4.3+, 2.9.4.1+, and 2.10.1.13+. For details about how to test AWS SQS source performance, see Performance Test on AWS SQS Source Connector.
AWS SQS source connector allows you to use three ways to connect to AWS SQS by configuring awsCredentialPluginName
.
Leave awsCredentialPluginName
empty to get the connector authenticated by passing accessKey
and secretKey
in awsCredentialPluginParam
.
Set awsCredentialPluginName
to org.apache.pulsar.io.aws.AwsDefaultProviderChainPlugin
to use the default AWS provider chain. With this option, you don’t need to configure awsCredentialPluginParam
. For more information, see AWS documentation.
Set awsCredentialPluginName
to org.apache.pulsar.io.aws.STSAssumeRoleProviderPlugin
to use the default AWS provider chain, and you need to configure roleArn
and roleSessionNmae
in awsCredentialPluginParam
. For more information, see AWS documentation.