The AWS Simple Queue Service (SQS) source connector feeds data from Amazon AWS SQS and writes data to Pulsar topics.

Quick start

Prerequisites

The prerequisites for connecting an AWS SQS source connector to external systems include:

1. Create SQS in AWS.
2. Create the AWS User and create AccessKey(Please record AccessKey and SecretAccessKey).
3. Assign the following permissions to the AWS User:

• sqs:CreateQueue
• sqs:DeleteMessage
• sqs:ChangeMessageVisibility
• sqs:GetQueueUrl
• sqs:GetQueueAttributes
• sqs:ReceiveMessage

1. Create a connector

Depending on the environment, there are several ways to create an AWS SQS source connector:

• Create Connector on StreamNative Cloud.
• Create Connector with Function worker. Using this way requires you to download a NAR package to create a built-in or non-built-in connector. You can download the version you need by clicking the Download icon on the upper-right corner of this page.
• Create Connector with Function mesh. Using this way requires you to set the docker image. You can choose the version you want to launch from Docker Hub.

No matter how you create an AWS SQS source connector, the minimum configuration contains the following parameters.

configs:
  awsRegion: "Your AWS SQS region"
  queueName: "Your AWS SQS name"
  awsCredentialPluginParam: "{\"accessKey\":\"Your access key\",\"secretKey\":\"Your secret key\"}"

• The configuration structure varies depending on how you create the AWS SQS source connector. For example, some are JSON, some are YAML, and some are Kubernetes YAML. You need to adapt the configs to the corresponding format.

• If you want to configure more parameters, see Configuration Properties for reference.

2. Send messages to AWS SQS

You can use the following simple code to send messages to AWS SQS.

    public static void main(String[] args) {
        AmazonSQS client = AmazonSQSClientBuilder.standard()
                .withCredentials(new AWSStaticCredentialsProvider(
                        new BasicAWSCredentials("Your access key", "Your secret key")))
                .withRegion("Your AWS SQS region").build();
        String queueUrl = client.getQueueUrl(new GetQueueUrlRequest("Your SQS name")).getQueueUrl();
        client.sendMessage(queueUrl, "Hello World!");
        client.shutdown();
    }

3. Show data using Pulsar client

bin/pulsar-client \
--url "Your Pulsar serviceUrl" \
consume "The topic that you specified when you created the connector" -s "test-sub" -n 10 -p Earliest

----- got message -----
key:[null], properties:[], content:Hello World!

Configuration Properties

Before using the AWS SQS source connector, you need to configure it. This table outlines the properties and the Descriptions of an AWS SQS source connector.

Configure AwsCredentialProviderPlugin

AWS SQS source connector allows you to use three ways to connect to AWS SQS by configuring awsCredentialPluginName.

• Leave awsCredentialPluginName empty to get the connector authenticated by passing accessKey and secretKey in awsCredentialPluginParam.

  {"accessKey":"Your access key","secretKey":"Your secret key"}
  

• Set awsCredentialPluginName to org.apache.pulsar.io.aws.AwsDefaultProviderChainPlugin to use the default AWS provider chain. With this option, you don't need to configure awsCredentialPluginParam. For more information, see AWS documentation.

• Set awsCredentialPluginName to org.apache.pulsar.io.aws.STSAssumeRoleProviderPlugin to use the default AWS provider chain, and you need to configure roleArn and roleSessionNmae in awsCredentialPluginParam. For more information, see AWS documentation.

  {"roleArn": "arn...", "roleSessionName": "name"}