The Kinesis source connector pulls data from Amazon Kinesis and persists data into Pulsar
AccessKey
(Please record the value of AccessKey
and its SecretKey
).builtin
connector. If you want to create a non-builtin
connector,
you need to replace --source-type kinesis
with --archive /path/to/pulsar-io-kinesis.nar
. You can find the button to download the nar
package at the beginning of the document.
--source-config
is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.
If you want to configure more parameters, see Configuration Properties for reference.
pulsar-admin
are similar to those of pulsarctl
. You can find an example for StreamNative Cloud Doc.Name | Type | Required | Sensitive | Default | Description |
---|---|---|---|---|---|
awsKinesisStreamName | String | true | false | ” ” (empty string) | The Kinesis stream name. |
awsRegion | String | false | false | ” ” (empty string) | The AWS region. Example us-west-1, us-west-2. |
awsCredentialPluginName | String | false | false | ” ” (empty string) | The fully-qualified class name of implementation of AwsCredentialProviderPlugin. For more information, see [Configure AwsCredentialProviderPlugin](###Configure AwsCredentialProviderPlugin). |
awsCredentialPluginParam | String | false | true | ” ” (empty string) | The JSON parameter to initialize awsCredentialsProviderPlugin . For more information, see [Configure AwsCredentialProviderPlugin](###Configure AwsCredentialProviderPlugin). |
awsEndpoint | String | false | false | ” ” (empty string) | The Kinesis end-point URL, which can be found at here. |
dynamoEndpoint | String | false | false | ” ” (empty string) | The Dynamo end-point URL, which can be found at here. |
cloudwatchEndpoint | String | false | false | ” ” (empty string) | The Cloudwatch end-point URL. For more information, seeAmazon documentation. |
applicationName | String | false | false | Pulsar IO connector | The name of the Amazon Kinesis application, which will be used as the table name for DynamoDB. |
initialPositionInStream | InitialPositionInStream | false | false | LATEST | The position where the connector starts from. Below are the available options: AT_TIMESTAMP : start from the record at or after the specified timestamp.LATEST : start after the most recent data record.TRIM_HORIZON : start from the oldest available data record. |
startAtTime | Date | false | false | ” ” (empty string) | If set to AT_TIMESTAMP , it specifies the time point to start consumption. |
checkpointInterval | Long | false | false | 60000 | The frequency of the Kinesis stream checkpoint in milliseconds. |
backoffTime | Long | false | false | 3000 | The amount of time to delay between requests when the connector encounters a throttling exception from AWS Kinesis in milliseconds. |
numRetries | int | false | false | 3 | The number of re-attempts when the connector encounters an exception while trying to set a checkpoint. |
receiveQueueSize | int | false | false | 1000 | The maximum number of AWS records that can be buffered inside the connector. Once the receiveQueueSize is reached, the connector does not consume any messages from Kinesis until some messages in the queue are successfully consumed. |
useEnhancedFanOut | boolean | false | false | true | If set to true, it uses Kinesis enhanced fan-out. If set to false, it uses polling. |
awsCredentialPluginName
.
awsCredentialPluginName
empty to get the connector authenticated by passing accessKey
and secretKey
in awsCredentialPluginParam
.
awsCredentialPluginName
to org.apache.pulsar.io.aws.AwsDefaultProviderChainPlugin
to use the default AWS provider chain. With this option, you don’t need to configure awsCredentialPluginParam
. For more information, see AWS documentation.
awsCredentialPluginName
to org.apache.pulsar.io.aws.STSAssumeRoleProviderPlugin
to use the default AWS provider chain, and you need to configure roleArn
and roleSessionNmae
in awsCredentialPluginParam
. For more information, see AWS documentation