The Kinesis sink connector pulls data from Pulsar and persists data into Amazon Kinesis.
AccessKey
(Please record the value of AccessKey
and its SecretKey
).builtin
connector. If you want to create a non-builtin
connector,
you need to replace --sink-type kinesis
with --archive /path/to/pulsar-io-kinesis.nar
. You can find the button to download the nar
package at the beginning of the document.
--sink-config
is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.
If you want to configure more parameters, see Configuration Properties for reference.
pulsar-admin
are similar to those of pulsarctl
. You can find an example for StreamNative Cloud Doc.Data Viewer
to view the data. Name | Type | Required | Sensitive | Default | Description |
---|---|---|---|---|---|
awsKinesisStreamName | String | true | false | ” ” (empty string) | The Kinesis stream name. |
awsRegion | String | true | false | ” ” (empty string) | The AWS Kinesis region. Example: us-west-1, us-west-2. |
awsCredentialPluginName | String | false | false | ” ” (empty string) | The fully-qualified class name of implementation of AwsCredentialProviderPlugin. Please refer to [Configure AwsCredentialProviderPlugin](###Configure AwsCredentialProviderPlugin) |
awsCredentialPluginParam | String | false | true | ” ” (empty string) | The JSON parameter to initialize awsCredentialsProviderPlugin . Please refer to [Configure AwsCredentialProviderPlugin](###Configure AwsCredentialProviderPlugin) |
awsEndpoint | String | false | false | ” ” (empty string) | A custom Kinesis endpoint. For more information, see AWS documentation. |
retainOrdering | Boolean | false | false | false | Whether Pulsar connectors retain the ordering when moving messages from Pulsar to Kinesis. |
messageFormat | MessageFormat | false | false | ONLY_RAW_PAYLOAD | Message format in which Kinesis sink converts Pulsar messages and publishes them to Kinesis streams. Available options include: ONLY_RAW_PAYLOAD : Kinesis sink directly publishes Pulsar message payload as a message into the configured Kinesis stream. FULL_MESSAGE_IN_JSON : Kinesis sink creates a JSON payload with Pulsar message payload, properties, and encryptionCtx, and publishes JSON payload into the configured Kinesis stream.FULL_MESSAGE_IN_FB : Kinesis sink creates a flatbuffers serialized payload with Pulsar message payload, properties, and encryptionCtx, and publishes flatbuffers payload into the configured Kinesis stream. FULL_MESSAGE_IN_JSON_EXPAND_VALUE : Kinesis sink sends a JSON structure containing the record topic name, key, payload, properties, and event time. The record schema is used to convert the value to JSON. |
jsonIncludeNonNulls | Boolean | false | false | true | Only the properties with non-null values are included when the message format is FULL_MESSAGE_IN_JSON_EXPAND_VALUE . |
jsonFlatten | Boolean | false | false | false | When it is set to true and the message format is FULL_MESSAGE_IN_JSON_EXPAND_VALUE , the output JSON is flattened. |
retryInitialDelayInMillis | Long | false | false | 100 | The initial delay (in milliseconds) between retries. |
retryMaxDelayInMillis | Long | false | false | 60000 | The maximum delay(in milliseconds) between retries. |
awsCredentialPluginName
.
awsCredentialPluginName
empty to get the connector authenticated by passing accessKey
and secretKey
in awsCredentialPluginParam
.awsCredentialPluginName
to org.apache.pulsar.io.aws.AwsDefaultProviderChainPlugin
to use the default AWS provider chain. With this option, you don’t need to configure awsCredentialPluginParam
. For more information, see AWS documentation.
awsCredentialPluginName
to org.apache.pulsar.io.aws.STSAssumeRoleProviderPlugin
to use the default AWS provider chain, and you need to configure roleArn
and roleSessionNmae
in awsCredentialPluginParam
. For more information, see AWS documentation