Source connector configurations
This table lists all the common configurations for a source connector.| Field | Description |
|---|---|
-a, --archive | The path to the NAR archive for the source. It supports the file-URL-path (file://) which assumes that the NAR file already exists on the worker host from which the worker can download the package. For a built-in connector, it should be set to builtin//<connector_name>. |
--classname | The source’s class name if the archive is set to a file-URL-path (file://). |
--cpu | The CPU (in cores) that needs to be allocated per source instance (applicable only to Docker runtime). |
--deserialization-classname | The SerDe classname for the source. |
--destination-topic-name | The Pulsar topic to which data is sent. |
--disk | The disk (in bytes) that needs to be allocated per source instance (applicable only to Docker runtime). |
--name | The source’s name. |
--namespace | The source’s namespace. |
--parallelism | The source’s parallelism factor, that is, the number of source instances to run. |
--processing-guarantees | The processing guarantees (also named as delivery semantics) applied to the source. A source connector receives messages from the external system and writes messages to a Pulsar topic. The --processing-guarantees ensures the processing guarantees for writing messages to the Pulsar topic. The available values are ATLEAST_ONCE, ATMOST_ONCE, EFFECTIVELY_ONCE. |
--ram | The RAM (in bytes) that needs to be allocated per source instance (applicable only to the process and Docker runtimes). |
-st, --schema-type | The schema type. Either a built-in schema (for example, AVRO and JSON) or a custom schema class name to be used to encode messages emitted from source. |
--source-config | The key/values configurations of the source. For example: ‘{"sleepBetweenMessages": 60}’; For configuration details, refer to the documentation of each connector |
--source-config-file | The path to a YAML config file that specifies the source’s configuration. |
--sn-service-account | The StreamNative Cloud service account that the source uses as its runtime identity. This snctl flag writes snServiceAccount to custom-runtime-options. You can use --use-sn-service-account to select a runtime service account interactively. You must have the cloud.serviceaccounts.describe permission on the specified service account (see account-admin). Only Pulsar 4.0.x clusters running version 4.0.10.6 or later, or Pulsar 4.2.x clusters running version 4.2.1.4 or later, support this config. |
-t, --source-type | The source’s connector provider. |
--tenant | The source’s tenant. |
--producer-config | The custom producer configuration (as a JSON string). |
Sink connector configurations
This table lists all the common configurations for a sink connector.| Field | Description |
|---|---|
-a, --archive | The path to the archive file for the sink. It supports the file-URL-path (file://) which assumes that the NAR file already exists on the worker host from which the worker can download the package. For a built-in connector, it should be set to builtin//<connector_name>. |
--classname | The sink’s class name if the archive is set to a file-URL-path (file://). |
--cpu | The CPU (in cores) that needs to be allocated per sink instance (applicable only to Docker runtime). |
--custom-schema-inputs | The map of input topics to schema types or class names (as a JSON string). |
--custom-serde-inputs | The map of input topics to SerDe class names (as a JSON string). |
--disk | The disk (in bytes) that needs to be allocated per sink instance (applicable only to Docker runtime). |
-i, --inputs | The sink’s input topic or topics (multiple topics can be specified as a comma-separated list). |
--name | The sink’s name. |
--namespace | The sink’s namespace. |
--parallelism | The sink’s parallelism factor, that is, the number of sink instances to run. |
--processing-guarantees | The processing guarantees (also known as delivery semantics) applied to the sink. The --processing-guarantees implementation in Pulsar also relies on sink implementation. The available values are ATLEAST_ONCE, ATMOST_ONCE, EFFECTIVELY_ONCE. |
--ram | The RAM (in bytes) that needs to be allocated per sink instance (applicable only to the process and Docker runtimes). |
--retain-ordering | Sink consumes messages in order. |
--sink-config | The key/values configurations of the sink. For example: ‘{"sleepBetweenMessages": 60}’; For configuration details, refer to the documentation of each connector |
--sink-config-file | The path to a YAML config file specifying the sink’s configuration. |
--sn-service-account | The StreamNative Cloud service account that the sink uses as its runtime identity. This snctl flag writes snServiceAccount to custom-runtime-options. You can use --use-sn-service-account to select a runtime service account interactively. |
-t, --sink-type | The sink’s connector provider. The sink-type parameter of the currently built-in connectors is determined by the setting of the name parameter. You can use the pulsar-admin sinks available-sinks command to get all built-in sink connectors. |
--subs-name | Pulsar source subscription name if you want to specify a subscription name for the input-topic consumer. |
--tenant | The sink’s tenant. |
--timeout-ms | The message timeout in milliseconds. |
--topics-pattern | The topic pattern to consume from a list of topics under a namespace that matches the pattern. --input and --topics-Pattern are mutually exclusive. Add SerDe class name for a pattern in --customSerdeInputs. |