Source connector configurations
This table lists all the common configurations for a source connector.| Field | Description |
|---|---|
-a, --archive | The path to the NAR archive for the source. It supports the file-URL-path (file://) which assumes that the NAR file already exists on the worker host from which the worker can download the package. For a built-in connector, it should be set to builtin//<connector_name>. |
--classname | The source’s class name if the archive is set to a file-URL-path (file://). |
--cpu | The CPU (in cores) that needs to be allocated per source instance (applicable only to Docker runtime). |
--deserialization-classname | The SerDe classname for the source. |
--destination-topic-name | The Pulsar topic to which data is sent. |
--disk | The disk (in bytes) that needs to be allocated per source instance (applicable only to Docker runtime). |
--name | The source’s name. |
--namespace | The source’s namespace. |
--parallelism | The source’s parallelism factor, that is, the number of source instances to run. |
--processing-guarantees | The processing guarantees (also named as delivery semantics) applied to the source. A source connector receives messages from the external system and writes messages to a Pulsar topic. The --processing-guarantees ensures the processing guarantees for writing messages to the Pulsar topic. The available values are ATLEAST_ONCE, ATMOST_ONCE, EFFECTIVELY_ONCE. |
--ram | The RAM (in bytes) that needs to be allocated per source instance (applicable only to the process and Docker runtimes). |
-st, --schema-type | The schema type. Either a built-in schema (for example, AVRO and JSON) or a custom schema class name to be used to encode messages emitted from source. |
--source-config | The key/values configurations of the source. For example: ‘{"sleepBetweenMessages": 60}’; For configuration details, refer to the documentation of each connector |
--source-config-file | The path to a YAML config file that specifies the source’s configuration. |
--sn-service-account | The StreamNative Cloud service account that the source uses as its runtime identity. This snctl flag writes snServiceAccount to custom-runtime-options. You can use --use-sn-service-account to select a runtime service account interactively. You must have the cloud.serviceaccounts.describe permission on the specified service account (see account-admin). Only Pulsar 4.0.x clusters running version 4.0.10.6 or later, or Pulsar 4.2.x clusters running version 4.2.1.4 or later, support this config. |
-t, --source-type | The source’s connector provider. |
--tenant | The source’s tenant. |
--producer-config | The custom producer configuration (as a JSON string). |
Sink connector configurations
This table lists all the common configurations for a sink connector.| Field | Description |
|---|---|
-a, --archive | The path to the archive file for the sink. It supports the file-URL-path (file://) which assumes that the NAR file already exists on the worker host from which the worker can download the package. For a built-in connector, it should be set to builtin//<connector_name>. |
--classname | The sink’s class name if the archive is set to a file-URL-path (file://). |
--cpu | The CPU (in cores) that needs to be allocated per sink instance (applicable only to Docker runtime). |
--custom-schema-inputs | The map of input topics to schema types or class names (as a JSON string). |
--custom-serde-inputs | The map of input topics to SerDe class names (as a JSON string). |
--disk | The disk (in bytes) that needs to be allocated per sink instance (applicable only to Docker runtime). |
-i, --inputs | The sink’s input topic or topics (multiple topics can be specified as a comma-separated list). |
--name | The sink’s name. |
--namespace | The sink’s namespace. |
--parallelism | The sink’s parallelism factor, that is, the number of sink instances to run. |
--processing-guarantees | The processing guarantees (also known as delivery semantics) applied to the sink. The --processing-guarantees implementation in Pulsar also relies on sink implementation. The available values are ATLEAST_ONCE, ATMOST_ONCE, EFFECTIVELY_ONCE. |
--ram | The RAM (in bytes) that needs to be allocated per sink instance (applicable only to the process and Docker runtimes). |
--retain-ordering | Sink consumes messages in order. |
--sink-config | The key/values configurations of the sink. For example: ‘{"sleepBetweenMessages": 60}’; For configuration details, refer to the documentation of each connector |
--sink-config-file | The path to a YAML config file specifying the sink’s configuration. |
--sn-service-account | The StreamNative Cloud service account that the sink uses as its runtime identity. This snctl flag writes snServiceAccount to custom-runtime-options. You can use --use-sn-service-account to select a runtime service account interactively. |
-t, --sink-type | The sink’s connector provider. The sink-type parameter of the currently built-in connectors is determined by the setting of the name parameter. You can use the pulsar-admin sinks available-sinks command to get all built-in sink connectors. |
--subs-name | Pulsar source subscription name if you want to specify a subscription name for the input-topic consumer. |
--tenant | The sink’s tenant. |
--timeout-ms | The message timeout in milliseconds. |
--topics-pattern | The topic pattern to consume from a list of topics under a namespace that matches the pattern. --input and --topics-Pattern are mutually exclusive. Add SerDe class name for a pattern in --customSerdeInputs. |
StreamNative Cloud custom runtime options
To facilitate submitting Pulsar functions based on your requirements, Function on Cloud service provides some custom options viacustom-runtime-options.
When update functions/sinks/sources with custom runtime options, the original custom runtime options will be replaced by the new ones, so make sure all the wanted fields are passed in the custom runtime options when you do the update.
| Name | Type | Default | Description |
|---|---|---|---|
clusterName | String | N/A | The Pulsar cluster of a Pulsar function, source, or sink. |
inputTypeClassName | String | [B | The map of input topics to Java class names. |
outputTypeClassName | String | [B | The map of output topics to Java class names. |
maxReplicas | Integer | 0 | The maximum number of Pulsar instances that you want to run for this Pulsar Function. When the value of the maxReplicas parameter is greater than the value of replicas, it indicates that the Functions controller automatically scales the Pulsar Functions based on the CPU usage. By default, maxReplicas is set to 0, which indicates that auto-scaling is disabled. |
env | Map<String, String> | N/A | The environment variables being attached to a Pod that is created by the Function Mesh Operator for the cluster. |
imagePullSecrets | List<String> | N/A | A list of references to secrets in the same namespace for pulling any of the images used by a Pod. |
logLevel | String | info | The log levels for Pulsar functions. For details, see log levels. |
logRotationPolicy | String | N/A | The log rotation policies for Pulsar functions. You can set the log rotation policies based on the time or the log file size. For details, see log rotation policies. |
runnerImageTag | String | N/A | The tag of the runner image that is used to submit a function, source, or sink. |
hpaSpec (Preview) | HPASpec | N/A | The Kubernetes HorizontalPodAutoscaler settings. For details, see Kubernetes documentation. This feature may not ready for your cluster enviroment, please file a ticket if you want this feature enabled. |
logFormat | String | text | The log format that defines how the content of a log file should be interpreted. Available options are json and text. The log format configurations are only available for the Java and Python runtimes. |
logTopic | String | N/A | Used for sinks/sources since they don’t have a log topic argument like functions |
logTopicAgent | String | runtime | The log agent that defines how StreamNative cloud redirects your functions / connectors log into a Pulsar topic. Available options are runtime and sidecar. When use sidecar all logs (including instance logs) will be sent to the log topic by filebeat (better performance than runtime), else will use the Pulsar Functions runtime’s log-topic implementation instead. |
genericKind | String | N/A | Used for functions written in languages other than Java and Python, available values are executable, nodejs, wasm |
snServiceAccount | String | N/A | The StreamNative Cloud service account that the function, source, or sink uses as its runtime identity. You must have the cloud.serviceaccounts.describe permission on the specified service account (see account-admin). Only Pulsar 4.0.x clusters running version 4.0.10.6 or later, or Pulsar 4.2.x clusters running version 4.2.1.4 or later, support this config. |
terminationGracePeriodSeconds | Long | N/A | The amount of time that kubernetes will give for a pod before terminating it. |
pauseRollout | Boolean | N/A | Whether to pause the rollout of functions/connectors, when set to true, running functions&connectors will not restart during the upgrade of backend function-mesh operator. |
enableStateStore | Boolean | N/A | Whether to enable stateful backend for this function, default to false, only available for Java function |
custom-runtime-options field. For example:
{
"inputTypeClassName": "java.lang.String",
"logTopic": "log-topic-name"
}
custom-runtime-options field as follows:
snctl pulsar admin sinks create --custom-runtime-options '{"inputTypeClassName":"java.lang.String","logTopic":"log-topic-name"}' --as-service-account $SERVICE_ACCOUNT ...
pulsarctl sinks create --custom-runtime-options '{"inputTypeClassName":"java.lang.String","logTopic":"log-topic-name"}' ...