Monitor and Troubleshoot Connectors
StreamNative Cloud allows you to monitor connectors status, logs, and exceptions that are thrown when a source or sink connector fails to be created, updated, or cannot work.
View connector status
This section describes how to view connector status using pulsarctl, pulsar-admin, Rest API, and console.
Note
If you want to monitor connectors using pulsarctl or pulsar-admin, Rest API, make sure you have set up your client tool. For more information, see set up client tools.
The following example introduces how to view the status of the data generator source connector named test.
To check the status of the source connector test, run the following command:
pulsarctl sources status --tenant public --namespace default --name test
{
"numInstances": 1,
"numRunning": 1,
"instances": [
{
"instanceId": 0,
"status": {
"running": true,
"error": "",
"numRestarts": 0,
"numReceivedFromSource": 2622,
"numSystemExceptions": 0,
"latestSystemExceptions": [],
"numSourceExceptions": 0,
"latestSourceExceptions": [],
"numWritten": 2622,
"lastReceivedTime": 1691532145625,
"workerId": "test"
}
]
}
View connector logs
View the connector logs using snctl
This section describes how to view connector logs using snctl.
Note
This example assumes you have installed snctl and initialized snctl configurations.
You can run the snctl logs command to view logs for a specific connector. This table outlines the configuration options that are used for viewing connector logs. For details about all supported fields, you can use the snctl logs -h command to list more information.
| Option | Descriptions |
|---|---|
-c or --cluster | The name of your Pulsar cluster where the connector is created. |
-p or --component | The type of component to monitor. Available options are function, sink, and source. |
-f or --follow | Continuously list the connector log history. |
-h or --help | Show usage information about the snctl logs command. |
-i or --instance | The name of your Pulsar instance where the connector is created. |
--name | The name of your connector. |
-o or --organization | The name of your organization where the connector is created. |
--previous | Print the logs that are generated before the configured timestamp. |
--pulsar-tenant | The name of your Pulsar tenant where the connector is created. |
--pulsar-namespace | The name of your Pulsar namespace where the connector is created. |
--since | List logs more recent than the specific time. Available units are second, minute, and hour, such as 24h. |
-s or --size | Specify how many lines of recent logs to display. |
--timestamp | Include timestamps on each line in the log output. |
The following command example shows how to view up to 60 lines of the data1 sink connector’s logs within the last 5 hours.
snctl logs --since 5h --organization sndev --instance aws --cluster aws --name data1 --pulsar-tenant public --pulsar-namespace default -p sink -f -s 60
You should see the following output:
2023-03-23T02:18:43,075+0000 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [public/default/my-topic][11] Subscribed to topic on aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650 -- consumer: 0
2023-03-23T02:18:43,075+0000 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [public/default/my-topic][11] Subscribed to topic on aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650 -- consumer: 0
2023-03-23T02:18:43,028+0000 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [public/default/my-topic][11] Subscribing to topic on cnx [id: 0xd4f91768, L:/172.16.136.145:53364 - R:aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650], consumerId 0
2023-03-23T02:18:43,028+0000 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerImpl - [public/default/my-topic][11] Subscribing to topic on cnx [id: 0xd4f91768, L:/172.16.136.145:53364 - R:aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650], consumerId 0
2023-03-23T02:18:43,026+0000 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xd4f91768, L:/172.16.136.145:53364 - R:aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650]] Connected to server
2023-03-23T02:18:43,026+0000 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xd4f91768, L:/172.16.136.145:53364 - R:aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650]] Connected to server
2023-03-23T02:18:42,951+0000 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Pulsar client config: {"serviceUrl":"pulsar://aws-broker.sndev.svc.cluster.local:6650","authPluginClassName":"org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2","authParams":"*****","authParamMap":null,"operationTimeoutMs":30000,"lookupTimeoutMs":30000,"statsIntervalSeconds":60,"numIoThreads":1,"numListenerThreads":1,"connectionsPerBroker":1,"useTcpNoDelay":true,"useTls":false,"tlsTrustCertsFilePath":null,"tlsAllowInsecureConnection":true,"tlsHostnameVerificationEnable":false,"concurrentLookupRequest":5000,"maxLookupRequest":50000,"maxLookupRedirects":20,"maxNumberOfRejectedRequestPerConnection":50,"keepAliveIntervalSeconds":30,"connectionTimeoutMs":10000,"requestTimeoutMs":60000,"initialBackoffIntervalNanos":100000000,"maxBackoffIntervalNanos":60000000000,"enableBusyWait":false,"listenerName":null,"useKeyStoreTls":false,"sslProvider":null,"tlsTrustStoreType":"JKS","tlsTrustStorePath":null,"tlsTrustStorePassword":null,"tlsCiphers":[],"tlsProtocols":[],"memoryLimitBytes":0,"proxyServiceUrl":null,"proxyProtocol":null,"enableTransaction":false,"socks5ProxyAddress":null,"socks5ProxyUsername":null,"socks5ProxyPassword":null}
2023-03-23T02:18:42,951+0000 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Pulsar client config: {"serviceUrl":"pulsar://aws-broker.sndev.svc.cluster.local:6650","authPluginClassName":"org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2","authParams":"*****","authParamMap":null,"operationTimeoutMs":30000,"lookupTimeoutMs":30000,"statsIntervalSeconds":60,"numIoThreads":1,"numListenerThreads":1,"connectionsPerBroker":1,"useTcpNoDelay":true,"useTls":false,"tlsTrustCertsFilePath":null,"tlsAllowInsecureConnection":true,"tlsHostnameVerificationEnable":false,"concurrentLookupRequest":5000,"maxLookupRequest":50000,"maxLookupRedirects":20,"maxNumberOfRejectedRequestPerConnection":50,"keepAliveIntervalSeconds":30,"connectionTimeoutMs":10000,"requestTimeoutMs":60000,"initialBackoffIntervalNanos":100000000,"maxBackoffIntervalNanos":60000000000,"enableBusyWait":false,"listenerName":null,"useKeyStoreTls":false,"sslProvider":null,"tlsTrustStoreType":"JKS","tlsTrustStorePath":null,"tlsTrustStorePassword":null,"tlsCiphers":[],"tlsProtocols":[],"memoryLimitBytes":0,"proxyServiceUrl":null,"proxyProtocol":null,"enableTransaction":false,"socks5ProxyAddress":null,"socks5ProxyUsername":null,"socks5ProxyPassword":null}
2023-03-23T02:18:42,939+0000 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Starting Pulsar consumer status recorder with config: {"topicNames":["public/default/my-topic"],"topicsPattern":null,"subscriptionName":"11","subscriptionType":"Shared","subscriptionMode":"Durable","receiverQueueSize":1000,"acknowledgementsGroupTimeMicros":100000,"negativeAckRedeliveryDelayMicros":60000000,"maxTotalReceiverQueueSizeAcrossPartitions":50000,"consumerName":null,"ackTimeoutMillis":0,"tickDurationMillis":1000,"priorityLevel":0,"maxPendingChunkedMessage":10,"autoAckOldestChunkedMessageOnQueueFull":false,"expireTimeOfIncompleteChunkedMessageMillis":60000,"cryptoFailureAction":"FAIL","properties":{"application":"pulsar-sink","id":"public/default/data1-cc47863d","instance_hostname":"data1-cc47863d-sink-0","instance_id":"0"},"readCompacted":false,"subscriptionInitialPosition":"Latest","patternAutoDiscoveryPeriod":60,"regexSubscriptionMode":"PersistentOnly","deadLetterPolicy":null,"retryEnable":false,"autoUpdatePartitions":true,"autoUpdatePartitionsIntervalSeconds":60,"replicateSubscriptionState":false,"resetIncludeHead":false,"keySharedPolicy":null,"batchIndexAckEnabled":false,"ackReceiptEnabled":false,"poolMessages":false,"maxPendingChuckedMessage":10}
2023-03-23T02:18:42,939+0000 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Starting Pulsar consumer status recorder with config: {"topicNames":["public/default/my-topic"],"topicsPattern":null,"subscriptionName":"11","subscriptionType":"Shared","subscriptionMode":"Durable","receiverQueueSize":1000,"acknowledgementsGroupTimeMicros":100000,"negativeAckRedeliveryDelayMicros":60000000,"maxTotalReceiverQueueSizeAcrossPartitions":50000,"consumerName":null,"ackTimeoutMillis":0,"tickDurationMillis":1000,"priorityLevel":0,"maxPendingChunkedMessage":10,"autoAckOldestChunkedMessageOnQueueFull":false,"expireTimeOfIncompleteChunkedMessageMillis":60000,"cryptoFailureAction":"FAIL","properties":{"application":"pulsar-sink","id":"public/default/data1-cc47863d","instance_hostname":"data1-cc47863d-sink-0","instance_id":"0"},"readCompacted":false,"subscriptionInitialPosition":"Latest","patternAutoDiscoveryPeriod":60,"regexSubscriptionMode":"PersistentOnly","deadLetterPolicy":null,"retryEnable":false,"autoUpdatePartitions":true,"autoUpdatePartitionsIntervalSeconds":60,"replicateSubscriptionState":false,"resetIncludeHead":false,"keySharedPolicy":null,"batchIndexAckEnabled":false,"ackReceiptEnabled":false,"poolMessages":false,"maxPendingChuckedMessage":10}
View the connector logs from log topics
StreamNative Cloud provide an alternative implementation of the Pulsar Functions log topic(see Pulsar Functions log topic) feature called sidecar mode, which supports using such feature in Connectors.
To enable the sidecar mode, user should provide the following configurations when deploying the connector:
pulsarctl sinks/sources create --custom-runtime-options '{"logTopic":"LOG-TOPIC", "logTopicAgent":"sidecar"}' ...
For more details about the custom-runtime-options, see Custom runtime options.
Once the connector is deployed with the sidecar mode, you can configure consumers to consume messages from the log topic.