Monitor and Troubleshoot Connectors

StreamNative Cloud allows you to monitor connectors status, logs, and exceptions that are thrown when a source or sink connector fails to be created, updated, or cannot work.

View connector status

This section describes how to view connector status using pulsarctl, pulsar-admin, and console.

Note

If you want to monitor connectors using pulsarctl or pulsar-admin, make sure you have set up your client tool. For more information, see set up client tools.

The following example introduces how to view the status of the data generator source connector named test.

To check the status of the source connector test, run the following command:

pulsarctl sources status --tenant public --namespace default --name test

{
  "numInstances": 1,
  "numRunning": 1,
  "instances": [
    {
      "instanceId": 0,
      "status": {
        "running": true,
        "error": "",
        "numRestarts": 0,
        "numReceivedFromSource": 2622,
        "numSystemExceptions": 0,
        "latestSystemExceptions": [],
        "numSourceExceptions": 0,
        "latestSourceExceptions": [],
        "numWritten": 2622,
        "lastReceivedTime": 1691532145625,
        "workerId": "test"
      }
  ]
}

View connector logs

This section describes how to view connector logs using snctl.

Note

This example assumes you have installed snctl and initialized snctl configurations.

You can run the snctl logs command to view logs for a specific connector. This table outlines the configuration options that are used for viewing connector logs. For details about all supported fields, you can use the snctl logs -h command to list more information.

Option	Descriptions
`-c` or `--cluster`	The name of your Pulsar cluster where the connector is created.
`-p` or `--component`	The type of component to monitor. Available options are `function`, `sink`, and `source`.
`-f` or `--follow`	Continuously list the connector log history.
`-h` or `--help`	Show usage information about the `snctl logs` command.
`-i` or `--instance`	The name of your Pulsar instance where the connector is created.
`--name`	The name of your connector.
`-o` or `--organization`	The name of your organization where the connector is created.
`--previous`	Print the logs that are generated before the configured timestamp.
`--pulsar-tenant`	The name of your Pulsar tenant where the connector is created.
`--pulsar-namespace`	The name of your Pulsar namespace where the connector is created.
`--since`	List logs more recent than the specific time. Available units are `second`, `minute`, and `hour`, such as `24h`.
`-s` or `--size`	Specify how many lines of recent logs to display.
`--timestamp`	Include timestamps on each line in the log output.

The following command example shows how to view up to 60 lines of the data1 sink connector’s logs within the last 5 hours.

snctl logs --since 5h --organization sndev --instance aws --cluster aws --name data1 --pulsar-tenant public --pulsar-namespace default -p sink -f -s 60

You should see the following output:

2023-03-23T02:18:43,075+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [public/default/my-topic][11] Subscribed to topic on aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650 -- consumer: 0
2023-03-23T02:18:43,075+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [public/default/my-topic][11] Subscribed to topic on aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650 -- consumer: 0
2023-03-23T02:18:43,028+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [public/default/my-topic][11] Subscribing to topic on cnx [id: 0xd4f91768, L:/172.16.136.145:53364 - R:aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650], consumerId 0
2023-03-23T02:18:43,028+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [public/default/my-topic][11] Subscribing to topic on cnx [id: 0xd4f91768, L:/172.16.136.145:53364 - R:aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650], consumerId 0
2023-03-23T02:18:43,026+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xd4f91768, L:/172.16.136.145:53364 - R:aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650]] Connected to server
2023-03-23T02:18:43,026+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xd4f91768, L:/172.16.136.145:53364 - R:aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650]] Connected to server
2023-03-23T02:18:42,951+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Pulsar client config: {"serviceUrl":"pulsar://aws-broker.sndev.svc.cluster.local:6650","authPluginClassName":"org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2","authParams":"*****","authParamMap":null,"operationTimeoutMs":30000,"lookupTimeoutMs":30000,"statsIntervalSeconds":60,"numIoThreads":1,"numListenerThreads":1,"connectionsPerBroker":1,"useTcpNoDelay":true,"useTls":false,"tlsTrustCertsFilePath":null,"tlsAllowInsecureConnection":true,"tlsHostnameVerificationEnable":false,"concurrentLookupRequest":5000,"maxLookupRequest":50000,"maxLookupRedirects":20,"maxNumberOfRejectedRequestPerConnection":50,"keepAliveIntervalSeconds":30,"connectionTimeoutMs":10000,"requestTimeoutMs":60000,"initialBackoffIntervalNanos":100000000,"maxBackoffIntervalNanos":60000000000,"enableBusyWait":false,"listenerName":null,"useKeyStoreTls":false,"sslProvider":null,"tlsTrustStoreType":"JKS","tlsTrustStorePath":null,"tlsTrustStorePassword":null,"tlsCiphers":[],"tlsProtocols":[],"memoryLimitBytes":0,"proxyServiceUrl":null,"proxyProtocol":null,"enableTransaction":false,"socks5ProxyAddress":null,"socks5ProxyUsername":null,"socks5ProxyPassword":null}
2023-03-23T02:18:42,951+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Pulsar client config: {"serviceUrl":"pulsar://aws-broker.sndev.svc.cluster.local:6650","authPluginClassName":"org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2","authParams":"*****","authParamMap":null,"operationTimeoutMs":30000,"lookupTimeoutMs":30000,"statsIntervalSeconds":60,"numIoThreads":1,"numListenerThreads":1,"connectionsPerBroker":1,"useTcpNoDelay":true,"useTls":false,"tlsTrustCertsFilePath":null,"tlsAllowInsecureConnection":true,"tlsHostnameVerificationEnable":false,"concurrentLookupRequest":5000,"maxLookupRequest":50000,"maxLookupRedirects":20,"maxNumberOfRejectedRequestPerConnection":50,"keepAliveIntervalSeconds":30,"connectionTimeoutMs":10000,"requestTimeoutMs":60000,"initialBackoffIntervalNanos":100000000,"maxBackoffIntervalNanos":60000000000,"enableBusyWait":false,"listenerName":null,"useKeyStoreTls":false,"sslProvider":null,"tlsTrustStoreType":"JKS","tlsTrustStorePath":null,"tlsTrustStorePassword":null,"tlsCiphers":[],"tlsProtocols":[],"memoryLimitBytes":0,"proxyServiceUrl":null,"proxyProtocol":null,"enableTransaction":false,"socks5ProxyAddress":null,"socks5ProxyUsername":null,"socks5ProxyPassword":null}
2023-03-23T02:18:42,939+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Starting Pulsar consumer status recorder with config: {"topicNames":["public/default/my-topic"],"topicsPattern":null,"subscriptionName":"11","subscriptionType":"Shared","subscriptionMode":"Durable","receiverQueueSize":1000,"acknowledgementsGroupTimeMicros":100000,"negativeAckRedeliveryDelayMicros":60000000,"maxTotalReceiverQueueSizeAcrossPartitions":50000,"consumerName":null,"ackTimeoutMillis":0,"tickDurationMillis":1000,"priorityLevel":0,"maxPendingChunkedMessage":10,"autoAckOldestChunkedMessageOnQueueFull":false,"expireTimeOfIncompleteChunkedMessageMillis":60000,"cryptoFailureAction":"FAIL","properties":{"application":"pulsar-sink","id":"public/default/data1-cc47863d","instance_hostname":"data1-cc47863d-sink-0","instance_id":"0"},"readCompacted":false,"subscriptionInitialPosition":"Latest","patternAutoDiscoveryPeriod":60,"regexSubscriptionMode":"PersistentOnly","deadLetterPolicy":null,"retryEnable":false,"autoUpdatePartitions":true,"autoUpdatePartitionsIntervalSeconds":60,"replicateSubscriptionState":false,"resetIncludeHead":false,"keySharedPolicy":null,"batchIndexAckEnabled":false,"ackReceiptEnabled":false,"poolMessages":false,"maxPendingChuckedMessage":10}
2023-03-23T02:18:42,939+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Starting Pulsar consumer status recorder with config: {"topicNames":["public/default/my-topic"],"topicsPattern":null,"subscriptionName":"11","subscriptionType":"Shared","subscriptionMode":"Durable","receiverQueueSize":1000,"acknowledgementsGroupTimeMicros":100000,"negativeAckRedeliveryDelayMicros":60000000,"maxTotalReceiverQueueSizeAcrossPartitions":50000,"consumerName":null,"ackTimeoutMillis":0,"tickDurationMillis":1000,"priorityLevel":0,"maxPendingChunkedMessage":10,"autoAckOldestChunkedMessageOnQueueFull":false,"expireTimeOfIncompleteChunkedMessageMillis":60000,"cryptoFailureAction":"FAIL","properties":{"application":"pulsar-sink","id":"public/default/data1-cc47863d","instance_hostname":"data1-cc47863d-sink-0","instance_id":"0"},"readCompacted":false,"subscriptionInitialPosition":"Latest","patternAutoDiscoveryPeriod":60,"regexSubscriptionMode":"PersistentOnly","deadLetterPolicy":null,"retryEnable":false,"autoUpdatePartitions":true,"autoUpdatePartitionsIntervalSeconds":60,"replicateSubscriptionState":false,"resetIncludeHead":false,"keySharedPolicy":null,"batchIndexAckEnabled":false,"ackReceiptEnabled":false,"poolMessages":false,"maxPendingChuckedMessage":10}