Monitor and Troubleshoot Connectors - StreamNative Documentation

StreamNative Cloud allows you to monitor connectors status, logs, and exceptions that are thrown when a source or sink connector fails to be created, updated, or cannot work.

View connector status

This section describes how to view connector status using snctl, pulsarctl, pulsar-admin, Rest API, and console.

If you want to monitor connectors using snctl, pulsarctl or pulsar-admin, Rest API, make sure you have set up your client tool. For more information, see set up client tools.

The following example introduces how to view the status of the data generator source connector named test.

To check the status of the source connector test, run the following command:

snctl pulsar admin sources status --tenant public --namespace default --name test

{
  "numInstances": 1,
  "numRunning": 1,
  "instances": [
    {
      "instanceId": 0,
      "status": {
        "running": true,
        "error": "",
        "numRestarts": 0,
        "numReceivedFromSource": 2622,
        "numSystemExceptions": 0,
        "latestSystemExceptions": [],
        "numSourceExceptions": 0,
        "latestSourceExceptions": [],
        "numWritten": 2622,
        "lastReceivedTime": 1691532145625,
        "workerId": "test"
      }
  ]
}

To check the status of the source connector test, run the following command:

snctl pulsar admin sources status --tenant public --namespace default --name test

{
  "numInstances": 1,
  "numRunning": 1,
  "instances": [
    {
      "instanceId": 0,
      "status": {
        "running": true,
        "error": "",
        "numRestarts": 0,
        "numReceivedFromSource": 2622,
        "numSystemExceptions": 0,
        "latestSystemExceptions": [],
        "numSourceExceptions": 0,
        "latestSourceExceptions": [],
        "numWritten": 2622,
        "lastReceivedTime": 1691532145625,
        "workerId": "test"
      }
  ]
}

To check the status of the source connector test, run the following command:

pulsarctl sources status --tenant public --namespace default --name test

{
  "numInstances": 1,
  "numRunning": 1,
  "instances": [
    {
      "instanceId": 0,
      "status": {
        "running": true,
        "error": "",
        "numRestarts": 0,
        "numReceivedFromSource": 2622,
        "numSystemExceptions": 0,
        "latestSystemExceptions": [],
        "numSourceExceptions": 0,
        "latestSourceExceptions": [],
        "numWritten": 2622,
        "lastReceivedTime": 1691532145625,
        "workerId": "test"
      }
  ]
}

To check the status of the source connector test, run the following command:

./bin/pulsar-admin sources status --tenant public --namespace default --name test

You should see the following output:

{
  "numInstances" : 1,
  "numRunning" : 1,
  "instances" : [ {
    "instanceId" : 0,
    "status" : {
      "running" : true,
      "error" : "",
      "numRestarts" : 0,
      "numReceivedFromSource" : 433,
      "numSystemExceptions" : 0,
      "latestSystemExceptions" : [ ],
      "numSourceExceptions" : 0,
      "latestSourceExceptions" : [ ],
      "numWritten" : 433,
      "lastReceivedTime" : 1691452485845,
      "workerId" : "test"
    }
  } ]
}

From the left navigation pane, under Resources, click Connectors.
Select the Created Sinks tab or the Created Sources tab to view the status and exceptions of the connector, as well as the system exceptions.

To check the status of the source connector test, run the following command:

curl -X GET https://${WEB_SERVICE_URL}/admin/v3/sources/public/default/test/status \
  --header 'Authorization: Bearer <API Key>' | jq '.'

You should see the following output:

{
  "numInstances" : 1,
  "numRunning" : 1,
  "instances" : [ {
    "instanceId" : 0,
    "status" : {
      "running" : true,
      "error" : "",
      "numRestarts" : 0,
      "numReceivedFromSource" : 433,
      "numSystemExceptions" : 0,
      "latestSystemExceptions" : [ ],
      "numSourceExceptions" : 0,
      "latestSourceExceptions" : [ ],
      "numWritten" : 433,
      "lastReceivedTime" : 1691452485845,
      "workerId" : "test"
    }
  } ]
}

View connector logs

View the connector logs using `snctl`

This section describes how to view connector logs using snctl.

This example assumes you have installed snctl and initialized snctl configurations.

You can run the snctl logs command to view logs for a specific connector. This table outlines the configuration options that are used for viewing connector logs. For details about all supported fields, you can use the snctl logs -h command to list more information.

Option	Descriptions
`-c` or `--cluster`	The name of your Pulsar cluster where the connector is created.
`-p` or `--component`	The type of component to monitor. Available options are `function`, `sink`, and `source`.
`-f` or `--follow`	Continuously list the connector log history.
`-h` or `--help`	Show usage information about the `snctl logs` command.
`-i` or `--instance`	The name of your Pulsar instance where the connector is created.
`--name`	The name of your connector.
`-o` or `--organization`	The name of your organization where the connector is created.
`--previous`	Print the logs that are generated before the configured timestamp.
`--pulsar-tenant`	The name of your Pulsar tenant where the connector is created.
`--pulsar-namespace`	The name of your Pulsar namespace where the connector is created.
`--since`	List logs more recent than the specific time. Available units are `second`, `minute`, and `hour`, such as `24h`.
`-s` or `--size`	Specify how many lines of recent logs to display.
`--timestamp`	Include timestamps on each line in the log output.

The following command example shows how to view up to 60 lines of the data1 sink connector’s logs within the last 5 hours.

snctl logs --since 5h --organization sndev --instance aws --cluster aws --name data1 --pulsar-tenant public --pulsar-namespace default -p sink -f -s 60

You should see the following output:

2023-03-23T02:18:43,075+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [public/default/my-topic][11] Subscribed to topic on aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650 -- consumer: 0
2023-03-23T02:18:43,075+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [public/default/my-topic][11] Subscribed to topic on aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650 -- consumer: 0
2023-03-23T02:18:43,028+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [public/default/my-topic][11] Subscribing to topic on cnx [id: 0xd4f91768, L:/172.16.136.145:53364 - R:aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650], consumerId 0
2023-03-23T02:18:43,028+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [public/default/my-topic][11] Subscribing to topic on cnx [id: 0xd4f91768, L:/172.16.136.145:53364 - R:aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650], consumerId 0
2023-03-23T02:18:43,026+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xd4f91768, L:/172.16.136.145:53364 - R:aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650]] Connected to server
2023-03-23T02:18:43,026+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xd4f91768, L:/172.16.136.145:53364 - R:aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650]] Connected to server
2023-03-23T02:18:42,951+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Pulsar client config: {"serviceUrl":"pulsar://aws-broker.sndev.svc.cluster.local:6650","authPluginClassName":"org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2","authParams":"*****","authParamMap":null,"operationTimeoutMs":30000,"lookupTimeoutMs":30000,"statsIntervalSeconds":60,"numIoThreads":1,"numListenerThreads":1,"connectionsPerBroker":1,"useTcpNoDelay":true,"useTls":false,"tlsTrustCertsFilePath":null,"tlsAllowInsecureConnection":true,"tlsHostnameVerificationEnable":false,"concurrentLookupRequest":5000,"maxLookupRequest":50000,"maxLookupRedirects":20,"maxNumberOfRejectedRequestPerConnection":50,"keepAliveIntervalSeconds":30,"connectionTimeoutMs":10000,"requestTimeoutMs":60000,"initialBackoffIntervalNanos":100000000,"maxBackoffIntervalNanos":60000000000,"enableBusyWait":false,"listenerName":null,"useKeyStoreTls":false,"sslProvider":null,"tlsTrustStoreType":"JKS","tlsTrustStorePath":null,"tlsTrustStorePassword":null,"tlsCiphers":[],"tlsProtocols":[],"memoryLimitBytes":0,"proxyServiceUrl":null,"proxyProtocol":null,"enableTransaction":false,"socks5ProxyAddress":null,"socks5ProxyUsername":null,"socks5ProxyPassword":null}
2023-03-23T02:18:42,951+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Pulsar client config: {"serviceUrl":"pulsar://aws-broker.sndev.svc.cluster.local:6650","authPluginClassName":"org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2","authParams":"*****","authParamMap":null,"operationTimeoutMs":30000,"lookupTimeoutMs":30000,"statsIntervalSeconds":60,"numIoThreads":1,"numListenerThreads":1,"connectionsPerBroker":1,"useTcpNoDelay":true,"useTls":false,"tlsTrustCertsFilePath":null,"tlsAllowInsecureConnection":true,"tlsHostnameVerificationEnable":false,"concurrentLookupRequest":5000,"maxLookupRequest":50000,"maxLookupRedirects":20,"maxNumberOfRejectedRequestPerConnection":50,"keepAliveIntervalSeconds":30,"connectionTimeoutMs":10000,"requestTimeoutMs":60000,"initialBackoffIntervalNanos":100000000,"maxBackoffIntervalNanos":60000000000,"enableBusyWait":false,"listenerName":null,"useKeyStoreTls":false,"sslProvider":null,"tlsTrustStoreType":"JKS","tlsTrustStorePath":null,"tlsTrustStorePassword":null,"tlsCiphers":[],"tlsProtocols":[],"memoryLimitBytes":0,"proxyServiceUrl":null,"proxyProtocol":null,"enableTransaction":false,"socks5ProxyAddress":null,"socks5ProxyUsername":null,"socks5ProxyPassword":null}
2023-03-23T02:18:42,939+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Starting Pulsar consumer status recorder with config: {"topicNames":["public/default/my-topic"],"topicsPattern":null,"subscriptionName":"11","subscriptionType":"Shared","subscriptionMode":"Durable","receiverQueueSize":1000,"acknowledgementsGroupTimeMicros":100000,"negativeAckRedeliveryDelayMicros":60000000,"maxTotalReceiverQueueSizeAcrossPartitions":50000,"consumerName":null,"ackTimeoutMillis":0,"tickDurationMillis":1000,"priorityLevel":0,"maxPendingChunkedMessage":10,"autoAckOldestChunkedMessageOnQueueFull":false,"expireTimeOfIncompleteChunkedMessageMillis":60000,"cryptoFailureAction":"FAIL","properties":{"application":"pulsar-sink","id":"public/default/data1-cc47863d","instance_hostname":"data1-cc47863d-sink-0","instance_id":"0"},"readCompacted":false,"subscriptionInitialPosition":"Latest","patternAutoDiscoveryPeriod":60,"regexSubscriptionMode":"PersistentOnly","deadLetterPolicy":null,"retryEnable":false,"autoUpdatePartitions":true,"autoUpdatePartitionsIntervalSeconds":60,"replicateSubscriptionState":false,"resetIncludeHead":false,"keySharedPolicy":null,"batchIndexAckEnabled":false,"ackReceiptEnabled":false,"poolMessages":false,"maxPendingChuckedMessage":10}
2023-03-23T02:18:42,939+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Starting Pulsar consumer status recorder with config: {"topicNames":["public/default/my-topic"],"topicsPattern":null,"subscriptionName":"11","subscriptionType":"Shared","subscriptionMode":"Durable","receiverQueueSize":1000,"acknowledgementsGroupTimeMicros":100000,"negativeAckRedeliveryDelayMicros":60000000,"maxTotalReceiverQueueSizeAcrossPartitions":50000,"consumerName":null,"ackTimeoutMillis":0,"tickDurationMillis":1000,"priorityLevel":0,"maxPendingChunkedMessage":10,"autoAckOldestChunkedMessageOnQueueFull":false,"expireTimeOfIncompleteChunkedMessageMillis":60000,"cryptoFailureAction":"FAIL","properties":{"application":"pulsar-sink","id":"public/default/data1-cc47863d","instance_hostname":"data1-cc47863d-sink-0","instance_id":"0"},"readCompacted":false,"subscriptionInitialPosition":"Latest","patternAutoDiscoveryPeriod":60,"regexSubscriptionMode":"PersistentOnly","deadLetterPolicy":null,"retryEnable":false,"autoUpdatePartitions":true,"autoUpdatePartitionsIntervalSeconds":60,"replicateSubscriptionState":false,"resetIncludeHead":false,"keySharedPolicy":null,"batchIndexAckEnabled":false,"ackReceiptEnabled":false,"poolMessages":false,"maxPendingChuckedMessage":10}

View the connector logs from log topics

StreamNative Cloud provide an alternative implementation of the Pulsar Functions log topic(see Pulsar Functions log topic) feature called sidecar mode, which supports using such feature in Connectors.

To enable the sidecar mode, user should provide the following configurations when deploying the connector:

pulsarctl sinks/sources create --custom-runtime-options '{"logTopic":"LOG-TOPIC", "logTopicAgent":"sidecar"}' ...

For more details about the custom-runtime-options, see Custom runtime options.

Once the connector is deployed with the sidecar mode, you can configure consumers to consume messages from the log topic.

Manage Connectors Configuration Reference

On this page

View connector status
View connector logs
View the connector logs using snctl
View the connector logs from log topics