> ## Documentation Index
> Fetch the complete documentation index at: https://docs.streamnative.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitor and Troubleshoot Connectors

StreamNative Cloud allows you to monitor connectors status, logs, and exceptions that are thrown when a source or sink connector fails to be created, updated, or cannot work.

## View connector status

This section describes how to view connector status using `snctl`, `pulsarctl`, `pulsar-admin`, `Rest API`, and console.

<Note title="Note">
  If you want to monitor connectors using `snctl`, `pulsarctl` or `pulsar-admin`, `Rest API`, make sure you have set up your client tool. For more information, see set up client tools.
</Note>

The following example introduces how to view the status of the data generator source connector named `test`.

<Tabs>
  <Tab title="snctl">
    To check the status of the source connector `test`, run the following command:

    ```bash theme={null}
    snctl pulsar admin sources status --tenant public --namespace default --name test
    ```

    ```bash theme={null}
    {
      "numInstances": 1,
      "numRunning": 1,
      "instances": [
        {
          "instanceId": 0,
          "status": {
            "running": true,
            "error": "",
            "numRestarts": 0,
            "numReceivedFromSource": 2622,
            "numSystemExceptions": 0,
            "latestSystemExceptions": [],
            "numSourceExceptions": 0,
            "latestSourceExceptions": [],
            "numWritten": 2622,
            "lastReceivedTime": 1691532145625,
            "workerId": "test"
          }
      ]
    }
    ```
  </Tab>

  <Tab title="pulsarctl">
    To check the status of the source connector `test`, run the following command:

    ```bash theme={null}
    pulsarctl sources status --tenant public --namespace default --name test
    ```

    ```bash theme={null}
    {
      "numInstances": 1,
      "numRunning": 1,
      "instances": [
        {
          "instanceId": 0,
          "status": {
            "running": true,
            "error": "",
            "numRestarts": 0,
            "numReceivedFromSource": 2622,
            "numSystemExceptions": 0,
            "latestSystemExceptions": [],
            "numSourceExceptions": 0,
            "latestSourceExceptions": [],
            "numWritten": 2622,
            "lastReceivedTime": 1691532145625,
            "workerId": "test"
          }
      ]
    }
    ```
  </Tab>

  <Tab title="pulsar-admin">
    To check the status of the source connector `test`, run the following command:

    ```bash theme={null}
    ./bin/pulsar-admin sources status --tenant public --namespace default --name test
    ```

    You should see the following output:

    ```bash theme={null}
    {
      "numInstances" : 1,
      "numRunning" : 1,
      "instances" : [ {
        "instanceId" : 0,
        "status" : {
          "running" : true,
          "error" : "",
          "numRestarts" : 0,
          "numReceivedFromSource" : 433,
          "numSystemExceptions" : 0,
          "latestSystemExceptions" : [ ],
          "numSourceExceptions" : 0,
          "latestSourceExceptions" : [ ],
          "numWritten" : 433,
          "lastReceivedTime" : 1691452485845,
          "workerId" : "test"
        }
      } ]
    }
    ```
  </Tab>

  <Tab title="Console">
    1. From the left navigation pane, under **Resources**, click **Connectors**.

    2. Select the **Created Sinks** tab or the **Created Sources** tab to view the status and exceptions of the connector, as well as the system exceptions.
  </Tab>

  <Tab title="Rest API">
    To check the status of the source connector `test`, run the following command:

    ```bash theme={null}
    curl -X GET https://${WEB_SERVICE_URL}/admin/v3/sources/public/default/test/status \
      --header 'Authorization: Bearer <API Key>' | jq '.'
    ```

    You should see the following output:

    ```bash theme={null}
    {
      "numInstances" : 1,
      "numRunning" : 1,
      "instances" : [ {
        "instanceId" : 0,
        "status" : {
          "running" : true,
          "error" : "",
          "numRestarts" : 0,
          "numReceivedFromSource" : 433,
          "numSystemExceptions" : 0,
          "latestSystemExceptions" : [ ],
          "numSourceExceptions" : 0,
          "latestSourceExceptions" : [ ],
          "numWritten" : 433,
          "lastReceivedTime" : 1691452485845,
          "workerId" : "test"
        }
      } ]
    }
    ```
  </Tab>
</Tabs>

## View connector logs

### View the connector logs using `snctl`

This section describes how to view connector logs using `snctl`.

<Note title="Note">
  This example assumes you have [installed snctl](/tools/cli/snctl/snctl-overview#install-snctl) and [initialized snctl configurations](/tools/cli/snctl/snctl-overview#initialize-snctl-configuration).
</Note>

You can run the `snctl logs` command to view logs for a specific connector. This table outlines the configuration options that are used for viewing connector logs. For details about all supported fields, you can use the `snctl logs -h` command to list more information.

| Option                   | Descriptions                                                                                                     |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------- |
| `-c` or `--cluster`      | The name of your Pulsar cluster where the connector is created.                                                  |
| `-p` or `--component`    | The type of component to monitor. Available options are `function`, `sink`, and `source`.                        |
| `-f` or `--follow`       | Continuously list the connector log history.                                                                     |
| `-h` or `--help`         | Show usage information about the `snctl logs` command.                                                           |
| `-i` or `--instance`     | The name of your Pulsar instance where the connector is created.                                                 |
| `--name`                 | The name of your connector.                                                                                      |
| `-o` or `--organization` | The name of your organization where the connector is created.                                                    |
| `--previous`             | Print the logs that are generated before the configured timestamp.                                               |
| `--pulsar-tenant`        | The name of your Pulsar tenant where the connector is created.                                                   |
| `--pulsar-namespace`     | The name of your Pulsar namespace where the connector is created.                                                |
| `--since`                | List logs more recent than the specific time. Available units are `second`, `minute`, and `hour`, such as `24h`. |
| `-s` or `--size`         | Specify how many lines of recent logs to display.                                                                |
| `--timestamp`            | Include timestamps on each line in the log output.                                                               |

The following command example shows how to view up to 60 lines of the `data1` sink connector’s logs within the last 5 hours.

```bash theme={null}
snctl logs --since 5h --organization sndev --instance aws --cluster aws --name data1 --pulsar-tenant public --pulsar-namespace default -p sink -f -s 60
```

You should see the following output:

```bash theme={null}
2023-03-23T02:18:43,075+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [public/default/my-topic][11] Subscribed to topic on aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650 -- consumer: 0
2023-03-23T02:18:43,075+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [public/default/my-topic][11] Subscribed to topic on aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650 -- consumer: 0
2023-03-23T02:18:43,028+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [public/default/my-topic][11] Subscribing to topic on cnx [id: 0xd4f91768, L:/172.16.136.145:53364 - R:aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650], consumerId 0
2023-03-23T02:18:43,028+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [public/default/my-topic][11] Subscribing to topic on cnx [id: 0xd4f91768, L:/172.16.136.145:53364 - R:aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650], consumerId 0
2023-03-23T02:18:43,026+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xd4f91768, L:/172.16.136.145:53364 - R:aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650]] Connected to server
2023-03-23T02:18:43,026+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xd4f91768, L:/172.16.136.145:53364 - R:aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650]] Connected to server
2023-03-23T02:18:42,951+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Pulsar client config: {"serviceUrl":"pulsar://aws-broker.sndev.svc.cluster.local:6650","authPluginClassName":"org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2","authParams":"*****","authParamMap":null,"operationTimeoutMs":30000,"lookupTimeoutMs":30000,"statsIntervalSeconds":60,"numIoThreads":1,"numListenerThreads":1,"connectionsPerBroker":1,"useTcpNoDelay":true,"useTls":false,"tlsTrustCertsFilePath":null,"tlsAllowInsecureConnection":true,"tlsHostnameVerificationEnable":false,"concurrentLookupRequest":5000,"maxLookupRequest":50000,"maxLookupRedirects":20,"maxNumberOfRejectedRequestPerConnection":50,"keepAliveIntervalSeconds":30,"connectionTimeoutMs":10000,"requestTimeoutMs":60000,"initialBackoffIntervalNanos":100000000,"maxBackoffIntervalNanos":60000000000,"enableBusyWait":false,"listenerName":null,"useKeyStoreTls":false,"sslProvider":null,"tlsTrustStoreType":"JKS","tlsTrustStorePath":null,"tlsTrustStorePassword":null,"tlsCiphers":[],"tlsProtocols":[],"memoryLimitBytes":0,"proxyServiceUrl":null,"proxyProtocol":null,"enableTransaction":false,"socks5ProxyAddress":null,"socks5ProxyUsername":null,"socks5ProxyPassword":null}
2023-03-23T02:18:42,951+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Pulsar client config: {"serviceUrl":"pulsar://aws-broker.sndev.svc.cluster.local:6650","authPluginClassName":"org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2","authParams":"*****","authParamMap":null,"operationTimeoutMs":30000,"lookupTimeoutMs":30000,"statsIntervalSeconds":60,"numIoThreads":1,"numListenerThreads":1,"connectionsPerBroker":1,"useTcpNoDelay":true,"useTls":false,"tlsTrustCertsFilePath":null,"tlsAllowInsecureConnection":true,"tlsHostnameVerificationEnable":false,"concurrentLookupRequest":5000,"maxLookupRequest":50000,"maxLookupRedirects":20,"maxNumberOfRejectedRequestPerConnection":50,"keepAliveIntervalSeconds":30,"connectionTimeoutMs":10000,"requestTimeoutMs":60000,"initialBackoffIntervalNanos":100000000,"maxBackoffIntervalNanos":60000000000,"enableBusyWait":false,"listenerName":null,"useKeyStoreTls":false,"sslProvider":null,"tlsTrustStoreType":"JKS","tlsTrustStorePath":null,"tlsTrustStorePassword":null,"tlsCiphers":[],"tlsProtocols":[],"memoryLimitBytes":0,"proxyServiceUrl":null,"proxyProtocol":null,"enableTransaction":false,"socks5ProxyAddress":null,"socks5ProxyUsername":null,"socks5ProxyPassword":null}
2023-03-23T02:18:42,939+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Starting Pulsar consumer status recorder with config: {"topicNames":["public/default/my-topic"],"topicsPattern":null,"subscriptionName":"11","subscriptionType":"Shared","subscriptionMode":"Durable","receiverQueueSize":1000,"acknowledgementsGroupTimeMicros":100000,"negativeAckRedeliveryDelayMicros":60000000,"maxTotalReceiverQueueSizeAcrossPartitions":50000,"consumerName":null,"ackTimeoutMillis":0,"tickDurationMillis":1000,"priorityLevel":0,"maxPendingChunkedMessage":10,"autoAckOldestChunkedMessageOnQueueFull":false,"expireTimeOfIncompleteChunkedMessageMillis":60000,"cryptoFailureAction":"FAIL","properties":{"application":"pulsar-sink","id":"public/default/data1-cc47863d","instance_hostname":"data1-cc47863d-sink-0","instance_id":"0"},"readCompacted":false,"subscriptionInitialPosition":"Latest","patternAutoDiscoveryPeriod":60,"regexSubscriptionMode":"PersistentOnly","deadLetterPolicy":null,"retryEnable":false,"autoUpdatePartitions":true,"autoUpdatePartitionsIntervalSeconds":60,"replicateSubscriptionState":false,"resetIncludeHead":false,"keySharedPolicy":null,"batchIndexAckEnabled":false,"ackReceiptEnabled":false,"poolMessages":false,"maxPendingChuckedMessage":10}
2023-03-23T02:18:42,939+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Starting Pulsar consumer status recorder with config: {"topicNames":["public/default/my-topic"],"topicsPattern":null,"subscriptionName":"11","subscriptionType":"Shared","subscriptionMode":"Durable","receiverQueueSize":1000,"acknowledgementsGroupTimeMicros":100000,"negativeAckRedeliveryDelayMicros":60000000,"maxTotalReceiverQueueSizeAcrossPartitions":50000,"consumerName":null,"ackTimeoutMillis":0,"tickDurationMillis":1000,"priorityLevel":0,"maxPendingChunkedMessage":10,"autoAckOldestChunkedMessageOnQueueFull":false,"expireTimeOfIncompleteChunkedMessageMillis":60000,"cryptoFailureAction":"FAIL","properties":{"application":"pulsar-sink","id":"public/default/data1-cc47863d","instance_hostname":"data1-cc47863d-sink-0","instance_id":"0"},"readCompacted":false,"subscriptionInitialPosition":"Latest","patternAutoDiscoveryPeriod":60,"regexSubscriptionMode":"PersistentOnly","deadLetterPolicy":null,"retryEnable":false,"autoUpdatePartitions":true,"autoUpdatePartitionsIntervalSeconds":60,"replicateSubscriptionState":false,"resetIncludeHead":false,"keySharedPolicy":null,"batchIndexAckEnabled":false,"ackReceiptEnabled":false,"poolMessages":false,"maxPendingChuckedMessage":10}
```

### View the connector logs from log topics

StreamNative Cloud provide an alternative implementation of the Pulsar Functions log topic(see [Pulsar Functions log topic](https://pulsar.apache.org/docs/en/functions-debug-log-topic)) feature called `sidecar` mode, which supports using such feature in Connectors.

To enable the `sidecar` mode, user should provide the following configurations when deploying the connector:

```yaml theme={null}
pulsarctl sinks/sources create --custom-runtime-options '{"logTopic":"LOG-TOPIC", "logTopicAgent":"sidecar"}' ...
```

For more details about the `custom-runtime-options`, see [Custom runtime options](/cloud/connect/pulsar-io/connector-config#stream-native-cloud-custom-runtime-options).

Once the connector is deployed with the `sidecar` mode, you can configure consumers to consume messages from the log topic.
