# Geo Replication on StreamNative Cloud
Source: https://docs.streamnative.io/cloud/clusters/cloud-geo-replication



Geo Replication on StreamNative Cloud provides an asynchronous mechanism for replicating data between geographically distributed Pulsar clusters. It is typically used for disaster recovery, enabling the replication of persistently stored message data across multiple data centers. For instance, your application is publishing data in one region and you would like to process it for consumption in other regions. With Pulsar's geo-replication mechanism, messages can be produced and consumed in different geo-locations.

## Components of Geo Replication

* Pulsar Instance: a Pulsar instance is a group of Pulsar clusters that function together as a unified entity. Multiple Pulsar clusters under the same Pulsar Instance are automatically configured for the Geo Replication.
* Pulsar Cluster: A Pulsar cluster is a messaging environment which can be distributed across geographical locations and can replicate among themselves using geo-replication.
* Replication policies: Geo-replication is managed at the namespace level or topic level which means you only need to create and configure a namespace or topic to replicate messages between two or more provisioned clusters that a tenant can access.
* Replicated subscriptions: Pulsar supports replicated subscriptions, so you can keep the subscription state in sync, within a sub-second timeframe, in the context of a topic that is being asynchronously replicated across multiple geographical regions.

## Cross-Cluster mTLS Authentication

StreamNative Cloud secures geo-replication traffic between clusters with mutual TLS (mTLS). Each Pulsar cluster is provisioned with its own dedicated `geo-replication-tls` certificate, scoped to that cluster only. This per-cluster certificate model isolates replication credentials, so a certificate event in one cluster does not affect other clusters in the same Pulsar Instance.

Per-cluster certificates also simplify rotation: certificates are issued, renewed, and revoked independently for each cluster, providing defense-in-depth across replicated clusters. No customer action is required—StreamNative Cloud manages the certificate lifecycle automatically.

<Note title="Limitation">
  * The first Pulsar cluster under the Instance can’t be deleted before the following created Pulsar clusters.
  * API Key is not supported for the cross-cluster authentication if the existing TS channel Pulsar cluster exists under the Instance. Users should use the oauth2 as the cross-cluster authentication.
</Note>

## Getting Started

<Tabs>
  <Tab title="Cloud Console">
    **Create Pulsar Instance and first Pulsar cluster**

    1. On the Cloud Console main page, click the **Create** icon to create a new Pulsar Instance and its first Pulsar cluster
       <img alt="Pulsar Instance list" />
    2. Input the Pulsar Instance name and select the Cloud provider.
       <img alt="Pulsar Instance create" />
    3. Input the Pulsar cluster name and select the location.
       <img alt="First Pulsar cluster create" />

    **Create the second Pulsar cluster under the Pulsar Instance**

    1. Get into the Pulsar Instance page and click to create a new Pulsar cluster under the Instance
       <img alt="Pulsar Instance page" />
    2. Input the second Pulsar cluster name
       <img alt="Second Pulsar cluster create" />
    3. After Pulsar Clusters complete the deployment,clusters under the same Pulsar Instance are automatically configured for the Geo Replication.
       <img alt="New Pulsar Instance page" />

    **Grant permissions on Tenant**

    1. Get into one Pulsar cluster and click the "Tenants" on the sidebar.
       <img alt="Tenant page" />
    2. Click the "New Tenant" button and select the allowed clusters from the dropdown. Please note that only the selected allowed clusters could be configured the replication clusters could be used for message replication on the namespace level or topic level.
       <img alt="Tenant replication page" />

    **Enable geo-replication at namespace level**

    1. Get into the Tenant, and Click the "Namespaces" on the sidebar
       <img alt="Namespace page" />

    2. Click the "New Namespace" button, select the replication clusters from the dropdown.
       <img alt="Namespace replication page" />
  </Tab>

  <Tab title="Terraform">
    To create multiple Pulsar clusters under the Pulsar instance through the StreamNative Terraform Provider, just declare the `streamnative_pulsar_cluster` resource in the terraform file and set the `instance_name` and `depends_on` fields:

    * `instance_name`: use the `streamnative_pulsar_instance` resource name.
    * `depends_on`: use the first `streamnative_pulsar_cluster` resource name.

    The full Terraform example:

    ```
    resource "streamnative_pulsar_instance" "geo-instance" {
      organization = "sndev"
      name = "geo-instance"
      availability_mode = "regional"
      pool_name = "shared-aws"
      pool_namespace = "streamnative"
    }

    resource "streamnative_pulsar_cluster" "us-east-cluster" {
      organization    = "sndev"
      name            = "us-east-cluster"
      instance_name   = streamnative_pulsar_instance.geo-instance.name
      location        = "us-east-2"
      release_channel = "rapid"
      bookie_replicas = 3
      broker_replicas = 2
      compute_unit    = 0.3
      storage_unit    = 0.3
    }

    resource "streamnative_pulsar_cluster" "us-west-cluster" {
      depends_on = [ streamnative_pulsar_cluster.us-east-cluster ]
      organization    = "sndev"
      name            = "us-west-cluster"
      instance_name   = streamnative_pulsar_instance.geo-instance.name
      location        = "us-east-2"
      release_channel = "rapid"
      bookie_replicas = 3
      broker_replicas = 2
      compute_unit    = 0.3
      storage_unit    = 0.3
    }
    ```
  </Tab>
</Tabs>


# Maintenance Notification
Source: https://docs.streamnative.io/cloud/clusters/cloud-maintenance-notification



## Overview

StreamNative Cloud performs regular maintenance on your Pulsar clusters and cloud environments to ensure their stability, security, and performance. The Maintenance Notification feature gives you a first-class view of planned and ongoing maintenance directly in the StreamNative Cloud Console. You can review what is changing, when it is scheduled, and — depending on your support plan — take action before the maintenance begins.

## Maintenance notifications

<Note title="Note">
  Please note that completed maintenance notices remain visible for the 90-day retention period
</Note>

All maintenance notifications for your organization are listed on the **Maintenance Notices** page under **Organization Settings** > **Observability**. Notices are sorted with **Action Required** items at the top, so you can quickly see what needs your attention.

<img alt="maintenance notice" />

Click any notice to open the detail page. The detail page shows:

<img alt="maintenance notice" />

* **Change summary** — a description of what StreamNative is changing, such as the Pulsar version being upgraded or the configuration being applied.
* **Target resource** — the cluster or cloud environment affected by the maintenance.
* **Scheduled window** — the planned start and end time for the maintenance.
* **StreamNative owner** — the StreamNative team member responsible for the change.
* **Decision deadline** — the time by which you need to respond if the notice requires your action.

## Email notifications

StreamNative sends email notifications at key points in the maintenance lifecycle:

| Email                 | When it is sent                                                       |
| --------------------- | --------------------------------------------------------------------- |
| Initial notice        | When a maintenance event requiring your attention is ready for review |
| Maintenance started   | When the scheduled maintenance begins                                 |
| Maintenance completed | When the maintenance finishes                                         |

By default, emails are sent to the `technical contact` configured in your [Organization Profile](https://docs.streamnative.io/cloud/security/access/resource-hierarchy/organizations#organization-profile).

<Note title="Note">
  Please note that all future maintenance notifications will be sent by default to the technical contact. Make sure you have configured the correct technical contact email. If you want multiple people to receive the notifications, please create an email group and configure it here.
</Note>

<img alt="maintenance notice" />

## Maintenance Notification and Maintenance Window

StreamNative's routine operational work — such as Pulsar version upgrades — follows the [Maintenance Window](/cloud/clusters/cloud-maintenance-window) you have configured for your cluster.

For maintenance events that require your explicit approval, you will receive an email and the notice will show an **Action Required** status in the console. You can respond directly from the detail page:

* **Approve** — confirm that StreamNative can proceed with the maintenance at the proposed time.
* **Reject** — decline the proposed maintenance. StreamNative will follow up to arrange a new time.
* **Reschedule** — submit your preferred time with an optional comment. StreamNative will review your request and confirm a new window.

If you do not respond before the decision deadline, the maintenance notification will be automatically rejected, and the maintenance will not proceed.

<Note title="Note">
  Please note that only users enrolled in the **Enterprise** or **Production** support plan tier could configure the maintenance window and use the **Approve**, **Reject**, and **Reschedule** actions to schedule the maintenance for your cluster. You can [contact StreamNative sales](https://www.streamnative.io/contact) to upgrade your support plan tier.
</Note>


# Maintenance Window
Source: https://docs.streamnative.io/cloud/clusters/cloud-maintenance-window



## Overview

StreamNative Cloud performs regular maintenance to ensure the stability, performance, and security of the Pulsar cluster. During these periods, we perform updates, patches, and infrastructure improvements. This document outlines the process and best practices for managing maintenance windows.

<Note title="Note">
  Please note that only users enrolled in the **Enterprise** or **Production** support plan tier could configure the maintenance window to schedule the maintenance for your cluster. You can [contact StreamNative sales](https://www.streamnative.io/contact) to upgrade your support plan tier.
</Note>

## What changes do and don't respect maintenance policies

<Tabs>
  <Tab title="Serverless">
    With maintenance window, you can control the timing of the following types of events:

    * Regular Pulsar cluster maintenance, including the Pulsar version upgrade or add some new features. The impact of change should be minimal and clients can handle gracefully.

    Other types of maintenance aren't dependent on maintenance policies:

    * Infrastructure Pools maintenance, including the including control plane upgrades, data plane upgrades which may cause temporary disruption to your cluster.
    * Underlying Cloud services, primarily compute, network, storage.
  </Tab>

  <Tab title="Dedicated">
    With maintenance window, you can control the timing of the following types of events:

    * Regular Pulsar cluster maintenance, including the Pulsar version upgrade or configuration changes. The impact of change should be minimal and clients can handle gracefully.

    Other types of maintenance aren't dependent on maintenance policies:

    * Infrastructure Pools maintenance, including the including control plane upgrades, data plane upgrades which may cause temporary disruption to your cluster.
    * Underlying Cloud services, primarily compute, network, storage.
  </Tab>

  <Tab title="BYOC">
    With maintenance window, you can control the timing of the following types of events:

    * Regular Pulsar cluster maintenance, including the Pulsar version upgrade or configuration changes. The impact of change should be minimal and clients can handle gracefully.
    * Infrastructure Pools maintenance, including the including control plane upgrades, data plane upgrades which may cause temporary disruption to your cluster.

    Other types of maintenance aren't dependent on maintenance policies:

    * Underlying Cloud services, primarily compute, network, storage.
  </Tab>
</Tabs>

## How to configure Maintenance windows

To configure the maintenance window when you create a Pulsar cluster, set the desired Maintenance window on the Cluster Operations.

<img alt="maintenance window" />

When configuring maintenance windows:

* When configuring the maintenance window, the page will use your local time zone but transform into UTC before storing in the end.
* Select the maintenance window start time
  <img alt="start-time" />
* Select the maintenance window duration
  <img alt="duration" />
* Check the permitted days for maintenance window
  <img alt="permitted-days" />


# Release Channel
Source: https://docs.streamnative.io/cloud/clusters/cloud-release-channel



## Introduction

Release channels in our platform provide users with the flexibility to choose between two main update streams: LTS (Long-Term Support) and Rapid. The LTS channel focuses on stability and long-term support, making it ideal for production environments that require minimal disruption. On the other hand, the Rapid channel offers access to the latest features and improvements at a faster pace, suitable for users who prioritize innovation and can adapt to frequent updates. This document aims to clarify the differences between these channels, helping users make informed decisions based on their specific requirements and use cases.

## Release Channel Overview

| Release Channel | Current version in channel |
| --------------- | -------------------------- |
| Rapid channel   | `4.2.1.8`                  |
| LTS channel     | `4.0.10.7`                 |

### LTS (Long-Term Support) Channel

* LTS releases prioritize stability and long-term API consistency.
* New features are introduced on a slower cadence.
* Non-GA (Generally Available) features are not accessible in LTS releases.
* LTS releases are recommended for production workloads and environments requiring stability and minimal disruption.

### Rapid Channel

* Rapid releases offer access to the latest features and improvements at a faster pace.
* New features are introduced approximately every three months.
* Users can access features in Private Preview or Public Preview through controlled flags.
* Rapid releases are suitable for users who prioritize innovation, are willing to adapt to frequent updates, and can accommodate potential changes in features and APIs.

### Channel limitation

* LTS channel clusters can't access the Private Preview or Public Preview features.
* There is a temporary restriction that the release channel cannot be changed after the cluster deployment. If you want to upgrade a cluster to a different release channel, please [reach out to the StreamNative support team](https://support.streamnative.io/hc/en-us/requests/new).

### Choose the release channel

<Tabs>
  <Tab title="Console">
    In the [cluster provision process](/cloud/clusters/manage-clusters/cluster#create-a-cluster), there is a new step for selecting the release channel:

    <img alt="Select Release Channel" />

    After cluster provision, you can view the current enrolled channel through the Configuration tab on the Pulsar Clusters page:

    <img alt="Select Release Channel" />
  </Tab>

  <Tab title="snctl">
    To specify the release channel when creating a Pulsar cluster using `snctl`, you can use the `--release-channel` flag. Here's how to do it:

    1. Create a new Pulsar cluster with a specific release channel:

       ```
       snctl create cluster <cluster-name> --release-channel <channel>
       ```

       Replace `<cluster-name>` with your desired cluster name and `<channel>` with either `lts` or `rapid`.

       For example, to create a cluster named "my-cluster" using the LTS channel:

       ```
       snctl create pulsarcluster my-cluster --release-channel lts
       ```

    2. View the release channel of an existing cluster:

       ```
       snctl get pulsarcluster <cluster-name> -o yaml
       ```

       Look for the `releaseChannel` field in the output:

       ```yaml theme={null}
       apiVersion: cloud.streamnative.io/v1alpha1
       kind: PulsarCluster
       metadata:
         name: my-cluster
         namespace: <organization-id>
       spec:
         releaseChannel: lts
       ```

    3. If you need to update the release channel of an existing cluster, you can use the `patch` command:

       ```
       snctl patch pulsarcluster <cluster-name> --type merge -p '{"spec":{"releaseChannel":"rapid"}}'
       ```

       Note: As mentioned in the channel limitation section, changing the release channel after deployment is currently restricted. Please contact StreamNative support if you need to change the release channel of an existing cluster.

    Remember that the `releaseChannel` field in the PulsarCluster spec determines which channel the cluster will use for updates. The available options are `lts` for the Long-Term Support channel and `rapid` for the Rapid channel.

    Alternatively, you can also include the release channel in the PulsarCluster custom resource YAML file when creating a new cluster:

    ```yaml theme={null}
    apiVersion: cloud.streamnative.io/v1alpha1
    kind: PulsarCluster
    metadata:
      name: my-cluster
      namespace: <organization-id>
    spec:
      <...>
      releaseChannel: lts
      <...>
    ```
  </Tab>

  <Tab title="Terraform">
    To specify the release channel when creating a cluster using [StreamNative Terraform Provider](https://registry.terraform.io/providers/streamnative/streamnative/latest), you can use the `release_channel` field in the `streamnative_pulsar_cluster` resource. Here's an example of how to structure the resource:

    ```hcl theme={null}
    resource "streamnative_pulsar_cluster" "example" {
      name = "<cluster-name>"
      namespace = "<organization-id>"
      release_channel = "rapid"
    }
    ```

    For more details about how to create a cluster using StreamNative Terraform Provider, see [create a pulsar cluster](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/resources/pulsar_cluster). You can also find more examples in the [StreamNative Terraform Provider GitHub repository](https://github.com/streamnative/terraform-provider-streamnative/tree/main/examples).
  </Tab>
</Tabs>

## Key Considerations

### Stability

* Both LTS and Rapid channels provide stable releases.
* Stability is not compromised in either channel; the primary difference lies in the frequency of feature updates.

### Feature Availability

* **LTS**: Prioritizes stability over feature availability, with new features introduced on a yearly basis.
* **Rapid**: Offers the latest features and improvements every three months, allowing users to access cutting-edge functionalities.

### Support

* Both LTS and Rapid channels receive support.
* Users can expect assistance and maintenance regardless of the chosen release channel.

### Compatibility

* Rapid releases eventually transition into LTS releases, ensuring compatibility and forward version upgradeability.
* Users need to consider compatibility when selecting the appropriate release channel, especially for long-term projects and deployments.

## Recommendations

* **Active Development Projects**: The Rapid release channel is recommended for projects that are actively evolving and require access to the latest features and improvements.
* **Stable Environments**: The LTS release channel is suitable for stable environments and production workloads where long-term stability and minimal disruption are paramount.

## Best Practices

* **Clear Channel Mapping**: Ensure a clear mapping of clusters to different release channels, especially in environments with multiple Pulsar clusters.
* **Communication and Training**: Educate users about the implications of choosing between LTS and Rapid channels, including potential impacts on development workflows, feature availability, and long-term support.
* **Continuous Integration and Deployment (CI/CD)**: Incorporate release channel selection into CI/CD pipelines to ensure alignment with project requirements and compatibility considerations.

## Conclusion

Understanding the differences between LTS and Rapid release channels is essential for making informed decisions regarding software updates and maintenance. By aligning channel selection with project requirements and considering factors such as stability, feature availability, and compatibility, users can optimize their experience with our platform and maximize the value derived from new features and improvements.


# Cluster Profiles Overview
Source: https://docs.streamnative.io/cloud/clusters/cluster-profiles-overview



A cluster profile is a preconfigured performance and cost setting that determines how a data streaming cluster behaves based on its underlying storage, latency expectations, and workload characteristics. Each profile packages the right combination of infrastructure choices—such as disk-based storage for low-latency access or object storage for cost-efficient retention—to deliver a predictable experience without requiring users to tune individual parameters. By selecting a cluster profile, you can align your cluster with the needs of your applications, whether you prioritize real-time responsiveness, throughput, or cost optimization.

All cluster profiles run on the [URSA engine](/cloud/overview/data-streaming-engine), StreamNative Cloud's unified data streaming engine. The profile you pick determines the write-ahead log (WAL), metadata store, and supported protocols; the cluster type (Pulsar or Kafka) determines the API surface.

## Cluster profile types

StreamNative Cloud offers two cluster profiles. Pick the one that matches your workload's latency and cost priorities.

### Latency-Optimized profile

The Latency-Optimized cluster profile is designed for real-time, interactive, and mission-critical workloads that require consistently fast data access and low end-to-end latency. Backed by high-performance disk storage, this profile delivers predictable, single-digit to tens-of-milliseconds latency for producers and consumers, making it ideal for use cases such as event processing, fraud detection, user activity tracking, and any application where responsiveness directly impacts user experience or business outcomes. By prioritizing speed over cost, the Latency-Optimized profile ensures smooth, high-throughput performance even under demanding conditions.

<Frame>
  <img alt="Latency-Optimized cluster profile architecture" />
</Frame>

<Tip>
  Clusters using the Latency-Optimized profile are expected to deliver end-to-end latencies in the 5–200 millisecond range.
</Tip>

The infrastructure underpinning the Latency-Optimized profile varies by cluster type:

* **Pulsar Clusters** use Apache BookKeeper as the low-latency WAL. The metadata store is ZooKeeper by default; Oxia is available on request. Pulsar Clusters support the native Pulsar protocol, and the Kafka protocol is available through the [KSN](/kafka/kafka-cluster-vs-ksn) protocol handler with full Kafka feature parity. Lakehouse integration (Iceberg, Delta) is built in via the URSA storage layer.
* **Kafka Clusters** use KRaft and local disks as the low-latency WAL, delivering native Apache Kafka with the full Kafka feature set. Lakehouse integration is built in.

### Cost-Optimized profile

The Cost-Optimized cluster profile is engineered for throughput-intensive and cost-sensitive workloads that benefit from large-scale, durable, and economical object storage. While this profile accepts higher access latencies—typically in the hundreds of milliseconds—it provides significant cost savings and unlimited scalability, making it a strong fit for data pipelines, analytics sinks, long-term retention, and applications that do not require ultra-fast consumption. By leveraging object storage, the Cost-Optimized profile offers an efficient balance of price, durability, and flexibility for workloads where latency tolerance is higher and storage economics matter most.

<Frame>
  <img alt="Cost-Optimized cluster profile architecture" />
</Frame>

<Tip>
  Clusters using the Cost-Optimized profile are expected to deliver end-to-end latencies above 200 milliseconds.
</Tip>

The infrastructure underpinning the Cost-Optimized profile uses object storage—such as Amazon S3, Google Cloud Storage, or Azure Blob Storage—as the WAL, and Oxia as the metadata store. The capabilities exposed vary by cluster type:

* **Pulsar Clusters** currently expose the **Kafka-compatible protocol only** on the Cost-Optimized profile. Native Pulsar protocol support for this profile is coming after the Apache Pulsar 5.0 release. Lakehouse integration is built in.
* **Kafka Clusters** expose the native Kafka protocol with lakehouse integration built in. Kafka transactions and topic compaction on the Cost-Optimized profile are coming soon.

## Pulsar Clusters

Pulsar Clusters support all three deployment options. Serverless automatically manages performance and cost characteristics without user-defined profiles.

<Note>
  On Pulsar Clusters, the Cost-Optimized profile currently supports the Kafka-compatible protocol only. Native Pulsar protocol support for this profile is coming after the Apache Pulsar 5.0 release.
</Note>

### Supported deployment options

| Deployment option               | Latency-Optimized | Cost-Optimized              |
| ------------------------------- | ----------------- | --------------------------- |
| **Serverless**                  | N/A               | N/A                         |
| **Dedicated**                   | Supported         | Not supported (Coming soon) |
| **Bring Your Own Cloud (BYOC)** | Supported         | Supported\*                 |

\* Cost-Optimized BYOC Pulsar Clusters currently support the Kafka-compatible protocol only. Native Pulsar protocol support is coming after the Apache Pulsar 5.0 release.

### Supported features by deployment type

<Tabs>
  <Tab title="Serverless">
    <Note>
      Cluster profiles do not apply to Serverless deployments. Serverless automatically manages performance and cost characteristics without user-defined profiles.
    </Note>

    | Category              | Features                                                            |
    | --------------------- | ------------------------------------------------------------------- |
    | **Protocol**          | Pulsar (native), Kafka (via KSN, with full Kafka feature parity)    |
    | **Table formats**     | Delta, Iceberg                                                      |
    | **Catalog**           | Unity Catalog, Snowflake Horizon Catalog, S3 Tables, Google BigLake |
    | **Storage**           | Disk (internal only)                                                |
    | **Availability Zone** | Single AZ (internal only)                                           |
  </Tab>

  <Tab title="Dedicated">
    <Note>
      The Cost-Optimized profile is coming soon on Dedicated Pulsar Clusters. The table below lists the features available today on the Latency-Optimized profile.
    </Note>

    | Category              | Latency-Optimized profile                                           |
    | --------------------- | ------------------------------------------------------------------- |
    | **Protocol**          | Pulsar (native), Kafka (via KSN, with full Kafka feature parity)    |
    | **Table formats**     | Delta, Iceberg                                                      |
    | **Catalog**           | Unity Catalog, Snowflake Horizon Catalog, S3 Tables, Google BigLake |
    | **WAL**               | Apache BookKeeper                                                   |
    | **Metadata store**    | ZooKeeper (default); Oxia on request                                |
    | **Availability Zone** | Single AZ (Not available), Multi AZ                                 |
  </Tab>

  <Tab title="BYOC">
    | Category              | Latency-Optimized profile                                           | Cost-Optimized profile                                              |
    | --------------------- | ------------------------------------------------------------------- | ------------------------------------------------------------------- |
    | **Protocol**          | Pulsar (native), Kafka (via KSN, with full Kafka feature parity)    | Kafka-compatible (Pulsar protocol coming after Pulsar 5.0)          |
    | **Table formats**     | Delta, Iceberg                                                      | Delta, Iceberg                                                      |
    | **Catalog**           | Unity Catalog, Snowflake Horizon Catalog, S3 Tables, Google BigLake | Unity Catalog, Snowflake Horizon Catalog, S3 Tables, Google BigLake |
    | **WAL**               | Apache BookKeeper                                                   | Object Storage (Amazon S3, Google Cloud Storage, Azure Blob)        |
    | **Metadata store**    | ZooKeeper (default); Oxia on request                                | Oxia                                                                |
    | **Availability Zone** | Single AZ (Not available), Multi AZ                                 | Single AZ (Not available), Multi AZ                                 |
  </Tab>
</Tabs>

## Kafka Clusters

<Note>
  Kafka Clusters are currently in Public Preview. For a comparison with Kafka compatibility on Pulsar Clusters (KSN), see [Kafka Cluster vs. KSN](/kafka/kafka-cluster-vs-ksn).
</Note>

<Note>
  Kafka transactions and topic compaction on the Cost-Optimized profile are coming soon. Use the Latency-Optimized profile today if your workloads require these features.
</Note>

Kafka Clusters run native Apache Kafka on the URSA engine and support both cluster profiles. Serverless support is planned.

### Supported deployment options

| Deployment option               | Latency-Optimized | Cost-Optimized |
| ------------------------------- | ----------------- | -------------- |
| **Serverless**                  | Coming soon       | Coming soon    |
| **Dedicated**                   | Supported         | Supported      |
| **Bring Your Own Cloud (BYOC)** | Supported         | Supported      |

### Supported features by deployment type

<Tabs>
  <Tab title="Serverless">
    <Note>
      Serverless support for Kafka Clusters is coming soon.
    </Note>
  </Tab>

  <Tab title="Dedicated">
    | Category              | Latency-Optimized profile                                           | Cost-Optimized profile                                              |
    | --------------------- | ------------------------------------------------------------------- | ------------------------------------------------------------------- |
    | **Protocol**          | Kafka (native)                                                      | Kafka (native)                                                      |
    | **Table formats**     | Delta, Iceberg                                                      | Delta, Iceberg                                                      |
    | **Catalog**           | Unity Catalog, Snowflake Horizon Catalog, S3 Tables, Google BigLake | Unity Catalog, Snowflake Horizon Catalog, S3 Tables, Google BigLake |
    | **WAL**               | Local disk (KRaft + ISR)                                            | Object Storage (Amazon S3, Google Cloud Storage, Azure Blob)        |
    | **Metadata store**    | KRaft                                                               | Oxia                                                                |
    | **Caveats**           | —                                                                   | Kafka transactions and topic compaction coming soon                 |
    | **Availability Zone** | Single AZ (Not available), Multi AZ                                 | Single AZ (Not available), Multi AZ                                 |
  </Tab>

  <Tab title="BYOC">
    | Category              | Latency-Optimized profile                                           | Cost-Optimized profile                                              |
    | --------------------- | ------------------------------------------------------------------- | ------------------------------------------------------------------- |
    | **Protocol**          | Kafka (native)                                                      | Kafka (native)                                                      |
    | **Table formats**     | Delta, Iceberg                                                      | Delta, Iceberg                                                      |
    | **Catalog**           | Unity Catalog, Snowflake Horizon Catalog, S3 Tables, Google BigLake | Unity Catalog, Snowflake Horizon Catalog, S3 Tables, Google BigLake |
    | **WAL**               | Local disk (KRaft + ISR)                                            | Object Storage (Amazon S3, Google Cloud Storage, Azure Blob)        |
    | **Metadata store**    | KRaft                                                               | Oxia                                                                |
    | **Caveats**           | —                                                                   | Kafka transactions and topic compaction coming soon                 |
    | **Availability Zone** | Single AZ (Not available), Multi AZ                                 | Single AZ (Not available), Multi AZ                                 |
  </Tab>
</Tabs>

## Next

* [Cluster Types and Regions](/cloud/clusters/cluster-types)
* [Kafka Cluster Guide](/kafka/kafka-cluster-guide)
* [Kafka Cluster vs. KSN](/kafka/kafka-cluster-vs-ksn)
* [Choose Kafka or Pulsar](/cloud/overview/choose-kafka-or-pulsar)


# Cluster Types & Regions in StreamNative Cloud
Source: https://docs.streamnative.io/cloud/clusters/cluster-types



StreamNative offers various cluster types in the StreamNative Cloud. The type of cluster you choose impacts its features, capabilities, and cost. Use this guide to identify the cluster that best meets your requirements.

StreamNative Cloud supports both **Kafka Clusters** and **Pulsar Clusters**. Kafka Clusters run native Apache Kafka protocol, while Pulsar Clusters support Pulsar protocol natively with optional Kafka compatibility through KSN. For help choosing between them, see [Choose Kafka or Pulsar](/cloud/overview/choose-kafka-or-pulsar).

For deployment models, use Serverless clusters for experimentation and early development. For a production cluster, choose from Dedicated (formerly Hosted) clusters, BYOC, or BYOC Pro clusters.

<Note>
  Native Kafka Clusters are currently available on Dedicated and BYOC deployment models. Serverless support for Kafka Clusters is coming soon.
</Note>

The table below offers a high-level comparison of features across StreamNative Cloud cluster types.

| Category                                                 | Feature                                         | Serverless                                          | Dedicated                                           | BYOC                                                | BYOC Pro                                            |
| -------------------------------------------------------- | ----------------------------------------------- | --------------------------------------------------- | --------------------------------------------------- | --------------------------------------------------- | --------------------------------------------------- |
| **Cluster Service**                                      | Single AZ Clusters                              | N/A                                                 | Yes                                                 | Yes                                                 | Yes                                                 |
|                                                          | Multi-AZ Clusters                               | N/A                                                 | Yes                                                 | Yes                                                 | Yes                                                 |
|                                                          | Uptime SLA                                      | 99.9%                                               | 99.5% Single AZ / 99.99% Multi AZ²                  | 99.5% Single AZ / 99.99% Multi AZ                   | 99.5% Single AZ / 99.99% Multi AZ²                  |
|                                                          | Unlimited Pulsar Clusters                       | Yes                                                 | Yes                                                 | Yes                                                 | Yes                                                 |
|                                                          | Autoscaling                                     | On by default                                       | Configurable                                        | Configurable                                        | Configurable                                        |
|                                                          | Latency Optimized Profile                       | N/A                                                 | Yes                                                 | Yes                                                 | Yes                                                 |
|                                                          | Cost Optimized Profile                          | N/A                                                 | Coming Soon                                         | Yes                                                 | Yes                                                 |
|                                                          | Geo-Replication                                 | Yes                                                 | Yes                                                 | Yes                                                 | Yes                                                 |
| **Infrastructure & Provisioning & Network Connectivity** | Supported Cloud Providers                       | AWS, GCP, Azure                                     | AWS, GCP, Azure                                     | AWS, GCP, Azure                                     | AWS, GCP, Azure                                     |
|                                                          | Choose any cloud region                         | No                                                  | No¹                                                 | Yes                                                 | Yes                                                 |
|                                                          | Dedicated VPC                                   | No                                                  | Yes                                                 | Yes                                                 | Yes                                                 |
|                                                          | Private Link                                    | No                                                  | Pro only                                            | Yes                                                 | Yes                                                 |
|                                                          | VPC/VNet Peering                                | No                                                  | Pro only                                            | No                                                  | Yes                                                 |
|                                                          | Transit Gateway                                 | No                                                  | Pro only                                            | No                                                  | Yes                                                 |
|                                                          | Bring Your Own Network (BYON)                   | No                                                  | No                                                  | No                                                  | Yes                                                 |
|                                                          | Bring Your Own Domain (Custom DNS)              | No                                                  | No                                                  | No                                                  | Yes                                                 |
| **Observability**                                        | Metrics API                                     | Yes                                                 | Yes                                                 | Yes                                                 | Yes                                                 |
|                                                          | Remote writes to external observability systems | No                                                  | Pro only                                            | No                                                  | Yes                                                 |
| **Maintenance & Operations**                             | Custom Maintenance Window                       | `Production` or `Enterprise` Support Plan tier only | `Production` or `Enterprise` Support Plan tier only | `Production` or `Enterprise` Support Plan tier only | `Production` or `Enterprise` Support Plan tier only |
| **Multi-Protocol Support**                               | Pulsar                                          | Yes                                                 | Yes                                                 | Yes                                                 | Yes                                                 |
|                                                          | Kafka                                           | Yes                                                 | Yes                                                 | Yes                                                 | Yes                                                 |
|                                                          | MQTT                                            | Yes                                                 | Yes                                                 | Yes                                                 | Yes                                                 |
|                                                          | WebSocket                                       | Yes                                                 | Yes                                                 | Yes                                                 | Yes                                                 |
|                                                          | REST                                            | Yes                                                 | Yes                                                 | Yes                                                 | Yes                                                 |
| **Connectivity and Processing**                          | Pulsar IO (Built-in & Custom)                   | Yes                                                 | Yes                                                 | Yes                                                 | Yes                                                 |
|                                                          | Kafka Connect (Built-in & Custom)               | Yes                                                 | Yes                                                 | Yes                                                 | Yes                                                 |
|                                                          | Pulsar Functions                                | Yes                                                 | Yes                                                 | Yes                                                 | Yes                                                 |
|                                                          | Managed Flink                                   | No                                                  | No                                                  | Yes (Private Preview)                               | Yes (Private Preview)                               |
|                                                          | Trusted Mode for Pulsar Functions and Pulsar IO | No                                                  | No                                                  | No                                                  | Yes                                                 |
| **Data Storage**                                         | Tiered Storage                                  | Transparent                                         | Transparent                                         | Your Bucket                                         | Your Bucket                                         |
|                                                          | Data Backup and Recovery                        | No                                                  | No                                                  | No                                                  | yes                                                 |
| **Security**                                             | Multi-tenancy                                   | Yes                                                 | Yes                                                 | Yes                                                 | Yes                                                 |
|                                                          | Authentication                                  | Yes                                                 | Yes                                                 | Yes                                                 | Yes                                                 |
|                                                          | Authorization                                   | Yes                                                 | Yes                                                 | Yes                                                 | Yes                                                 |
|                                                          | Audit Logs                                      | Yes                                                 | Yes                                                 | Yes                                                 | Yes                                                 |
|                                                          | Data-at-rest Encryption                         | Yes                                                 | Yes                                                 | Yes                                                 | Yes                                                 |
|                                                          | TLS Encryption                                  | Yes                                                 | Yes                                                 | Yes                                                 | Yes                                                 |
|                                                          | End-to-end Encryption                           | Yes                                                 | Yes                                                 | Yes                                                 | Yes                                                 |
|                                                          | Bring Your Own Key                              | No                                                  | Pro only                                            | No                                                  | Yes                                                 |
| \*\*                                                     |                                                 |                                                     |                                                     |                                                     |                                                     |

<Note title="Important">
  The capabilities provided in this topic are for planning purposes, and are not a guarantee of performance, which varies depending on each unique configuration.
</Note>

¹ Contact [StreamNative support](https://support.streamnative.io/hc/en-us/requests/new) for custom region requirements in Dedicated clusters.

² Dedicated Pro and BYOC Pro geo-replicated clusters achieve a 99.999% uptime SLA.

## Serverless Clusters

<span />

Serverless clusters are the newest addition to StreamNative Cloud, offering a fully managed, auto-scaling solution with minimal operational overhead. Key features include:

1. Instant provisioning with a minimum base cost of 1 ETU
2. Automatic scaling based on your workload, with billing only for resources used.
3. Simplified management with StreamNative handling all infrastructure concerns.
4. Ideal for development, testing, and production workloads with variable traffic patterns.

StreamNative uses [Elastic Throughput Units (ETUs)](/cloud/billing/billing-overview#elastic-throughput-unit-etu) to provision and bill for the Serverless clusters.

### ETU limits per Serverless cluster

Serverless clusters are elastic, shrinking and expanding automatically based on load. You don't need to size your cluster. When you need more capacity, your Serverless cluster expands up to the fixed maximum. When no capacity is in use, a minimum charge of 1 ETU applies.

**Serverless cluster capacity**

| Dimension | Minimum | Maximum |
| --------- | ------- | ------- |
| ETUs      | 1       | 20      |

If no capacity is consumed, the system defaults to a minimum billing level of 1 ETU. For more information, see [Elastic Throughput Unit (ETU)](/cloud/billing/billing-overview#elastic-throughput-unit-etu).

### ETU capacity guidance

The dimensions in the following table describe the capacity of a single ETU. For more information about ETU, see [Elastic Throughput Unit (ETU)](/cloud/billing/billing-overview#elastic-throughput-unit-etu) and [ETU vs CU/SU](/cloud/billing/billing-overview#etu-vs-cu-su).

| Dimension         | ETU Capacity                   |
| ----------------- | ------------------------------ |
| Ingress (Data In) | 5 megabytes per second (MBps)  |
| Egress (Data Out) | 15 megabytes per second (MBps) |
| Data Entries      | 500 entries per second         |

### Serverless limits per cluster

| Dimension         | Capability            | Additional details                                                                                                                                                                                                                                                                                                         |
| ----------------- | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Ingress (Data In) | Max 100 MBps          | Number of bytes that can be produced to the cluster in one second. To reduce usage on this dimension, you can compress your messages. `lz4` is recommended for compression.                                                                                                                                                |
| Egress (Data Out) | Max 300 MBps          | Number of bytes that can be consumed from the cluster in one second. To reduce usage on this dimension, you can compress your messages and ensure each consumer is only consuming from the topics it requires. `lz4` is recommended for compression.                                                                       |
| Data Entries      | Max 10,000 per second | Number of data entries produced to and consumed from the cluster in one second. Each data entry represents a batch of messages. Both Pulsar and Kafka clients do batching at the client side. To reduce usage on this dimension, you can adjust producer batching configurations and shut down otherwise inactive clients. |

### Serverless limits per partition

The partition capabilities that follow are based on benchmarking and intended as practical guidelines for planning purposes. Performance per partition will vary based on your specific configuration, and these benchmarks do not guarantee performance.

| Dimension             | Capability |
| --------------------- | ---------- |
| Ingress per partition | 5 MBps     |
| Egress per partition  | 15 MBps    |
| Storage per partition | Unlimited  |

### Serverless Cloud Providers & Regions

Serverless clusters are currently available in limited regions. Please check the [StreamNative Cloud console](/cloud/get-started/cloud-console) for the most up-to-date information on available regions.

### Serverless Features and Usage Limits

Here are some guidelines for the serverless cluster limits:

<Note title="Note">
  The following max capacity is a guidance. Whether your serverless cluster can reach such capacity depending on the workload, number of topics, number of connections, and etc. If you want more predictable capacity, please contact StreamNative Sales
</Note>

| Dimension    | Capability            | Additional details                                                                                                                                                                                                                                                                                                                                                                                                          |
| ------------ | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Ingress      | Max 100 MBps          | Number of bytes that can be produced to the cluster in one second. To reduce usage on this dimension, you can compress your messages. `lz4` is recommended for compression.                                                                                                                                                                                                                                                 |
| Egress       | Max 300 MBps          | Number of bytes that can be consumed from the cluster in one second. To reduce usage on this dimension, you can compress your messages and ensure each consumer is only consuming from the topics it requires. `lz4` is recommended for compression.                                                                                                                                                                        |
| Storage      | Unlimited             | Number of bytes retained on the cluster, pre-replication. You can configure retention policy settings at namespace or topic level so you can control exactly how much and how long to retain data in a way that makes sense for your applications and helps control your costs. To reduce usage on this dimension, you can compress your messages and reduce your retention settings. `lz4` is recommended for compression. |
| Data Entries | Max 10,000 per second | Number of data entries produced to and consumed from the cluster in one second. Each data entry represents a batch of messages. To reduce usage on this dimension, you can adjust producer batching configurations and shut down otherwise inactive clients.                                                                                                                                                                |
| Message size | Max 5 MB              | None                                                                                                                                                                                                                                                                                                                                                                                                                        |

In addition to the usage limits, there are some additional limitations in features:

1. Tiered Storage is transparent, meaning you don't need to configure it. However, Serverless clusters don't support bringing your own bucket. If you need to use your own object storage bucket, you should consider using [BYOC](#byoc-clusters) or [BYOC Pro](#byoc-pro-clusters) clusters.

2. Serverless clusters use [Rapid Release Channel](/cloud/clusters/cloud-release-channel) by default. You can't choose a different release channel.

3. [Auto-scaling](/cloud/clusters/scale-clusters/cloud-autoscaling) is enabled by default. You don't need to configure it.

4. Remote writes to external observability systems are not supported.

5. Custom maintenance windows are not available.

6. Private networking options are not available.

## Dedicated Clusters

<span />

Dedicated (formerly Hosted) clusters are fully managed on StreamNative's cloud infrastructure. They support the following capabilities:

1. Deployment in supported regions on AWS, GCP, and Azure with [a 99.95% uptime SLA for Single-Zone and 99.99% for Multi-Zone](https://streamnative.io/pricing).

2. Optional multi-zone high availability, spreading a cluster across three availability zones for enhanced resilience.

3. Simplified scaling in terms of [Compute Units (CUs) and Storage Units (SUs)](/cloud/billing/billing-overview#cu-and-su).

4. Programmable or [automatic scaling](/cloud/clusters/scale-clusters/cloud-autoscaling) options.

### Dedicated Pro Offering

StreamNative offers a **Pro** offering within Dedicated clusters that provides enhanced networking and security capabilities for enterprise workloads. Dedicated Pro includes:

* **Private Link**: Secure private connectivity to your cluster
* **VPC/VNet Peering**: Connect your cluster to your existing network infrastructure
* **Transit Gateway**: Advanced network routing and connectivity options
* **Remote writes**: Send metrics to external observability systems
* **Managed Flink**: Stream processing capabilities (Private Preview, currently GCP only)
* **Bring Your Own Key**: Use your own encryption keys for data-at-rest encryption

Dedicated Pro maintains the same uptime SLAs and scaling capabilities as standard Dedicated clusters while providing additional enterprise-grade features.

### Dedicated Cloud Providers & Regions

<span />

The following is a list of cloud providers along with the regions and zones supported for Dedicated (formerly Hosted) Clusters:

#### AWS

| Identifier     | Location              |
| -------------- | --------------------- |
| ap-southeast-2 | Asia Pacific (Sydney) |
| eu-central-1   | Europe (Frankfurt)    |
| eu-west-1      | Europe (Ireland)      |
| us-east-2      | US East (Ohio)        |

#### GCP

| Identifier   | Location                            |
| ------------ | ----------------------------------- |
| europe-west1 | St. Ghislain, Belgium, Europe       |
| us-central1  | Council Bluffs, Iowa, North America |

#### Azure

| Identifier | Location |
| ---------- | -------- |
| eastus     | US East  |

<Note title="Need a Different Region?">
  If you need to deploy a Dedicated cluster in a region not listed above, [contact StreamNative support](https://support.streamnative.io/hc/en-us/requests/new) to discuss your requirements. We may be able to accommodate specific region requests for Dedicated clusters.
</Note>

### Dedicated Features and Usage Limits

<span />

The following table outlines the features and usage limits for the Dedicated (formerly Hosted) Clusters:

<table>
  <tbody>
    <tr>
      <th>Type</th>
      <th>Features</th>
      <th>Capability</th>
    </tr>

    <tr>
      <td>Service</td>
      <td>Uptime SLA</td>
      <td>99.95%</td>
    </tr>

    <tr>
      <td>Multi AZ</td>
      <td>99.99%</td>
    </tr>

    <tr>
      <td>Scale</td>
      <td>Throughput limit per topic</td>
      <td>Max 100 MBps </td>
    </tr>

    <tr>
      <td>Storage limit per topic </td>
      <td>Max 1000 TB</td>
    </tr>

    <tr>
      <td>Tenant limit</td>
      <td>Max 128</td>
    </tr>

    <tr>
      <td>Namespace limit </td>
      <td>Max 1024</td>
    </tr>

    <tr>
      <td>Topic limit</td>
      <td>Max 10240</td>
    </tr>

    <tr>
      <td>Cloud providers</td>
      <td>GCP</td>
      <td>Yes</td>
    </tr>

    <tr>
      <td>AWS</td>
      <td>Yes</td>
    </tr>

    <tr>
      <td>Azure</td>
      <td>Yes</td>
    </tr>
  </tbody>
</table>

In addition to the above, there are additional limitations in Dedicated (formerly Hosted) clusters:

1. Tiered Storage is transparent, meaning you don't need to configure it. However, Dedicated clusters don't support bringing your own bucket. If you need to use your own bucket, you should consider using [BYOC](#byoc-clusters) or [BYOC Pro](#byoc-pro-clusters) clusters.

2. Remote writes to external observability systems are only available in Dedicated Pro.

3. Custom maintenance windows are not available.

4. Private networking options (Private Link, VPC/VNet Peering, Transit Gateway) are only available in Dedicated Pro.

## BYOC Clusters

<span />

BYOC clusters are designed for production-ready deployments in your Cloud account, tailored to meet your data security, compliance, and sovereignty requirements. They offer the following capabilities:

1. Dedicated deployments in your chosen region within your cloud account (AWS, GCP, Azure) with [a 99.5% uptime SLA for Single-Zone and 99.99% for Multi-Zone](https://streamnative.io/pricing).

2. Private networking options including AWS PrivateLink, Azure PrivateLink, and GCP Private Service Connect.

3. Optional multi-zone high availability, spreading a cluster across three availability zones for increased resilience.

4. Simplified scaling in terms of CUs and SUs.

5. Programmable or [automatic scaling](/cloud/clusters/scale-clusters/cloud-autoscaling) options.

6. You can choose between **Latency Optimized Profile** and **Cost Optimized Profile**.

StreamNative uses [CU/SU](/cloud/billing/billing-overview#cu-and-su) to bill for the clusters with **Latency Optimized Profile** and uses [Elastic Throughput Units (ETUs)](/cloud/billing/billing-overview#elastic-throughput-unit-etu) to bill for the clusters with **Cost Optimized Profile**.

### BYOC Cloud Providers & Regions

A BYOC cluster can be deployed in any selected region in your cloud account across AWS, GCP, and Azure.

### BYOC Features and Usage Limits

1. The performance limit is majorly bound by the underlying resources of your cloud account.

2. You can use your own S3-compatible storage bucket for the clusters with **Cost Optimized Profile** or **Lakehouse tiered storage** for the clusters with **Latency Optimized Profile**.

3. You can use your own keys for data-at-rest encryption.

4. Only PrivateLink is supported for private networking. Other private networking options are not supported. You can manually configure them if you choose to use a different private networking option, but it will be out of scope for StreamNative support.

5. Geo-replication via private networking is not supported. Only public networking geo-replication is supported. If you need geo-replication via private networking, you should consider using [BYOC Pro](#byoc-pro-clusters).

6. Remote writes to external observability systems are not supported.

7. Custom maintenance windows are not available.

8. Bring Your Own Network (BYON) is not supported. Only StreamNative-managed VPCs are available.

9. Custom DNS (Bring Your Own Domain) is not supported.

### ETU capacity guidance

The dimensions in the following table describe the capacity of a single ETU in a BYOC cluster. For more information about ETUs, see [Elastic Throughput Unit (ETU)](/cloud/billing/billing-overview#elastic-throughput-unit-etu) and [ETU vs CU/SU](/cloud/billing/billing-overview#etu-vs-cu-su).

| Dimension         | ETU Capacity                   |
| ----------------- | ------------------------------ |
| Ingress (Data In) | 25 megabytes per second (MBps) |
| Egress (Data Out) | 75 megabytes per second (MBps) |
| Data Entries      | 2500 entries per second        |

## BYOC Pro Clusters

<span />

BYOC Pro Clusters are designed for critical production workloads and offer enhanced security and networking features, including:

1. Advanced private networking options like VPC/VNet Peering and Transit Gateway.

2. Remote writes to external observability systems.

3. Self-managed keys (Bring-Your-Own-Key) for AWS, Azure, or GCP.

4. Bring Your Own Network (BYON) — use your own VPC/VNet instead of a StreamNative-managed VPC.

5. Custom DNS (Bring Your Own Domain) — use your own DNS zone for cluster endpoints.

<Note>
  **Lakehouse tiered storage** is available in both BYOC standard and BYOC Pro clusters, allowing you to use your own S3-compatible storage bucket for the **Cost Optimized Profile** or **Lakehouse tiered storage** for the **Latency Optimized Profile**.
</Note>

Similar to BYOC clusters, BYOC Pro uses [CU/SU](/cloud/billing/billing-overview#cu-and-su) to bill for clusters with **Latency Optimized Profile** and uses [Elastic Throughput Units (ETUs)](/cloud/billing/billing-overview#elastic-throughput-unit-etu) for clusters with **Cost Optimized Profile**.

### BYOC Pro Cloud Providers & Regions

BYOC Pro Clusters can be deployed in any selected region in your cloud account across AWS, GCP, and Azure.

### BYOC Pro Features and Usage Limits

BYOC Pro is the most secure and flexible option, which provides you with full control over your data and network configurations.

1. The performance limit is majorly bound by the underlying resources of your cloud account.

2. You can use your own S3-compatible storage bucket for the clusters with **Cost Optimized Profile** or **Lakehouse tiered storage** for the clusters with **Latency Optimized Profile**.

3. You can use your own keys for data-at-rest encryption.

4. Private Link, VPC/VNet Peering, and Transit Gateway are supported for private networking.

5. Geo-replication via private networking is supported.

### ETU capacity guidance

The dimensions in the following table describe the capacity of a single ETU in a BYOC Pro cluster with **Cost Optimized Profile** . For more information about ETUs, see [Elastic Throughput Unit (ETU)](/cloud/billing/billing-overview#elastic-throughput-unit-etu) and [ETU vs CU/SU](/cloud/billing/billing-overview#etu-vs-cu-su).

| Dimension         | ETU Capacity                   |
| ----------------- | ------------------------------ |
| Ingress (Data In) | 25 megabytes per second (MBps) |
| Egress (Data Out) | 75 megabytes per second (MBps) |
| Data Entries      | 2500 entries per second        |


# Cluster Configuration Overview
Source: https://docs.streamnative.io/cloud/clusters/configure-clusters/cluster-configuration-overview



The `config` section in a PulsarCluster specification allows users to configure the features and settings to be used for their Pulsar cluster. This topic aims to provide a comprehensive overview of the available configuration options, helping users tailor their Pulsar clusters to specific requirements and use cases.

## Available Configuration Options

The available settings in `config` section in a PulsarCluster specification can be grouped into the following three categories:

1. **Cluster Features**: These settings control the various features and functionalities of your Pulsar cluster.
2. **Protocols**: This setting allows you to configure the supported protocols for your Pulsar cluster, including Kafka, AMQP, MQTT, and Websocket.
3. **Cluster Settings**: This setting allows you to configure custom cluster settings for your Pulsar cluster.

The detailed descriptions of the available settings are as follows:

The `config` section in a PulsarCluster specification includes the following key components:

| Category  | Configuration Option | Description                                        |
| --------- | -------------------- | -------------------------------------------------- |
| Features  | FunctionEnabled      | Controls whether Pulsar Functions are enabled      |
| Features  | TransactionEnabled   | Controls whether Pulsar Transactions are enabled   |
| Features  | AuditLog             | Configures audit log settings                      |
| Protocols | Protocols            | Configures supported protocols (Kafka, AMQP, MQTT) |
| Protocols | WebsocketEnabled     | Controls whether WebSocket protocol is enabled     |
| Settings  | Custom               | Allows for custom cluster settings                 |

## Update Cluster Configuration

To update the cluster configuration, you can use one of the following methods:

<Tabs>
  <Tab title="Console">
    In the [cluster provision process](/cloud/clusters/manage-clusters/cluster#create-a-cluster), when you select a release channel, you can choose to enable or disable the cluster features and protocols based on your requirements. You can also click the "Add optional custom configuration" to set custom cluster settings for your Pulsar cluster.

    <img alt="Configure Cluster" />

    After cluster is provisioned, you can view the current cluster configuration through the "Configuration" page of your Pulsar cluster. You can also update the cluster configuration by clicking the "Edit Cluster" button at the top right corner of the page. You will be redirected to the same configuration page you see in the provision process, where you can make the desired changes to the configuration options.

    <img alt="Cluster Configuration" />
  </Tab>

  <Tab title="snctl">
    To update the cluster configuration using `snctl`:

    1. Ensure you have the latest version of [`snctl`](/tools/cli/snctl/snctl-overview) installed and configured.

    2. Update the PulsarCluster CRD file (e.g., `<cluster-name>.yaml`) with the desired configuration options.

       ```yaml theme={null}
       apiVersion: pulsar.streamnative.cloud/v1beta1
       kind: PulsarCluster
       metadata:
         name: <cluster-name>
       spec:
         config:
           # Add your desired configuration options here
           functionEnabled: true
           transactionEnabled: true
           <...>
       ```

    3. Apply the updated PulsarCluster CRD file to the cluster:

       ```bash theme={null}
       snctl apply -f <cluster-name>.yaml
       ```

    To view the cluster configuration, you can use the following command:

    ```bash theme={null}
    snctl get pulsarcluster <cluster-name> -o yaml
    ```
  </Tab>

  <Tab title="Terraform">
    To update the cluster configuration using the StreamNative Terraform Provider:

    1. Ensure you have the [StreamNative Terraform Provider](https://registry.terraform.io/providers/streamnative/streamnative/latest) configured in your environment.

    2. Use the `streamnative_pulsar_cluster` resource to define or update your cluster configuration.

    3. Specify the desired configuration options within the `config` block.

    Here's an example of how to update the cluster configuration using Terraform:

    ```hcl theme={null}
    resource "streamnative_pulsar_cluster" "example" {
      name = "example"
      config {
        websocket_enabled = true
        transaction_enabled = true
        <...>
      }
    }
    ```

    To apply the changes, run the following command:

    ```bash theme={null}
    terraform apply
    ```

    To view the cluster configuration, you can use the following command:

    ```bash theme={null}
    terraform show
    ```

    For more details, please refer to the [StreamNative Terraform Provider documentation](https://registry.terraform.io/providers/streamnative/streamnative/latest). See [a complete example](https://github.com/streamnative/terraform-provider-streamnative/blob/main/examples/pulsarclusters/hosted.tf) in the StreamNative Terraform Provider GitHub repository.
  </Tab>
</Tabs>

Please note that not all configuration options are modifiable. Some settings are determined by the release channel you selected during the cluster provision process and cannot be changed. Updating the cluster configuration will cause the cluster to roll restart all the brokers. Please make sure to update the cluster configuration during off-peak hours to avoid any disruptions to your service.

## Next Steps

* [Configure Cluster Features](/cloud/clusters/configure-clusters/configure-cluster-features)
* [Configure Protocols](/cloud/clusters/configure-clusters/configure-protocols)
* [Configure Cluster Settings](/cloud/clusters/configure-clusters/configure-cluster-settings)


# Configure Cluster Features
Source: https://docs.streamnative.io/cloud/clusters/configure-clusters/configure-cluster-features



This topic describes how to enable, disable, and configure cluster features for your cluster. For general guidance on configuring your cluster, see [Cluster Configuration Overview](/cloud/clusters/configure-clusters/cluster-configuration-overview).

<Note>
  To protect running workloads, the Cloud Console blocks both disabling and deleting a cluster feature while active resources are still attached to it. Remove or migrate the dependent resources first, then retry the disable or delete action.
</Note>

The following features are available for configuration:

1. [Pulsar Functions](#pulsar-functions)
2. [Pulsar Transactions](#pulsar-transactions)
3. [Audit Log](#audit-log)
4. [Auto Scaling](#auto-scaling)

## Pulsar Functions

[Pulsar Functions](/cloud/process/pulsar-functions/functions-overview) allow you to process messages as they move between topics and external systems. This feature can be enabled or disabled for your cluster.

To enable Pulsar Functions for your cluster, you can set the `config.functionEnabled` field in your PulsarCluster specification to `true` if you are using `snctl` or `config.function_enabled` field in your PulsarCluster specification to `true` if you are using Terraform.

Please note that if you disable Pulsar Functions, you will not able to create new functions in the cluster. Existing functions will continue to run and process messages until they are deleted, but they are not be managed or monitored anymore.

## Pulsar Transactions

[Pulsar Transactions](/cloud/build/pulsar-clients/transactions-overview) provide atomic operations across multiple topics and partitions. This feature can be enabled or disabled for your cluster.

To enable Pulsar Transactions for your cluster, you can set the `config.transactionEnabled` field in your PulsarCluster specification to `true` if you are using `snctl` or `config.transaction_enabled` field in your PulsarCluster specification to `true` if you are using Terraform.

Please note that if you disable Pulsar Transactions on your cluster where you have enabled it before, you are not recommended to disable it because you will disrupt the existing applications that are using transactions and corrupt the data.

## Audit Log

[Audit logging](/cloud/security/monitor-activity/cloud-audit-log) helps track and record important events and actions within your Pulsar cluster. You can configure audit log settings for enhanced monitoring and compliance.

To configure audit logging for your cluster, you can use the `config.auditLog` field in your PulsarCluster specification. The audit log configuration allows you to specify which categories of events you want to log.

Here's how you can configure audit logging:

1. Using `snctl`:

   ```yaml theme={null}
   apiVersion: cloud.streamnative.io/v1alpha1
   kind: PulsarCluster
   metadata:
     name: my-cluster
   spec:
     config:
       auditLog:
         categories:
           - 'Management'
           - 'Describe'
           - 'Produce'
           - 'Consume'
   ```

2. Using Terraform:

   ```hcl theme={null}
   resource "streamnative_cluster" "my_cluster" {
     # ... other configuration ...

     config {
       audit_log {
         categories = [
           "Management",
           "Describe",
           "Produce",
           "Consume"
         ]
       }
     }
   }
   ```

The `categories` field is a list of strings that specify which types of events you want to include in the audit log. See the [Audit Log Overview](/cloud/security/monitor-activity/cloud-audit-log) for more details on the available categories.

You can add or remove categories based on your specific auditing needs. The exact list of available categories may depend on your StreamNative Cloud version, so consult the [documentation](/cloud/security/monitor-activity/cloud-audit-log) for a complete list of supported categories.

Note that enabling extensive audit logging may have performance implications and increase storage requirements. It's recommended to carefully consider which categories are necessary for your use case and compliance requirements.

To disable audit logging, you can remove the `config.auditLog` field in your PulsarCluster specification.

## Auto Scaling

Auto-scaling is a cluster-level feature that allows you to dynamically adjust the number of Broker nodes. You can refer [Configure Auto-Scaling](/cloud/clusters/scale-clusters/cloud-autoscaling) for more details.


# Configure Cluster Settings
Source: https://docs.streamnative.io/cloud/clusters/configure-clusters/configure-cluster-settings



This topic describes how to configure different cluster settings for your StreamNative Cloud clusters.

## Update cluster settings

To configure the cluster configuration, you can add the key-value setting pair to the `config.custom` field in the `PulsarCluster` specification.

For example, to configure the `ttlDurationDefaultInSeconds` setting via `snctl`, you can add the following to the `PulsarCluster` specification:

```yaml theme={null}
apiVersion: cloud.streamnative.io/v1alpha1
kind: PulsarCluster
metadata:
  name: my-cluster
spec:
  config:
    custom:
      ttlDurationDefaultInSeconds: 10
```

Alternatively, you can configure the `ttlDurationDefaultInSeconds` setting via Terraform by adding the following to the `PulsarCluster` resource:

```hcl theme={null}
resource "streamnative_pulsar_cluster" "my-cluster" {
  name = "my-cluster"
  config = {
    custom = {
      ttlDurationDefaultInSeconds = 10
    }
    }
}
```

## Available cluster settings

<Note title="Note">
  The following table of available cluster settings is subject to change based on StreamNative's ongoing development and improvements. Always refer to the latest documentation or consult with StreamNative support for the most up-to-date list of available settings.
</Note>

### Backlog Settings

| Setting                              | Description                                                            |
| ------------------------------------ | ---------------------------------------------------------------------- |
| `backlogQuotaCheckEnabled`           | Enables or disables the backlog quota check.                           |
| `backlogQuotaCheckIntervalInSeconds` | The interval at which the backlog quota check is performed in seconds. |
| `backlogQuotaDefaultLimitBytes`      | The default limit for the backlog quota in bytes.                      |
| `backlogQuotaDefaultLimitSecond`     | The default limit for the backlog quota in seconds.                    |
| `backlogQuotaDefaultRetentionPolicy` | The default retention policy for the backlog quota.                    |

### Retention Settings

| Setting                           | Description                                                        |
| --------------------------------- | ------------------------------------------------------------------ |
| `retentionCheckIntervalInSeconds` | The interval at which the retention check is performed in seconds. |

### TTL Settings

| Setting                               | Description                                                      |
| ------------------------------------- | ---------------------------------------------------------------- |
| `ttlDurationDefaultInSeconds`         | The default Time-To-Live (TTL) duration for messages in seconds. |
| `messageExpiryCheckIntervalInMinutes` | The interval at which message expiry is checked, in minutes.     |

### Resource Management Settings

| Setting                                                | Description                                                                                                                       |
| ------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- |
| `allowAutoTopicCreation`                               | Enables or disables automatic topic creation.                                                                                     |
| `allowAutoTopicCreationType`                           | Specifies the type of topics that can be automatically created.                                                                   |
| `allowAutoSubscriptionCreation`                        | Enables or disables automatic subscription creation.                                                                              |
| `defaultNumPartitions`                                 | Sets the default number of partitions for automatically created partitioned topics.                                               |
| `brokerDeleteInactiveTopicsEnabled`                    | Enables or disables the deletion of inactive topics by the broker.                                                                |
| `brokerDeleteInactiveTopicsFrequencySeconds`           | Sets the frequency (in seconds) at which the broker checks for and deletes inactive topics.                                       |
| `brokerDeleteInactiveTopicsMode`                       | Specifies the mode for deleting inactive topics (e.g., delete\_when\_no\_subscriptions, delete\_when\_subscriptions\_caught\_up). |
| `brokerDeleteInactivePartitionedTopicMetadataEnabled`  | Enables or disables the deletion of metadata for inactive partitioned topics.                                                     |
| `brokerDeleteInactiveTopicsMaxInactiveDurationSeconds` | Sets the maximum duration (in seconds) a topic can be inactive before it's eligible for deletion.                                 |
| `forceDeleteTenantAllowed`                             | Allows or disallows forced deletion of tenants.                                                                                   |
| `forceDeleteNamespaceAllowed`                          | Allows or disallows forced deletion of namespaces.                                                                                |
| `subscriptionExpirationTimeMinutes`                    | Sets the expiration time (in minutes) for inactive subscriptions.                                                                 |
| `subscriptionExpiryCheckIntervalInMinutes`             | Sets the interval (in minutes) at which the system checks for expired subscriptions.                                              |

### Throttling Settings

| Setting                                            | Description                                                                                |
| -------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| `maxPendingPublishRequestsPerConnection`           | Maximum number of pending publish requests per connection.                                 |
| `brokerMaxConnections`                             | Maximum number of connections allowed to the broker.                                       |
| `brokerMaxConnectionsPerIp`                        | Maximum number of connections allowed per IP address.                                      |
| `maxUnackedMessagesPerConsumer`                    | Maximum number of unacknowledged messages allowed per consumer.                            |
| `maxUnackedMessagesPerSubscription`                | Maximum number of unacknowledged messages allowed per subscription.                        |
| `maxUnackedMessagesPerBroker`                      | Maximum number of unacknowledged messages allowed per broker.                              |
| `maxUnackedMessagesPerSubscriptionOnBrokerBlocked` | Maximum number of unacknowledged messages allowed per subscription when broker is blocked. |
| `topicPublisherThrottlingTickTimeMillis`           | Tick time for topic publisher throttling in milliseconds.                                  |
| `preciseTopicPublishRateLimiterEnable`             | Enables or disables precise topic publish rate limiter.                                    |
| `brokerPublisherThrottlingTickTimeMillis`          | Tick time for broker publisher throttling in milliseconds.                                 |
| `brokerPublisherThrottlingMaxMessageRate`          | Maximum message rate for broker publisher throttling.                                      |
| `brokerPublisherThrottlingMaxByteRate`             | Maximum byte rate for broker publisher throttling.                                         |
| `maxPublishRatePerTopicInMessages`                 | Maximum publish rate per topic in messages.                                                |
| `maxPublishRatePerTopicInBytes`                    | Maximum publish rate per topic in bytes.                                                   |
| `subscribeThrottlingRatePerConsumer`               | Subscribe throttling rate per consumer.                                                    |
| `subscribeRatePeriodPerConsumerInSecond`           | Subscribe rate period per consumer in seconds.                                             |
| `dispatchThrottlingRatePerTopicInMsg`              | Dispatch throttling rate per topic in messages.                                            |
| `dispatchThrottlingRatePerTopicInByte`             | Dispatch throttling rate per topic in bytes.                                               |
| `dispatchThrottlingRatePerSubscriptionInMsg`       | Dispatch throttling rate per subscription in messages.                                     |
| `dispatchThrottlingRatePerSubscriptionInByte`      | Dispatch throttling rate per subscription in bytes.                                        |
| `dispatchThrottlingRatePerReplicatorInMsg`         | Dispatch throttling rate per replicator in messages.                                       |
| `dispatchThrottlingRatePerReplicatorInByte`        | Dispatch throttling rate per replicator in bytes.                                          |
| `dispatchThrottlingRateRelativeToPublishRate`      | Dispatch throttling rate relative to publish rate.                                         |
| `dispatchThrottlingOnNonBacklogConsumerEnabled`    | Enables or disables dispatch throttling on non-backlog consumers.                          |
| `maxMessagePublishBufferSizeInMB`                  | Maximum message publish buffer size in MB.                                                 |
| `dispatcherMaxReadBatchSize`                       | Maximum read batch size for the dispatcher.                                                |

### Quota Settings

| Setting                               | Description                                                          |
| ------------------------------------- | -------------------------------------------------------------------- |
| `maxNamespacesPerTenant`              | Maximum number of namespaces allowed per tenant.                     |
| `maxTopicsPerNamespace`               | Maximum number of topics allowed per namespace.                      |
| `maxProducersPerTopic`                | Maximum number of producers allowed per topic.                       |
| `maxSameAddressProducersPerTopic`     | Maximum number of producers with the same address allowed per topic. |
| `maxConsumersPerTopic`                | Maximum number of consumers allowed per topic.                       |
| `maxSameAddressConsumersPerTopic`     | Maximum number of consumers with the same address allowed per topic. |
| `maxSubscriptionsPerTopic`            | Maximum number of subscriptions allowed per topic.                   |
| `maxConsumersPerSubscription`         | Maximum number of consumers allowed per subscription.                |
| `maxNumPartitionsPerPartitionedTopic` | Maximum number of partitions allowed per partitioned topic.          |
| `replicationProducerQueueSize`        | Size of the queue for replication producers.                         |

### Topic & Subscription Settings

| Setting                                               | Description                                                                           |
| ----------------------------------------------------- | ------------------------------------------------------------------------------------- |
| `enablePersistentTopics`                              | Enables or disables persistent topics.                                                |
| `enableNonPersistentTopics`                           | Enables or disables non-persistent topics.                                            |
| `subscriptionTypesEnabled`                            | Specifies the enabled subscription types.                                             |
| `subscriptionKeySharedEnable`                         | Enables or disables key-shared subscriptions.                                         |
| `subscriptionKeySharedUseConsistentHashing`           | Enables or disables consistent hashing for key-shared subscriptions.                  |
| `subscriptionKeySharedConsistentHashingReplicaPoints` | Sets the number of replica points for consistent hashing in key-shared subscriptions. |

### Deduplication Settings

| Setting                                               | Description                                                                                                     |
| ----------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| `brokerDeduplicationEnabled`                          | Enables or disables message deduplication on the broker.                                                        |
| `brokerDeduplicationMaxNumberOfProducers`             | Maximum number of producers for which information will be kept for deduplication purposes.                      |
| `brokerDeduplicationSnapshotFrequencyInSeconds`       | How often to take a snapshot of the deduplication information.                                                  |
| `brokerDeduplicationSnapshotIntervalSeconds`          | Interval between deduplication information snapshots for indexing.                                              |
| `brokerDeduplicationEntriesInterval`                  | Number of entries after which a deduplication informational snapshot is taken.                                  |
| `brokerDeduplicationProducerInactivityTimeoutMinutes` | Time of inactivity after which the broker will discard deduplication information about a disconnected producer. |

### Namespace Bundles Settings

| Setting                           | Description                          |
| --------------------------------- | ------------------------------------ |
| `defaultNumberOfNamespaceBundles` | Default number of namespace bundles. |

### Message Size Settings

| Setting          | Description                   |
| ---------------- | ----------------------------- |
| `maxMessageSize` | Maximum allowed message size. |

### Compaction Settings

| Setting                                            | Description                                       |
| -------------------------------------------------- | ------------------------------------------------- |
| `brokerServiceCompactionMonitorIntervalInSeconds`  | Interval for monitoring compaction in seconds.    |
| `brokerServiceCompactionThresholdInBytes`          | Threshold for compaction in bytes.                |
| `brokerServiceCompactionPhaseOneLoopTimeInSeconds` | Time for phase one of compaction loop in seconds. |

### Delayed Delivery Settings

| Setting                                  | Description                                                                                                                                                                                                                                                              |
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `delayedDeliveryEnabled`                 | Enables or disables delayed message delivery.                                                                                                                                                                                                                            |
| `delayedDeliveryTickTimeMillis`          | Tick time for delayed delivery in milliseconds.                                                                                                                                                                                                                          |
| `isDelayedDeliveryDeliverAtTimeStrict`   | When `true`, delayed messages are delivered strictly at (not before) their scheduled time. Defaults to `false`.                                                                                                                                                          |
| `delayedDeliveryTrackerFactoryClassName` | Delayed delivery tracker implementation. Allowed values: `org.apache.pulsar.broker.delayed.InMemoryDelayedDeliveryTrackerFactory` (default) or `org.apache.pulsar.broker.delayed.bucket.BucketDelayedDeliveryTrackerFactory` (bucket-based, for large delayed backlogs). |

### Batch-index Acknowledgement Settings

| Setting                                  | Description                                              |
| ---------------------------------------- | -------------------------------------------------------- |
| `acknowledgmentAtBatchIndexLevelEnabled` | Enables or disables acknowledgment at batch index level. |

### Replicated Subscription Settings

| Setting                                                   | Description                                                             |
| --------------------------------------------------------- | ----------------------------------------------------------------------- |
| `enableReplicatedSubscriptions`                           | Enables or disables replicated subscriptions.                           |
| `replicatedSubscriptionsSnapshotFrequencyMillis`          | Frequency of replicated subscriptions snapshot in milliseconds.         |
| `replicatedSubscriptionsSnapshotTimeoutSeconds`           | Timeout for replicated subscriptions snapshot in seconds.               |
| `replicatedSubscriptionsSnapshotMaxCachedPerSubscription` | Maximum cached snapshots per subscription for replicated subscriptions. |

### Topic-level Policy Settings

| Setting                               | Description                                                     |
| ------------------------------------- | --------------------------------------------------------------- |
| `systemTopicEnabled`                  | Enables or disables system topic.                               |
| `topicLevelPoliciesEnabled`           | Enables or disables topic-level policies.                       |
| `exposeTopicLevelMetricsInPrometheus` | Enables or disables exposing topic-level metrics in Prometheus. |

### Load Balancer Settings

| Setting                                          | Description                                                         |
| ------------------------------------------------ | ------------------------------------------------------------------- |
| `loadBalancerBundleUnloadMinThroughputThreshold` | Minimum throughput threshold for bundle unloading in load balancer. |
| `loadBalancerNamespaceBundleMaxSessions`         | Maximum sessions for namespace bundle in load balancer.             |
| `loadBalancerReportUpdateMaxIntervalMinutes`     | Maximum interval for load balancer report update in minutes.        |
| `loadBalancerSheddingEnabled`                    | Enables or disables load shedding in load balancer.                 |
| `loadBalancerSheddingGracePeriodMinutes`         | Grace period for load shedding in minutes.                          |
| `loadBalancerCPUResourceWeight`                  | CPU resource weight for load balancer.                              |

### Other Settings

| Setting                    | Description                     |
| -------------------------- | ------------------------------- |
| `keepAliveIntervalSeconds` | Keep-alive interval in seconds. |
| `numIOThreads`             | Number of I/O threads.          |


# Configure Protocols
Source: https://docs.streamnative.io/cloud/clusters/configure-clusters/configure-protocols



This topic describes how to enable, disable, and configure different messaging and data streaming protocols for your cluster. For general guidance on configuring your cluster, see [Cluster Configuration Overview](/cloud/clusters/configure-clusters/cluster-configuration-overview).

The following protocols are available for configuration:

1. [Kafka Protocol](#kafka-protocol)
2. [AMQP (Advanced Message Queuing Protocol)](#amqp-advanced-message-queuing-protocol)
3. [MQTT (Message Queuing Telemetry Transport)](#mqtt-message-queuing-telemetry-transport)
4. [WebSocket](#websocket)

Making changes to the protocols configuration results in restarting the Pulsar brokers in your cluster. Please plan these changes accordingly to avoid disruption to your business applications.

### Kafka Protocol

Kafka protocol support allows Kafka clients to interact with your Pulsar cluster. This enables seamless integration for applications already using Kafka.

To enable and configure the Kafka protocol for your Pulsar cluster, you can use the `config` section in your PulsarCluster specification. Here's how to do it:

1. In your PulsarCluster specification, locate or add the `config` section.
2. Within the `config` section, add a `protocols` field.
3. In the `protocols` field, add a `kafka` object to configure Kafka-specific settings.

Here's how you can configure Kafka protocol support:

1. Using `snctl`:

   ```yaml theme={null}
   apiVersion: cloud.streamnative.io/v1alpha1
   kind: PulsarCluster
   metadata:
     name: my-cluster
   spec:
     config:
       protocols:
         kafka: {}
   ```

2. Using Terraform:

   ```hcl theme={null}
   resource "streamnative_cluster" "my_cluster" {
     # ... other configuration ...

     config {
       protocols {
         kafka = {
           enabled = "true"
         }
       }
     }
   }
   ```

**Important:** If you try to disable the Kafka protocol in your cluster after you have enabled it, you will disrupt the existing Kafka clients connected to your cluster and cause data corruption.

### AMQP (Advanced Message Queuing Protocol)

<Note title="Note">
  AMQP protocol support is currently in Private Preview and is only available to select users. If you're interested in trying out this feature, please contact StreamNative support for more information.
</Note>

AMQP support allows AMQP-compliant clients to connect to your Pulsar cluster. This is useful for applications that require AMQP compatibility.

To enable and configure the AMQP protocol for your Pulsar cluster, you can use the `config` section in your PulsarCluster specification. Here's how to do it:

1. In your PulsarCluster specification, locate or add the `config` section.
2. Within the `config` section, add a `protocols` field.
3. In the `protocols` field, add a `amqp` object to configure AMQP-specific settings.

Here's how you can configure AMQP protocol support:

1. Using `snctl`:

   ```yaml theme={null}
   apiVersion: cloud.streamnative.io/v1alpha1
   kind: PulsarCluster
   metadata:
     name: my-cluster
   spec:
     config:
       protocols:
         amqp: {}
   ```

2. Using Terraform:

   ```hcl theme={null}
   resource "streamnative_cluster" "my_cluster" {
     # ... other configuration ...

     config {
       protocols {
         amqp = {
           enabled = "true"
         }
       }
     }
   }
   ```

**Important:** If you try to disable the AMQP protocol in your cluster after you have enabled it, you will disrupt the existing AMQP clients connected to your cluster and cause data corruption.

### MQTT (Message Queuing Telemetry Transport)

MQTT support enables IoT devices and applications using the MQTT protocol to communicate with your Pulsar cluster. This is particularly useful for IoT and mobile scenarios.

To enable and configure the MQTT protocol for your Pulsar cluster, you can use the `config` section in your PulsarCluster specification. Here's how to do it:

1. In your PulsarCluster specification, locate or add the `config` section.
2. Within the `config` section, add a `protocols` field.
3. In the `protocols` field, add a `mqtt` object to configure MQTT-specific settings.

Here's how you can configure MQTT protocol support:

1. Using `snctl`:

   ```yaml theme={null}
   apiVersion: cloud.streamnative.io/v1alpha1
   kind: PulsarCluster
   metadata:
     name: my-cluster
   spec:
     config:
       protocols:
         mqtt: {}
   ```

2. Using Terraform:

   ```hcl theme={null}
   resource "streamnative_cluster" "my_cluster" {
     # ... other configuration ...

     config {
       protocols {
         mqtt = {
           enabled = "true"
         }
       }
     }
   }
   ```

**Important:** If you try to disable the MQTT protocol in your cluster after you have enabled it, you will disrupt the existing MQTT clients connected to your cluster and cause data corruption.

### WebSocket

[Pulsar WebSocket](https://pulsar.apache.org/docs/3.3.x/client-libraries-websocket/) support enables real-time, bidirectional communication between clients and your Pulsar cluster via WebSocket. This feature can be enabled or disabled based on your requirements.

To enable WebSocket for your cluster, you can set the `config.websocketEnabled` field in your PulsarCluster specification to `true` if you are using `snctl` or `config.websocket_enabled` field in your PulsarCluster specification to `true` if you are using Terraform.

Please note that if you disable WebSocket, you will not be able to create new WebSocket clients in the cluster. Existing WebSocket clients will be disconnected.


# Manage StreamNative Clusters
Source: https://docs.streamnative.io/cloud/clusters/manage-clusters/cluster



## Cluster Overview

A [StreamNative instance](/cloud/clusters/manage-instances/instance) comprises one or more **StreamNative clusters** that operate in unison. Clusters can be distributed across geographical locations and can replicate among themselves using geo-replication.

<Note title="Note">
  In StreamNative Console, you can create one and only one cluster for an instance.
</Note>

### Cluster Features and Usage limits

You can find the features and usage limits of different Clusters in [Cluster Types and Regions](/cloud/clusters/cluster-types#hosted-features-and-limits).

For details about how to work with clusters, such as creating, editing, checking, and deleting clusters, see [work with clusters](/cloud/clusters/manage-clusters/cluster).

### Cluster Location

Each StreamNative cluster is associated with a geographical location. The locations are available at [Cluster Types & Regions in StreamNative Cloud](/cloud/clusters/cluster-types).

### Cluster Protocols and Settings

Each StreamNative Cluster has a set of protocols and settings that you can enable, disable, and configure.

For details, see:

* [Configuration Overview](/cloud/clusters/configure-clusters/cluster-configuration-overview)
* [Cluster Features](/cloud/clusters/configure-clusters/configure-cluster-features)
* [Available Protocols](/cloud/clusters/configure-clusters/configure-protocols)
* [Cluster Settings](/cloud/clusters/configure-clusters/configure-cluster-settings)

### Cluster Service URLs

Once provisioned, a StreamNative cluster is accessible through service URLs. A service URL is an HTTPS or TLS endpoint that is exposed to the Internet or private network and protected by authentication methods ([OAuth2](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview) and [API Keys](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview)). The Fully Qualified Domain Name (FQDN) of a service endpoint is based on the name of the StreamNative cluster.

StreamNative Cloud provides the following service URLs for a StreamNative cluster:

* **Pulsar HTTP Service URL**: the HTTP URL of a StreamNative cluster. After being granted the Admin access right, you can use the `pulsar-admin` or [`pulsarctl`](/tools/cli/pulsarctl/pulsarctl-overview) CLI tool to connect to and manage the StreamNative cluster.
* **Pulsar Broker Service URL**: the broker URL of a StreamNative cluster. After being granted the `produce` and `consume` permissions, you can use the `pulsar-client` CLI tool or [Pulsar clients](/cloud/build/pulsar-clients/qs-connect) to connect to and communicate with the StreamNative cluster.
* **Pulsar WebSocket Service URL**: the URL for a WebSocket-enabled StreamNative cluster. After being granted the Admin access right, you can use the [WebSocket API](/cloud/build/pulsar-clients/cloud-connect-websocket) to connect to and communicate with the StreamNative cluster.
* **Kafka Service URL**: the URL for a StreamNative cluster with the Kafka protocol enabled. After being granted the Admin access right, you can use the [Kafka CLI tool](/cloud/build/kafka-clients/quick-starts/cloud-connect-kafka-cli) or [Kafka clients](/cloud/build/kafka-clients/kafka-on-cloud#kafka-clients) to connect to and communicate with the StreamNative cluster.
* **Kafka Schema Registry Service URL**: the URL for [Kafka Schema Registry](/cloud/governance/kafka-schemas/kafka-schema-registry) service enabled on a StreamNative cluster.
* **MQTT Service URL**: the URL for [MQTT service](/cloud/build/mqtt-clients/mqtt-on-cloud-overview) enabled on a StreamNative cluster.

## Next steps

* [Manage Serverless Clusters](/cloud/clusters/manage-clusters/manage-serverless-clusters)
* [Manage Dedicated Clusters](/cloud/clusters/manage-clusters/manage-dedicated-clusters)
* [Manage BYOC Clusters](/cloud/clusters/manage-clusters/manage-byoc-clusters)

After creating a cluster, you can:

* [Manage Data Streams](/cloud/manage-data-streams/data-streams-overview)
* [Build Kafka Applications](/cloud/build/kafka-clients/kafka-on-cloud)
* [Build Pulsar Applications](/cloud/build/pulsar-clients/qs-connect)
* [Create a second cluster and configure geo-replication](/cloud/clusters/cloud-geo-replication)


# Manage BYOC Clusters on StreamNative Cloud
Source: https://docs.streamnative.io/cloud/clusters/manage-clusters/manage-byoc-clusters



## Prerequisites

Before creating a BYOC instance and clusters, you need to prepare for the BYOC infrastructure. See [BYOC Overview](/cloud/clusters/byoc/byoc-overview) for more information.

After you have prepared the BYOC infrastructure, you can create a BYOC instance and clusters.

## Create a cluster

<Tabs>
  <Tab title="Console">
    You can follow the steps in [Create a BYOC Instance](/cloud/clusters/manage-instances/manage-byoc-instances#create-an-instance) to create a BYOC instance and its first cluster in it.
  </Tab>
</Tabs>

## View cluster details

<Tabs>
  <Tab title="Console">
    Navigate to the **Clusters Dashboard** page.

    * On the **Dashboard** tab, you can view some metrics about the cluster, including the number of topics, subscriptions, consumers, producers, throughput, storage size, and backlog size.
    * On the **Details** tab, you can view the details about the cluster, including the cluster name, location, availability mode, cloud provider, cluster type, service URLs, the features enabled on the cluster, and your Pulsar, BookKeeper, and ZooKeeper versions.
  </Tab>

  <Tab title="snctl">
    To list all the clusters available for an organization, run the following command:

    ```bash theme={null}
    snctl get pulsarclusters -O <your-org-id>
    ```

    If you want to get more details about a cluster, you can run the following command:

    ```bash theme={null}
    snctl get pulsarcluster <your-cluster-name> -O <your-org-id>
    ```
  </Tab>

  <Tab title="Terraform">
    If you want to get more details about a cluster, you can define a data source in the Terraform configuration file.

    ```hcl theme={null}
    data "streamnative_pulsar_cluster" "test-cluster" {
      organization = "<your-organization>"
      name = "<your-cluster-name>"
    }
    ```

    You can checkout [PulsarCluster](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/data-sources/pulsar_cluster) for more information.
  </Tab>
</Tabs>

## Update a cluster

<Tabs>
  <Tab title="Console">
    1. Navigate to the **Cluster Dashboard** page.

    2. Click **Configuration** at the left navigation pane.

    3. On the **Cluster Configuration** page, you can click **Edit Cluster** button to update the cluster.

    4. On the **Cluster Operations** page, you can update the cluster:

       * **Release Channel**: Currently, it doesn't support switching the release channel after the cluster is created.
       * **Features**: You can enable or disable features on the cluster.
       * **Custom Configuration**: You can update the custom configuration of the cluster.

    5. After update the cluster settings, click **Cluster Size** to change the cluster size.

       You can use **Basic** or **Advanced** tabs to update the cluster size based on your needs.

    6. Click **Finish** when you are done updating your cluster.
  </Tab>
</Tabs>

## Delete a cluster

<Note title="Note">
  You cannot delete a cluster if there are resources associated with the cluster.
</Note>

<Tabs>
  <Tab title="Console">
    1. Navigate to the **Clusters** page.

    2. Click the ellipsis at the top right corner of the cluster card that you want to delete, and then click **Delete**.

    3. In the **Are you sure you want to delete this?** dialog, enter the cluster name and then click **Confirm**.
  </Tab>

  <Tab title="snctl">
    There are two ways to delete a cluster.

    * Delete the cluster by the cluster name.

      ```bash theme={null}
      snctl delete pulsarcluster <your-cluster-name>
      ```

    * Delete the cluster by the cluster manifest file `cluster.yaml`.

      ```bash theme={null}
      snctl delete -f cluster.yaml
      ```
  </Tab>

  <Tab title="Terraform">
    Remove the cluster resource from the Terraform configuration file and run `terraform apply` to delete the cluster.
  </Tab>
</Tabs>


# Manage Dedicated Clusters on StreamNative Cloud
Source: https://docs.streamnative.io/cloud/clusters/manage-clusters/manage-dedicated-clusters



## Create a cluster

<Tabs>
  <Tab title="Console">
    You can follow the steps in [Create a Dedicated Instance](/cloud/clusters/manage-instances/manage-dedicated-instances#create-an-instance) to create a Dedicated instance and its first cluster in it.
  </Tab>
</Tabs>

## View cluster details

<Tabs>
  <Tab title="Console">
    Navigate to the **Clusters Dashboard** page.

    * On the **Dashboard** tab, you can view some metrics about the cluster, including the number of topics, subscriptions, consumers, producers, throughput, storage size, and backlog size.
    * On the **Details** tab, you can view the details about the cluster, including the cluster name, location, availability mode, cloud provider, cluster type, service URLs, the features enabled on the cluster, and your Pulsar, BookKeeper, and ZooKeeper versions.
  </Tab>

  <Tab title="snctl">
    To list all the clusters available for an organization, run the following command:

    ```bash theme={null}
    snctl get pulsarclusters -O <your-org-id>
    ```

    If you want to get more details about a cluster, you can run the following command:

    ```bash theme={null}
    snctl get pulsarcluster <your-cluster-name> -O <your-org-id>
    ```
  </Tab>

  <Tab title="Terraform">
    If you want to get more details about a cluster, you can define a data source in the Terraform configuration file.

    ```hcl theme={null}
    data "streamnative_pulsar_cluster" "test-cluster" {
      organization = "<your-organization>"
      name = "<your-cluster-name>"
    }
    ```

    You can checkout [PulsarCluster](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/data-sources/pulsar_cluster) for more information.
  </Tab>
</Tabs>

## Update a cluster

<Tabs>
  <Tab title="Console">
    1. Navigate to the **Cluster Dashboard** page.

    2. Click **Configuration** at the left navigation pane.

    3. On the **Cluster Configuration** page, you can click **Edit Cluster** button to update the cluster.

    4. On the **Cluster Operations** page, you can update the cluster:

       * **Release Channel**: Currently, it doesn't support switching the release channel after the cluster is created.
       * **Features**: You can enable or disable features on the cluster.
       * **Custom Configuration**: You can update the custom configuration of the cluster.

    5. After update the cluster settings, click **Cluster Size** to change the cluster size.

       You can use **Basic** or **Advanced** tabs to update the cluster size based on your needs.

    6. Click **Finish** when you are done updating your cluster.
  </Tab>
</Tabs>

## Delete a cluster

<Note title="Note">
  You cannot delete a cluster if there are resources associated with the cluster.
</Note>

<Tabs>
  <Tab title="Console">
    1. Navigate to the **Clusters** page.

    2. Click the ellipsis at the top right corner of the cluster card that you want to delete, and then click **Delete**.

    3. In the **Are you sure you want to delete this?** dialog, enter the cluster name and then click **Confirm**.
  </Tab>

  <Tab title="snctl">
    There are two ways to delete a cluster.

    * Delete the cluster by the cluster name.

      ```bash theme={null}
      snctl delete pulsarcluster <your-cluster-name>
      ```

    * Delete the cluster by the cluster manifest file `cluster.yaml`.

      ```bash theme={null}
      snctl delete -f cluster.yaml
      ```
  </Tab>

  <Tab title="Terraform">
    Remove the cluster resource from the Terraform configuration file and run `terraform apply` to delete the cluster.
  </Tab>
</Tabs>


# Manage Serverless Clusters on StreamNative Cloud
Source: https://docs.streamnative.io/cloud/clusters/manage-clusters/manage-serverless-clusters



## Create a cluster

<Tabs>
  <Tab title="Console">
    You can follow the steps in [Create a Serverless Instance](/cloud/clusters/manage-instances/manage-serverless-instances#create-an-instance) to create a Serverless instance and its first cluster in it.
  </Tab>

  <Tab title="Terraform">
    Creating a serverless cluster is straightforward. You just need to select a region and define the cluster in the Terraform configuration file.

    ```hcl theme={null}
    resource "streamnative_pulsar_instance" "test-serverless" {
      ...
    }

    resource "streamnative_pulsar_cluster" "test-serverless" {
      depends_on      = [streamnative_pulsar_instance.test-serverless]
      organization    = streamnative_pulsar_instance.test-serverless.organization
      display_name    = "<your-cluster-name>"
      instance_name   = streamnative_pulsar_instance.test-serverless.name
      location        = "us-central1"
    }
    ```

    * `organization`: The organization ID.
    * `display_name`: The display name of the cluster. Replace `<your-cluster-name>` with the actual name.
    * `instance_name`: The name of the instance where the cluster is created.
    * `location`: The location of the cluster. In this example, it is `us-central1`.

    See [PulsarCluster](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/resources/pulsar_cluster) for more information.
  </Tab>
</Tabs>

## View cluster details

<Tabs>
  <Tab title="Console">
    Navigate to the **Clusters Dashboard** page.

    * On the **Dashboard** tab, you can view some metrics about the cluster, including the number of topics, subscriptions, consumers, producers, throughput, storage size, and backlog size.
    * On the **Details** tab, you can view the details about the cluster, including the cluster name, location, availability mode, cloud provider, cluster type, service URLs, the features enabled on the cluster, and your Pulsar, BookKeeper, and ZooKeeper versions.
  </Tab>

  <Tab title="snctl">
    To list all the clusters available for an organization, run the following command:

    ```bash theme={null}
    snctl get pulsarclusters -O <your-org-id>
    ```

    If you want to get more details about a cluster, you can run the following command:

    ```bash theme={null}
    snctl get pulsarcluster <your-cluster-name> -O <your-org-id>
    ```
  </Tab>

  <Tab title="Terraform">
    If you want to get more details about a cluster, you can define a data source in the Terraform configuration file.

    ```hcl theme={null}
    data "streamnative_pulsar_cluster" "test-cluster" {
      organization = "<your-organization>"
      name = "<your-cluster-name>"
    }
    ```

    You can checkout [PulsarCluster](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/data-sources/pulsar_cluster) for more information.
  </Tab>
</Tabs>

## Update a cluster

**Serverless clusters** are fully autonomous and automatically scaled. You don't need to manually update them.

## Delete a cluster

<Note title="Note">
  You cannot delete a cluster if there are resources associated with the cluster.
</Note>

<Tabs>
  <Tab title="Console">
    1. Navigate to the **Clusters** page.

    2. Click the ellipsis at the top right corner of the cluster card that you want to delete, and then click **Delete**.

    3. In the **Are you sure you want to delete this?** dialog, enter the cluster name and then click **Confirm**.
  </Tab>

  <Tab title="snctl">
    There are two ways to delete a cluster.

    * Delete the cluster by the cluster name.

      ```bash theme={null}
      snctl delete pulsarcluster <your-cluster-name>
      ```

    * Delete the cluster by the cluster manifest file `cluster.yaml`.

      ```bash theme={null}
      snctl delete -f cluster.yaml
      ```
  </Tab>

  <Tab title="Terraform">
    Remove the cluster resource from the Terraform configuration file and run `terraform apply` to delete the cluster.
  </Tab>
</Tabs>


# Configure Auto-Scaling
Source: https://docs.streamnative.io/cloud/clusters/scale-clusters/cloud-autoscaling



<Note title="Note">
  [Serverless clusters](/cloud/clusters/cluster-types#serverless-clusters) are elastic and automatically scale based on workloads. There is no need to configure autoscaling.
</Note>

Auto-Scaling automatically adjusts the available resources of your deployments, and eliminates the need for scripts or consulting services to make scaling decisions. It currently supports scaling for Pulsar Broker nodes only and works on a rolling basis, meaning the process doesn't incur any downtime.

<Note title="Note">
  Auto-scaling for BookKeeper clusters is not supported at this time.
</Note>

You can specify a range of minimum and maximum Pulsar Broker nodes that your Pulsar cluster can automatically scale to, our Cloud auto-scaler will monitor the CPU workload of Broker nodes and adjust their nodes based on the scaling rule.

## Overview of autoscaling

One of the significant challenges faced by organizations dealing with real-time data is ensuring that the underlying infrastructure can handle varying workloads. Traditional scaling methods often involve manual intervention, leading to inefficiency and increased operational costs. Pulsar Broker Autoscaling addresses these challenges by providing an intelligent, automated solution.

The Power of Pulsar Broker Autoscaling:

* Dynamic Resource Allocation: Pulsar Broker Autoscaling dynamically adjusts resources based on the incoming workload. Whether it's handling a sudden spike in traffic or scaling down during periods of low activity, Pulsar ensures optimal resource utilization, leading to cost savings and improved performance.

* Efficient Load Balancing: Autoscaling in Pulsar ensures that the message processing load is evenly distributed across brokers. This prevents any single broker from becoming a bottleneck, allowing the system to maintain high throughput and low latency even under heavy loads.

* Cost-Effective Scaling: Traditional scaling methods often result in over-provisioning to handle peak loads, leading to unnecessary costs. Pulsar Broker Autoscaling optimizes resource allocation, ensuring that organizations pay only for the resources they need, making it a cost-effective solution for real-time data processing.

## When does autoscaling occur?

StreamNative Cloud has an observability stack to collect the Pulsar cluster workload status in real-time. After enabling the Broker nodes Auto-Scaling feature, the scaling controller will track the average CPU usage of the Broker nodes and adjust the Brokers to keep them at the target CPU usage level. If the average CPU usage for the Brokers is over the target, the scaling controller will scale out to bring in more Brokers. If the average CPU usage for the Brokers is less than the target, the scaling controller will downscale brokers to save resources.

## Enable or disable autoscaling

Auto-scaling is a cluster-level feature that allows you to dynamically adjust the number of Broker nodes. Here's how you can enable or disable Broker auto-scaling for your cluster:

<Tabs>
  <Tab title="Cloud Console">
    1. Log in to the [StreamNative Cloud Console](https://console.streamnative.cloud/).

    2. Navigate to your target [Cluster Workspace](/cloud/get-started/cloud-console#cluster-workspace).

    3. Click **Configuration** in the left navigation pane to access **Cluster Configuration**.

    4. On the **Cluster Configuration** page, click the **Edit Cluster** button in the top-right corner. You will be redirected to the **Cluster Provisioning** page where you can update the cluster configuration.
       <img alt="cluster-page" />

    5. Find the `Cluster Autoscaling` feature, enable the switch, and choose the minimum and maximum number of nodes.
       <img alt="enable-autoscaling" />

    6. Click the **Cluster Size** button to move to the next step. Skip the **Cluster Size** page by clicking the **Finish** button to complete the operation.

    7. After enabling Broker autoscaling, you can go back to the **Cluster Dashboard** page to check the number of Broker nodes.
  </Tab>

  <Tab title="snctl">
    To enable or disable Broker auto-scaling using `snctl`, you need to modify the PulsarCluster Custom Resource Definition (CRD). Follow these steps:

    1. Retrieve the current cluster configuration:

       ```
       snctl get -O <organization-id> pulsarcluster <cluster-name> -o yaml > cluster-config.yaml
       ```

    2. Open the `cluster-config.yaml` file in your preferred text editor.

    3. Locate the `spec.broker` section in the YAML file. Add or modify the `autoScalingPolicy` field under `spec.broker`:

       ```yaml theme={null}
       spec:
         broker:
           replicas: 3 # This will be the initial number of replicas
           autoScalingPolicy:
             minReplicas: 2 # Minimum number of broker replicas
             maxReplicas: 5 # Maximum number of broker replicas
       ```

       * Set `minReplicas` to the minimum number of Broker nodes you want to maintain.
       * Set `maxReplicas` to the maximum number of Broker nodes you want to allow.

    4. Save the changes to the `cluster-config.yaml` file.

    5. Apply the updated configuration:

       ```
       snctl apply -f cluster-config.yaml
       ```

    6. To disable auto-scaling, you can either remove the `autoScalingPolicy` field entirely or set `minReplicas` and `maxReplicas` to the same value as `replicas`.

    7. After applying the changes, you can verify the auto-scaling configuration:

       ```
       snctl get -O <organization-id> pulsarcluster <cluster-name> -o yaml
       ```

       Look for the `autoScalingPolicy` field under `spec.broker` to confirm your changes.

    Remember that the auto-scaling feature will automatically adjust the number of Broker nodes based on the CPU usage, within the range you've specified.
  </Tab>
</Tabs>

For other cluster features, You can follow [Configure Cluster Features](/cloud/clusters/configure-clusters/configure-cluster-features) to enable or disable Broker autoscaling.


# Resize a Cluster
Source: https://docs.streamnative.io/cloud/clusters/scale-clusters/resize-a-cluster



<Note title="Note">
  [Serverless clusters](/cloud/clusters/cluster-types#serverless-clusters) are elastic and automatically scale based on workloads. There is no need to manually size or resize a Serverless cluster.
</Note>

This topic describes how to resize an existing StreamNative cluster, including Dedicated, BYOC, and BYOC Pro clusters. These clusters are provisioned and billed in terms of [CUs and SUs](/cloud/billing/billing-overview#cu-su). Evaluating the number of CUs and SUs your cluster requires helps you reduce costs or scale to meet your data streaming needs.

When you resize a StreamNative cluster, StreamNative automatically redistributes the traffic to help ensure a balanced load across the remaining brokers in the cluster. If you're reducing the size of the cluster, StreamNative removes unused brokers at the end of load balancing and removes unused bookies after re-replication.

<Note title="Important">
  Once you request a resize for your cluster, you cannot request another update until the initially requested resize has completed.
</Note>

## Resize a Cluster

This section shows you how to resize an existing StreamNative Cloud cluster, including scaling up and down the cluster and adjusting the required resources.

Prerequisites:

* An existing [StreamNative Cluster](/cloud/clusters/cluster-types)

<Tabs>
  <Tab title="Cloud Console">
    You can expand the number of CUs and SUs in the StreamNative Cloud Console.

    1. Log in to the [StreamNative Cloud Console](https://console.streamnative.cloud/).

    2. Navigate to the [**Cluster Workspace**](/cloud/get-started/cloud-console#cluster-workspace).

    3. Click **Configuration** in the left navigation pane to access **Cluster Configuration**.

    4. On the **Cluster Configuration** page, click the **Edit Cluster** button in the top-right corner. You will be redirected to the **Cluster Provisioning** page where you can update the cluster configuration and adjust the sizing of the cluster.

    5. Skip the **Cluster Configuration** page by clicking the **Cluster Size** button. You will be redirected to the **Cluster Size** page.

    6. On the **Cluster Size** page, you can choose between **Basic** or **Advanced** mode to expand the cluster.

       * **Basic Mode**: Use the slider to adjust the required throughput. StreamNative Cloud will automatically calculate the number of brokers and bookies, as well as their respective CUs and SUs per node.

       * **Advanced Mode**: Manually adjust the number of brokers, bookies, CUs, and SUs per node.

    7. Click the **Finish** button to complete the cluster expansion.

    The cluster will start scaling up, and you will be redirected to a **Deploying** page that shows the progress of the deployment process. The deployment might take a few minutes to complete, depending on how many resources you are scaling up. You can always click **Go To The Dashboard** to return to the **Cluster Dashboard** page. You can see the number of brokers and bookies in real-time on the **Cluster Dashboard** page.
  </Tab>

  <Tab title="snctl">
    CUs and SUs are reflected as the number of CPU and Memory used by the brokers and bookies. When expanding a cluster via `snctl`, you can modify the number of bookies and brokers, as well as their corresponding CPU, Memory, and Disk resources. This is done by updating the **PulsarCluster** Custom Resource Definition (CRD).

    To expand your cluster using [`snctl`](/tools/cli/snctl/snctl-overview), follow these steps:

    1. Retrieve the current cluster configuration:

       ```
       snctl get -O <organization-id> pulsarcluster <cluster-name> -o yaml > cluster-config.yaml
       ```

    2. Edit the `cluster-config.yaml` file to increase the resources. For example:

       * Increase the number of brokers or bookies:

         ```yaml theme={null}
         spec:
           broker:
             replicas: 5 # Increase from default 3 to 5
           bookkeeper:
             replicas: 4 # Increase from default 3 to 4
         ```

       * Adjust the CPU, memory, or storage for brokers or bookies:
         ```yaml theme={null}
         spec:
           broker:
             resources:
               cpu: '4'
               memory: '8Gi'
               heapPercentage: 50
               directPercentage: 50
           bookkeeper:
             resources:
               cpu: '4'
               memory: '8Gi'
               heapPercentage: 50
               directPercentage: 50
               ledgerDisk: '100Gi'
               journalDisk: '50Gi'
         ```

       These examples demonstrate how to modify the PulsarCluster CRD to scale your cluster's resources.

    3. Apply the updated configuration:
       ```
       snctl apply -f cluster-config.yaml
       ```

    After `snctl apply` is completed, the cluster will initiate the expansion process. The operation may take several minutes to complete, depending on the scale of the expansion.

    You can check the status of your PulsarCluster to monitor the progress. The PulsarCluster resource includes a status block that provides valuable information about the current state of your cluster components.

    To check the status, you can use the following command:

    ```bash theme={null}
    snctl clusters -O <organization-id> get <cluster-name> -o yaml
    ```

    Here's an example of what the status block might look like:

    ```yaml theme={null}
    status:
      bookkeeper:
        readyReplicas: 3
        replicas: 3
        updatedReplicas: 3
      broker:
        readyReplicas: 3
        replicas: 3
        updatedReplicas: 3
    ```

    When all the `readyReplicas`, `replicas`, and `updatedReplicas` are the same, the cluster expansion is complete.
  </Tab>

  <Tab title="Terraform Provider">
    CUs (Compute Units) and SUs (Storage Units) are reflected in the CPU and memory resources used by brokers and bookies, respectively. When expanding a cluster via the **Terraform Provider**, you can modify the number of bookies and brokers, as well as the CUs per broker and SUs per bookie. This is accomplished by updating the [**streamnative\_pulsar\_cluster**](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/resources/pulsar_cluster) resource.

    To resize a cluster using the StreamNative Terraform Provider, follow these steps:

    1. Update your Terraform configuration file (e.g., `main.tf`) to modify the cluster resources. Here's an example of how to update the cluster configuration:

       ```hcl theme={null}
       resource "streamnative_pulsar_cluster" "my_cluster" {
         # ... other configuration ...

         broker_replicas = 5  # Increase the number of broker replicas
         bookie_replicas = 5  # Increase the number of bookkeeper replicas
         compute_unit    = 0.4  # Increase compute units per broker
         storage_unit    = 0.4  # Increase storage units per bookie

         config {
           # ... other configuration ...
         }
       }
       ```

       ```

       This example demonstrates how to modify the `streamnative_pulsar_cluster` resource to scale your cluster's resources.

       ```

    2. After updating the configuration, run the following Terraform commands:

       ```bash theme={null}
       terraform plan
       ```

       This command will show you the planned changes to your StreamNative cluster.

    3. If the plan looks correct, apply the changes:

       ```bash theme={null}
       terraform apply
       ```

       Terraform will then initiate the cluster expansion process. The operation may take several minutes to complete, depending on the scale of the expansion.

    4. You can monitor the progress of the cluster scaling operation by checking the Terraform output or by using the **Cluster Dashboard** on the StreamNative Cloud console.

    5. To verify the status of your cluster after the operation, you can use the `terraform show` command or check the **Cluster Dashboard** on StreamNative Cloud console for the most up-to-date information on your cluster's resources and status.

    Remember to always review the changes carefully before applying them to your production environment. It's recommended to test any scaling operations in a non-production environment first.
  </Tab>
</Tabs>


# StreamNative Cloud Clusters Overview
Source: https://docs.streamnative.io/cloud/clusters/streamnative-cluster-overview



Use this topic to understand the cluster concepts in StreamNative Cloud.

## Management Concepts

StreamNative clusters in StreamNative Cloud differ from general Pulsar or Kafka clusters in that StreamNative offers a fully managed service. Use the following concepts to learn more.

### Basics

You can choose from different types of clusters for use in development, production, and high-traffic workloads. StreamNative Cloud is available from all major cloud providers in regions around the world, either fully hosted by StreamNative or as BYOC (Bring-Your-Own-Cloud). This provides access to StreamNative cluster and configuration settings for resources such as tenants, namespaces, and topics. For more information, see [Cluster Types and Regions](/cloud/clusters/cluster-types).

### Fault Tolerance

StreamNative Cloud provides a centralized global control plane that manages a distributed collection of servers running in multiple regions, either fully hosted by StreamNative or as BYOC (Bring-Your-Own-Cloud). Clusters can span cloud provider availability zones (AZs) and provide highly scalable and fault-tolerant systems that support mission-critical applications.

### Hosted vs BYOC

StreamNative Cloud offers two primary deployment models: Hosted and Bring-Your-Own-Cloud (BYOC). Hosted clusters, which include **Serverless** and **Dedicated** options, are deployed and managed entirely on StreamNative's cloud infrastructure. These are ideal for teams seeking a turnkey solution with minimal operational overhead. On the other hand, BYOC clusters allow customers to leverage StreamNative's management expertise while maintaining the clusters within their own cloud environment. This option is particularly suited for organizations with existing cloud infrastructure, specific compliance requirements, or those desiring greater control over their cloud resources. Both models benefit from StreamNative's full management and expertise, ensuring optimal performance and reliability. The choice between Hosted and BYOC depends on factors such as operational preferences, compliance needs, and existing cloud investments. For a detailed comparison of cluster types and available regions, refer to [Cluster Types and Regions](/cloud/clusters/cluster-types).

## Cluster Types

StreamNative Cloud supports two types of clusters:

* **Kafka Clusters**: Native Apache Kafka clusters powered by the Ursa Engine. Kafka Clusters run native Kafka protocol and support standard Kafka clients, tools, and ecosystems. For more information, see the [Kafka Cluster Guide](/kafka/kafka-cluster-guide).
* **Pulsar Clusters**: Apache Pulsar clusters that support Pulsar protocol natively and can also serve Kafka clients through KSN. Pulsar Clusters support additional capabilities such as Functions, Connectors, and multi-protocol access.

<Note>
  Currently, Kafka Clusters and Pulsar Clusters cannot co-exist in the same instance. Each instance supports one cluster type.
</Note>

## Instances & Clusters

StreamNative Cloud organizes its clusters into **Instances**. Within any StreamNative Cloud [organization](/cloud/security/access/resource-hierarchy/organizations), you can create one or more instances. Each instance serves as a dedicated environment within a cloud provider, capable of hosting multiple clusters. This setup allows different departments or teams to operate in separate instances to prevent overlap and interference.

An **Instance** can be either fully **Hosted** on StreamNative's cloud account or on your own public cloud account through the **Bring-Your-Own-Cloud (BYOC)** deployment option.

Within each instance, multiple clusters can be deployed across various regions. Each cluster within an instance is deployed to a specific cloud region as configured in its cloud provider settings. Each cluster provides various service endpoints to enable client libraries to connect, produce, and consume messages.

Clusters within an instance can also replicate data among themselves using geo-replication.

StreamNative Cloud utilizes Public Cloud infrastructure to host instances and clusters. Below is a diagram illustrating the relationship between instances, clusters, and the cloud infrastructure:

<img alt="image of Pulsar Instances" />

## Infrastructure Pools

<span />

As illustrated above, StreamNative Cloud infrastructure is organized into infrastructure **pools**. Each *pool* encompasses a collection of infrastructure environments, known as **pool members**, distributed across multiple regions within a cloud provider. These members can be equated to Kubernetes clusters dedicated to deploying Pulsar clusters.

A **Pulsar Instance** can be created within and deployed to a specific infrastructure *pool*. Each infrastructure *pool* can support multiple **Pulsar Instances**. When setting up a **Pulsar Instance**, you select the appropriate pool using its namespace and name. Within each Pulsar Instance, you can deploy clusters to various regions by specifying the desired region. StreamNative Cloud then deploys the Pulsar Cluster to the corresponding **pool member** in that region.

The type of *Pulsar Instance*, whether **Hosted** or **BYOC**, is determined by the location and configuration of its *Infrastructure Pool*. Infrastructure **Pools** can either be fully **Hosted** by StreamNative or set up in your own public cloud account through the **BYOC** deployment option.

### Hosted Infrastructure Pools

Hosted Infrastructure *Pools* are fully managed and maintained on StreamNative's cloud account. These pools are pre-provisioned, allowing you to utilize them when setting up your Pulsar Instances and Clusters.

If you provision a Pulsar instance or cluster via the StreamNative Cloud console, you can simply select the cloud provider and the respective location. The console will automatically select the appropriate Hosted Pool for you.

If you prefer to provision a Pulsar instance or cluster using the `snctl` command-line tool, you can specify the pool using the `--pool` flag. For example:

```bash theme={null}
snctl create pulsarinstance --name <instance-name> --pool <pool-name> --organization <organization-id>
```

#### Finding Available Hosted Pools

You can use the `snctl get pooloptions` command to list all available Hosted Infrastructure Pools.

```bash theme={null}
$ snctl get pooloptions
NAME                        CREATED AT
streamnative-shared-aws     2022-03-04T00:44:51Z
streamnative-shared-azure   2024-03-25T20:32:15Z
streamnative-shared-gcp     2022-04-26T16:48:31Z
```

Each PoolOption contains the following key information of a **Hosted** pool:

1. **CloudType**: Indicates the cloud provider (e.g., AWS, GCP, Azure) for the pool.
2. **DeploymentType**: The type of deployment (e.g., Hosted, BYOC).
3. **PoolRef**: A reference to the specific infrastructure pool.
4. **Locations**: A list of available locations (regions) for the pool.
5. **Features**: A map of enabled features for the pool.

To view the details of a specific PoolOption, you can use the `snctl get pooloption <pool-name>` command. It will display detailed information about the pool, including its configuration, features, and other relevant settings.

An example of the output of the `snctl get pooloption <pool-name>` command is as follows:

```bash theme={null}
$ snctl get pooloption streamnative-shared-aws -o yaml
apiVersion: cloud.streamnative.io/v1alpha1
kind: PoolOption
metadata:
  creationTimestamp: "2022-03-04T00:44:51Z"
  generation: 28
  name: streamnative-shared-aws
  namespace: sndev
  resourceVersion: "34349179"
  uid: bf56b5c8-e99e-4376-96ab-aecf728343f7
spec:
  cloudType: aws
  deploymentType: ""
  features:
    AOP: true
    ApiKey: true
    AutoScaling: true
    Function: true
    Istio: true
    KOP: true
    MOP: true
    SnRBAC: true
    Transaction: true
    WebSocket: true
  locations:
  - location: ap-southeast-2
  - location: eu-central-1
  - location: eu-west-1
  - location: us-east-2
  poolRef:
    name: shared-aws
    namespace: streamnative
status: {}
```

When selecting a pool for your Pulsar Instance or Cluster, consider the following:

* Choose a pool with a CloudType that matches your preferred cloud provider.
* Ensure the pool has Locations (regions) that meet your geographical requirements.
* Check that the Features align with your needs.
* Verify that the DeploymentType is "Hosted" for fully managed StreamNative infrastructure.

By carefully reviewing the PoolOptions, you can select the most suitable Hosted Infrastructure Pool for your StreamNative Cloud deployment.

### BYOC Infrastructure Pools

Unlike Hosted Infrastructure Pools, BYOC Infrastructure *Pools* must be **provisioned** prior to deploying any BYOC Pulsar Instances or Clusters. The setup process for a BYOC Pool includes establishing a **Cloud Connection**—this allows the StreamNative Cloud control plane to interact with your cloud account. It also involves defining and provisioning **Pool Members** within the Pool to deploy Pulsar Clusters across different regions of your cloud account. Provisioning **Pool Members** within a *Pool* is done by creating **Cloud Environments** in different regions. Each **Cloud Environment** defines the essential infrastructure resources such as compute, storage, and networking that are necessary for a *Pool Member* to deploy Pulsar clusters.

For more information about how to provision and manage BYOC infrastructure pools, see [Manage BYOC Infrastructure](/cloud/clusters/byoc/byoc-overview).

## Related Content

* Learn more information about cluster types and available regions, see [Cluster Types and Regions](/cloud/clusters/cluster-types).
* Learn more about how to provision & manage BYOC infrastructure pools, see [Manage BYOC Infrastructure](/cloud/clusters/byoc/byoc-overview).
* Learn more about how to manage StreamNative instances, see [Manage Instances](/cloud/clusters/manage-instances/instance).
* Learn more about how to manage StreamNative clusters, see [Manage Clusters](/cloud/clusters/manage-clusters/cluster).


# Pulsar IO Connectors
Source: https://docs.streamnative.io/cloud/connect/connector-index



Messaging systems are most powerful when you can easily use them with external systems like databases, cloud services, and other messaging systems.

In Apache Pulsar, connectors are components that facilitate data ingestion and processing within the Pulsar ecosystem. They provide a way to connect Pulsar with various data sources and sinks, allowing seamless integration and exchange of data.

Find out more:

* [Connector overview](/cloud/connect/pulsar-io/connector-overview)

- [Deploy connectors](/cloud/connect/pulsar-io/deploy-connectors/deploy-connector-index)

- [Manage connectors](/cloud/connect/pulsar-io/connector-manage)

- [Monitor and troubleshoot connectors](/cloud/connect/pulsar-io/connector-monitoring)

- [Configuration reference](/cloud/connect/pulsar-io/connector-config)


# Deploy kafka connectors
Source: https://docs.streamnative.io/cloud/connect/kafka-connect/deploy-kafka-connectors/deploy-kafka-connect-index



StreamNative cloud provides a pluggable architecture for kafka connectors, allowing you to deploy pre-built connectors and
develop custom connectors for external systems, such as databases, messaging systems, block storage, cloud services, or
any other system capable of producing or consuming data.

<Tip title="Tip">
  Currently, the Kafka Connect doesn't support multi-tenancy. All Kafka Connects are deployed to the `public/default` namespace.

  And all of configured topics are also under the `public/default` namespace by default, unless you specified topics with the prefix: `${tenant}.${namespace}.`.

  For more details about the multi-tenancy support for Kafka topics in StreamNative cloud, please refer to the [Kafka Multi-Tenancy](/cloud/build/kafka-clients/advanced-features/kafka-multi-tenancy).
</Tip>

To deploy kafka connectors on StreamNative Cloud, follow the following instructions.

* [Set up your environment](/cloud/connect/kafka-connect/deploy-kafka-connectors/kafka-connect-setup)

* [Check kafka connect availability](/cloud/connect/kafka-connect/deploy-kafka-connectors/kafka-connect-check)

* [Create kafka connectors](/cloud/connect/kafka-connect/deploy-kafka-connectors/kafka-connect-create)


# Check Kafka Connect Availability
Source: https://docs.streamnative.io/cloud/connect/kafka-connect/deploy-kafka-connectors/kafka-connect-check



You can check the latest availability of built-in kafka connectors on StreamNative Cloud using `snctl` or `kcctl`.

## Get Built-in Connectors

<Tabs>
  <Tab title="snctl">
    ```
     snctl kafka admin connect get plugins

     TYPE     CLASS                                                       VERSION
     sink     com.milvus.io.kafka.MilvusSinkConnector                     1.0.1
     sink     com.mongodb.kafka.connect.MongoSinkConnector                1.13.0
     sink     com.snowflake.kafka.connector.SnowflakeSinkConnector        2.4.1
     sink     com.wepay.kafka.connect.bigquery.BigQuerySinkConnector      2.5.7
     sink     io.aiven.connect.elasticsearch.ElasticsearchSinkConnector   7.0.0
     sink     io.aiven.connect.jdbc.JdbcSinkConnector                     6.10.0
     sink     io.debezium.connector.jdbc.JdbcSinkConnector                3.2.0.Final
     sink     io.tabular.iceberg.connect.IcebergSinkConnector             0.6.19
     source   com.mongodb.kafka.connect.MongoSourceConnector              1.13.0
     source   io.aiven.connect.jdbc.JdbcSourceConnector                   6.10.0
     source   io.confluent.kafka.connect.datagen.DatagenConnector         0.6.5
     source   io.debezium.connector.mongodb.MongoDbConnector              3.2.0.Final
     source   io.debezium.connector.mysql.MySqlConnector                  3.2.0.Final
     source   io.debezium.connector.postgresql.PostgresConnector          3.2.0.Final
     source   io.debezium.connector.sqlserver.SqlServerConnector          3.2.0.Final
     source   io.debezium.connector.yugabytedb.YugabyteDBConnector        1.9.5.y.220.3
     source   io.jrnd.kafka.connect.connector.JRSourceConnector           0.4.0
    ```
  </Tab>

  <Tab title="kcctl">
    ```
     kcctl get plugins

     TYPE     CLASS                                                       VERSION
     sink     com.milvus.io.kafka.MilvusSinkConnector                     1.0.1
     sink     com.mongodb.kafka.connect.MongoSinkConnector                1.13.0
     sink     com.snowflake.kafka.connector.SnowflakeSinkConnector        2.4.1
     sink     com.wepay.kafka.connect.bigquery.BigQuerySinkConnector      2.5.7
     sink     io.aiven.connect.elasticsearch.ElasticsearchSinkConnector   7.0.0
     sink     io.aiven.connect.jdbc.JdbcSinkConnector                     6.10.0
     sink     io.debezium.connector.jdbc.JdbcSinkConnector                3.2.0.Final
     sink     io.tabular.iceberg.connect.IcebergSinkConnector             0.6.19
     source   com.mongodb.kafka.connect.MongoSourceConnector              1.13.0
     source   io.aiven.connect.jdbc.JdbcSourceConnector                   6.10.0
     source   io.confluent.kafka.connect.datagen.DatagenConnector         0.6.5
     source   io.debezium.connector.mongodb.MongoDbConnector              3.2.0.Final
     source   io.debezium.connector.mysql.MySqlConnector                  3.2.0.Final
     source   io.debezium.connector.postgresql.PostgresConnector          3.2.0.Final
     source   io.debezium.connector.sqlserver.SqlServerConnector          3.2.0.Final
     source   io.debezium.connector.yugabytedb.YugabyteDBConnector        1.9.5.y.220.3
     source   io.jrnd.kafka.connect.connector.JRSourceConnector           0.4.0
    ```
  </Tab>

  <Tab title="Rest API">
    ```
    curl  "https://public%2Fdefault:${APIKEY}@${KAFKA-SERVICE-URL}/admin/kafkaconnect/connector-plugins" | jq '.'

    [
        {
            "class": "io.debezium.connector.mongodb.MongoDbConnector",
            "type": "source",
            "version": "3.2.0.Final"
        },
        {
            "class": "io.aiven.connect.jdbc.JdbcSinkConnector",
            "type": "sink",
            "version": "6.10.0"
        },
        {
            "class": "io.debezium.connector.yugabytedb.YugabyteDBConnector",
            "type": "source",
            "version": "1.9.5.y.220.3"
        },
        {
            "class": "io.confluent.kafka.connect.datagen.DatagenConnector",
            "type": "source",
            "version": "0.6.5"
        },
        {
            "class": "io.debezium.connector.postgresql.PostgresConnector",
            "type": "source",
            "version": "3.2.0.Final"
        },
        {
            "class": "io.debezium.connector.mysql.MySqlConnector",
            "type": "source",
            "version": "3.2.0.Final"
        },
        {
            "class": "io.debezium.connector.sqlserver.SqlServerConnector",
            "type": "source",
            "version": "3.2.0.Final"
        },
        {
            "class": "io.jrnd.kafka.connect.connector.JRSourceConnector",
            "type": "source",
            "version": "0.4.0"
        },
        {
            "class": "com.milvus.io.kafka.MilvusSinkConnector",
            "type": "sink",
            "version": "1.0.1"
        },
        {
            "class": "io.tabular.iceberg.connect.IcebergSinkConnector",
            "type": "sink",
            "version": "0.6.19"
        },
        {
            "class": "com.mongodb.kafka.connect.MongoSinkConnector",
            "type": "sink",
            "version": "1.13.0"
        },
        {
            "class": "com.wepay.kafka.connect.bigquery.BigQuerySinkConnector",
            "type": "sink",
            "version": "2.5.7"
        },
        {
            "class": "io.aiven.connect.jdbc.JdbcSourceConnector",
            "type": "source",
            "version": "6.10.0"
        },
        {
            "class": "io.debezium.connector.jdbc.JdbcSinkConnector",
            "type": "sink",
            "version": "3.2.0.Final"
        },
        {
            "class": "com.snowflake.kafka.connector.SnowflakeSinkConnector",
            "type": "sink",
            "version": "2.4.1"
        },
        {
            "class": "io.aiven.connect.elasticsearch.ElasticsearchSinkConnector",
            "type": "sink",
            "version": "7.0.0"
        },
        {
            "class": "com.mongodb.kafka.connect.MongoSourceConnector",
            "type": "source",
            "version": "1.13.0"
        }
    ]
    ```
  </Tab>
</Tabs>

<Note>
  In addition to the built-in connectors, you can also deploy any custom connectors. For more information, see [Deploy Custom Connectors](/cloud/connect/kafka-connect/deploy-kafka-connectors/kafka-connect-create#create-a-custom-kafka-connect).
</Note>

## What’s next?

* [Create kafka connectors](/cloud/connect/kafka-connect/deploy-kafka-connectors/kafka-connect-create)


# Create Kafka Connectors
Source: https://docs.streamnative.io/cloud/connect/kafka-connect/deploy-kafka-connectors/kafka-connect-create



## Prerequisites

Before deploying a kafka connect to StreamNative Cloud, make sure the following prerequisites have been met:

* A running external data system service.

* A running [Pulsar Cluster](/cloud/clusters/manage-clusters/cluster#create-a-cluster) with Kop feature enabled on StreamNative Cloud and the [required environment](/cloud/connect/kafka-connect/deploy-kafka-connectors/kafka-connect-setup) has been set up.

## Create a built-in kafka connect

<Tip title="Tip">
  Before creating a kafka connect, it’s highly recommended to do the following:

  1. [Check kafka connect availability](/cloud/connect/kafka-connect/deploy-kafka-connectors/kafka-connect-check) to ensure the version number of the kafka connect you want to create is supported on StreamNative Cloud.
  2. Go to [StreamNative Hub](/connect/overview) and find the connect-specific docs of your version for configuration reference.
</Tip>

<Note title="Note">
  You may see below error logs for the first time you create a connector:

  ```
  org.apache.kafka.common.config.ConfigException: Topic '__kafka_connect_offset_storage' supplied via the 'offset.storage.topic' property is required to have 'cleanup.policy=compact' to guarantee consistency and durability of source connector offsets, but found the topic currently has 'cleanup.policy=delete'. Continuing would likely result in eventually losing source connector offsets and problems restarting this Connect cluster in the future. Change the 'offset.storage.topic' property in the Connect worker configurations to use a topic with 'cleanup.policy=compact'.
  ```

  You should set the `cleanup.policy` of the `__kafka_connect_offset_storage` topic to `compact` to avoid the above error with below command:

  ```bash theme={null}
  ./bin/kafka-configs.sh --bootstrap-server xxxx:9093 --command-config ~/kafka/kafka-token.properties --alter --topic __kafka_connect_offset_storage --add-config cleanup.policy=compact
  ```
</Note>

The following example shows how to create a data generator source connect named `test` on Streamnative Cloud using different tools.

<Tabs>
  <Tab title="snctl">
    To create a data generator source connect named `test`, run the following command.

    ```bash theme={null}
    > cat datagen.json
    {
      "name": "test",
      "config": {
        "connector.class": "io.confluent.kafka.connect.datagen.DatagenConnector",
        "kafka.topic": "testusers",
        "quickstart": "users",
        "key.converter": "org.apache.kafka.connect.storage.StringConverter",
        "value.converter": "org.apache.kafka.connect.json.JsonConverter",
        "value.converter.schemas.enable": "false",
        "max.interval": 1000,
        "iterations": 10000000,
        "tasks.max": "1"
      }
    }
    > snctl kafka admin connect apply -f datagen.json --use-service-account
    ```

    You should see the following output:

    ```bash theme={null}
    Created connector test
    ```

    If you want to verify whether the data generator source connect has been created successfully, run the following command:

    ```bash theme={null}
    snctl kafka admin connect get connectors
    ```

    You should see the following output:

    ```bash theme={null}

     NAME                  TYPE     STATE        TASKS
     test                  source   RUNNING      0: RUNNING
    ```
  </Tab>

  <Tab title="kcctl">
    To create a data generator source connect named `test`, run the following command.

    ```bash theme={null}
    > cat datagen.json
    {
      "name": "test",
      "config": {
        "connector.class": "io.confluent.kafka.connect.datagen.DatagenConnector",
        "kafka.topic": "testusers",
        "quickstart": "users",
        "key.converter": "org.apache.kafka.connect.storage.StringConverter",
        "value.converter": "org.apache.kafka.connect.json.JsonConverter",
        "value.converter.schemas.enable": "false",
        "max.interval": 1000,
        "iterations": 10000000,
        "tasks.max": "1"
      }
    }
    > kcctl apply -f datagen.json
    ```

    You should see the following output:

    ```bash theme={null}
    Created connector test
    ```

    If you want to verify whether the data generator source connect has been created successfully, run the following command:

    ```bash theme={null}
    kcctl get connectors
    ```

    You should see the following output:

    ```bash theme={null}

     NAME                  TYPE     STATE        TASKS
     test                  source   RUNNING      0: RUNNING
    ```
  </Tab>

  <Tab title="Rest API">
    To create a data generator source connect named `test`, run the following command.

    ```bash theme={null}
    > cat datagen.json
    {
      "name": "test",
      "config": {
        "connector.class": "io.confluent.kafka.connect.datagen.DatagenConnector",
        "kafka.topic": "testusers",
        "quickstart": "users",
        "key.converter": "org.apache.kafka.connect.storage.StringConverter",
        "value.converter": "org.apache.kafka.connect.json.JsonConverter",
        "value.converter.schemas.enable": "false",
        "max.interval": 1000,
        "iterations": 10000000,
        "tasks.max": "1"
      }
    }
    > curl -X POST --header "Content-Type: application/json" "https://public%2Fdefault:${APIKEY}@${KAFKA-SERVICE-URL}/admin/kafkaconnect/connectors/" --data @datagen.json
    {"name":"test","config":{"connector.class":"io.confluent.kafka.connect.datagen.DatagenConnector","kafka.topic":"testusers","quickstart":"users","key.converter":"org.apache.kafka.connect.storage.StringConverter","value.converter":"org.apache.kafka.connect.json.JsonConverter","value.converter.schemas.enable":"false","max.interval":"1000","iterations":"10000000","tasks.max":"1","name":"test"},"tasks":[],"type":"source"}
    ```

    If you want to list the submitted connect for a double check, run the following command:

    ```bash theme={null}
    > curl "https://public%2Fdefault:${APIKEY}@${KAFKA-SERVICE-URL}/admin/kafkaconnect/connectors/"

    ["test"]
    ```
  </Tab>

  <Tab title="Console">
    To create a data generator source connect named `test`, follow these steps:

    1. Login to the [StreamNative Cloud Console](https://cloud.streamnative.io/).
    2. In the left navigation pane, click **Connectors**, then click the **Kafka Sources** tab.
    3. Click the **Create Kafka Source** button:
       <img alt="image of creating kafka datagen source connect" />
    4. Fill in the required fields and optional fields as you wish, and then Click the **Submit** button.

    If you want to verify whether the data generator source connect has been created successfully, go back to the **Connectors** page, and you should see the created connector in the **Kafka Sources** tab, like below:

    <img alt="image of kafka source connector list overview" />
  </Tab>
</Tabs>

## Create kafka connect with SMT

StreamNative Cloud supports Single Message Transformations (SMTs) for Kafka Connect. You can use SMTs to transform messages before they are written to the target system.

The following example shows how to create a Datagen source connector named `test` on StreamNative Cloud using different tools.

Please refer to the [Kafka Connect SMTs](/cloud/connect/kafka-connect/kafka-connect-smt) to check the supported SMTs in StreamNative cloud.

<Tabs>
  <Tab title="snctl">
    To create a data generator source connect named `test` with SMT, run the following command.

    ```bash theme={null}
    > cat datagen.json
    {
      "name": "test",
      "config": {
        "connector.class": "io.confluent.kafka.connect.datagen.DatagenConnector",
        "kafka.topic": "testusers",
        "quickstart": "users",
        "key.converter": "org.apache.kafka.connect.storage.StringConverter",
        "value.converter": "org.apache.kafka.connect.json.JsonConverter",
        "value.converter.schemas.enable": "false",
        "max.interval": 1000,
        "iterations": 10000000,
        "tasks.max": "1",
        "transforms": "InsertSource",
        "transforms.InsertSource.type": "org.apache.kafka.connect.transforms.InsertField$Value",
        "transforms.InsertSource.static.field": "data_source",
        "transforms.InsertSource.static.value": "test-file-source"
      }
    }
    > snctl kafka admin connect apply -f datagen.json --use-service-account
    ```

    You should see the following output:

    ```bash theme={null}
    Created connector test
    ```

    If you want to verify whether the data generator source connect has been created successfully, run the following command:

    ```bash theme={null}
    snctl kafka admin connect get connectors
    ```

    You should see the following output:

    ```bash theme={null}

     NAME                  TYPE     STATE        TASKS
     test                  source   RUNNING      0: RUNNING
    ```
  </Tab>

  <Tab title="kcctl">
    To create a data generator source connect named `test` with SMT, run the following command.

    ```bash theme={null}
    > cat datagen.json
    {
      "name": "test",
      "config": {
        "connector.class": "io.confluent.kafka.connect.datagen.DatagenConnector",
        "kafka.topic": "testusers",
        "quickstart": "users",
        "key.converter": "org.apache.kafka.connect.storage.StringConverter",
        "value.converter": "org.apache.kafka.connect.json.JsonConverter",
        "value.converter.schemas.enable": "false",
        "max.interval": 1000,
        "iterations": 10000000,
        "tasks.max": "1",
        "transforms": "InsertSource",
        "transforms.InsertSource.type": "org.apache.kafka.connect.transforms.InsertField$Value",
        "transforms.InsertSource.static.field": "data_source",
        "transforms.InsertSource.static.value": "test-file-source"
      }
    }
    > kcctl apply -f datagen.json
    ```

    You should see the following output:

    ```bash theme={null}
    Created connector test
    ```

    If you want to verify whether the data generator source connect has been created successfully, run the following command:

    ```bash theme={null}
    kcctl get connectors
    ```

    You should see the following output:

    ```bash theme={null}

     NAME                  TYPE     STATE        TASKS
     test                  source   RUNNING      0: RUNNING
    ```
  </Tab>

  <Tab title="Rest API">
    To create a data generator source connect named `test` with SMT, run the following command.

    ```bash theme={null}
    > cat datagen.json
    {
      "name": "test",
      "config": {
        "connector.class": "io.confluent.kafka.connect.datagen.DatagenConnector",
        "kafka.topic": "testusers",
        "quickstart": "users",
        "key.converter": "org.apache.kafka.connect.storage.StringConverter",
        "value.converter": "org.apache.kafka.connect.json.JsonConverter",
        "value.converter.schemas.enable": "false",
        "max.interval": 1000,
        "iterations": 10000000,
        "tasks.max": "1",
        "transforms": "InsertSource",
        "transforms.InsertSource.type": "org.apache.kafka.connect.transforms.InsertField$Value",
        "transforms.InsertSource.static.field": "data_source",
        "transforms.InsertSource.static.value": "test-file-source"
      }
    }
    > curl -X POST --header "Content-Type: application/json" "https://public%2Fdefault:${APIKEY}@${KAFKA-SERVICE-URL}/admin/kafkaconnect/connectors/" --data @datagen.json
    {"name":"test","config":{"connector.class":"io.confluent.kafka.connect.datagen.DatagenConnector","kafka.topic":"testusers","quickstart":"users","key.converter":"org.apache.kafka.connect.storage.StringConverter","value.converter":"org.apache.kafka.connect.json.JsonConverter","value.converter.schemas.enable":"false","max.interval":"1000","iterations":"10000000","tasks.max":"1","transforms":"InsertSource","transforms.InsertSource.type":"org.apache.kafka.connect.transforms.InsertField$Value","transforms.InsertSource.static.value":"test-file-source","name":"test"},"tasks":[],"type":"source"}
    ```

    If you want to list the submitted connect for a double check, run the following command:

    ```bash theme={null}
    > curl "https://public%2Fdefault:${APIKEY}@${KAFKA-SERVICE-URL}/admin/kafkaconnect/connectors/"

    ["test"]
    ```
  </Tab>

  <Tab title="Console">
    To create a data generator source connect named `test` with SMT, follow these steps:

    1. Login to the [StreamNative Cloud Console](https://cloud.streamnative.io/).
    2. In the left navigation pane, click **Connectors**, then click the **Kafka Sources** tab.
    3. Click the **Create Kafka Source** button:
       <img alt="image of creating kafka datagen source connect" />
    4. Fill in the required fields and optional fields as you wish
    5. Click the **Advance Settings** tab, and then fill in the SMT fields:
       <img alt="image of creating kafka datagen source connect with SMT" />
    6. Click the **Submit** button.

    If you want to verify whether the data generator source connect has been created successfully, go back to the **Connectors** page, and you should see the created connector in the **Kafka Sources** tab, like below:

    <img alt="image of kafka source connector list overview" />
  </Tab>
</Tabs>

## Create kafka connect with secret

Some connects require sensitive information, such as passwords, token, to be passed to the connector. And you may not want to expose these sensitive information in the connector configuration.
To solve this problem, you can use the following methods to pass sensitive information to the connector:

* **Create a secret**

  For example, the Milvus sink connector requires a token to be passed to the connector.
  You can create a secret in the console UI and pass the secret name to the connector configuration.

  <img alt="screenshot of creating secret" />

<Tip title="Tip">
  The `location` should be the same as the region of your Pulsar cluster.

  The `awsAccessKey` and `awsSecretKey` is the field name, and the `lambda-sink-secret` can be any unique name you want to give to the secret.

  For a Milvus sink, we should create a secret with a `token` field.
</Tip>

* **Pass secrets to the connector configuration**

  The following example shows how to create a Milvus sink connector named `test` on StreamNative Cloud using different tools.

<Tabs>
  <Tab title="snctl">
    To create a Milvus sink connector named `test`, run the following command.

    ```bash theme={null}
    > cat milvus.json
    {
      "name": "test",
      "config": {
        "connector.class": "com.milvus.io.kafka.MilvusSinkConnector",
        "public.endpoint": "http://dockerhost.default.svc.cluster.local:19530",
        "collection.name": "demo",
        "token": "${snsecret:miluvs-sec:token}",
        "topics": "kafka-milvus-input",
        "key.converter": "org.apache.kafka.connect.json.JsonConverter",
        "key.converter.schemas.enable": "false",
        "value.converter": "org.apache.kafka.connect.json.JsonConverter",
        "value.converter.schemas.enable": "false",
        "tasks.max": "1"
      }
    }
    > snctl kafka admin connect apply -f milvus.json --use-service-account
    ```

    <Tip title="Tip">
      The `miluvs-sec` is the name of the secret you created
    </Tip>

    You should see the following output:

    ```bash theme={null}
    Created connector test
    ```

    If you want to verify whether the data generator source connect has been created successfully, run the following command:

    ```bash theme={null}
    snctl kafka admin connect get connectors
    ```

    You should see the following output:

    ```bash theme={null}

    NAME                  TYPE     STATE        TASKS
    test                  sink     RUNNING      0: RUNNING
    ```
  </Tab>

  <Tab title="kcctl">
    To create a Milvus sink connector named `test`, run the following command.

    ```bash theme={null}
    > cat milvus.json
    {
      "name": "test",
      "config": {
        "connector.class": "com.milvus.io.kafka.MilvusSinkConnector",
        "public.endpoint": "http://dockerhost.default.svc.cluster.local:19530",
        "collection.name": "demo"
        "token": "${snsecret:miluvs-sec:token}",
        "topics": "kafka-milvus-input",
        "key.converter": "org.apache.kafka.connect.json.JsonConverter",
        "key.converter.schemas.enable": "false",
        "value.converter": "org.apache.kafka.connect.json.JsonConverter",
        "value.converter.schemas.enable": "false",
        "tasks.max": "1"
      }
    }
    > kcctl apply -f milvus.json
    ```

    <Tip title="Tip">
      The `miluvs-sec` is the name of the secret you created
    </Tip>

    You should see the following output:

    ```bash theme={null}
    Created connector test
    ```

    If you want to verify whether the data generator source connect has been created successfully, run the following command:

    ```bash theme={null}
    kcctl get connectors
    ```

    You should see the following output:

    ```bash theme={null}

    NAME                  TYPE     STATE        TASKS
    test                  sink     RUNNING      0: RUNNING
    ```
  </Tab>

  <Tab title="Rest API">
    To create a Milvus sink connector named `test`, run the following command.

    ```bash theme={null}
    > cat milvus.json
    {
      "name": "test",
      "config": {
        "connector.class": "com.milvus.io.kafka.MilvusSinkConnector",
        "public.endpoint": "http://dockerhost.default.svc.cluster.local:19530",
        "collection.name": "demo"
        "token": "${snsecret:miluvs-sec:token}",
        "topics": "kafka-milvus-input",
        "key.converter": "org.apache.kafka.connect.json.JsonConverter",
        "key.converter.schemas.enable": "false",
        "value.converter": "org.apache.kafka.connect.json.JsonConverter",
        "value.converter.schemas.enable": "false",
        "tasks.max": "1"
      }
    }
    > curl -X POST --header "Content-Type: application/json" "https://public%2Fdefault:${APIKEY}@${KAFKA-SERVICE-URL}/admin/kafkaconnect/connectors/" --data @miluvs.json
    {"name":"test","config":{"connector.class":"com.milvus.io.kafka.MilvusSinkConnector","topics":"kafka-milvus-input","public.endpoint":"http://dockerhost.default.svc.cluster.local:19530","collection.name":"demo","token":"${snsecret:miluvs-sec:token}","key.converter":"org.apache.kafka.connect.json.JsonConverte","key.converter.schemas.enable":"false","value.converter":"org.apache.kafka.connect.json.JsonConverter","value.converter.schemas.enable":"false","tasks.max":"1","name":"test"},"tasks":[],"type":"sink"}
    ```

    If you want to list the submitted connect for a double check, run the following command:

    ```bash theme={null}
    > curl "https://public%2Fdefault:${APIKEY}@${KAFKA-SERVICE-URL}/admin/kafkaconnect/connectors/"

    ["test"]
    ```
  </Tab>

  <Tab title="Console">
    To create a Milvus sink connect named `test`, follow these steps:

    1. Login to the [StreamNative Cloud Console](https://cloud.streamnative.io/).
    2. In the left navigation pane, click **Connectors**, then click the **Kafka Sinks** tab.
    3. Click the **Create Kafka Sink** button, and then choose the Milvus connect:
       <img alt="image of creating kafka milvus sink connect" />
    4. In the `Authentication Secrets` selection box, you can choose an existing secret or create a new secret.
    5. Fill in the required fields and optional fields as you wish, and then Click the **Submit** button.

    If you want to verify whether the data generator source connect has been created successfully, go back to the **Connectors** page, and you should see the created connector in the **Kafka Sources** tab, like below:

    <img alt="image of kafka source connect list overview" />
  </Tab>
</Tabs>

## Create a custom kafka connect

<Tip title="Tip">
  Before creating a kafka connect, it’s highly recommended to do the following:

  1. [Check kafka connect availability](/cloud/connect/kafka-connect/deploy-kafka-connectors/kafka-connect-check) to ensure the version number of the kafka connect you want to create is supported on StreamNative Cloud.
  2. Go to [StreamNative Hub](/connect/overview) and find the connect-specific docs of your version for configuration reference.
</Tip>

To create a custom kafka connect, you need to upload the connector jar/zip file to the StreamNative Cloud Package service first. Below are the steps:

### Upload your connector file to Pulsar

<Tabs>
  <Tab title="snctl">
    Upload packages

    ```bash theme={null}
    snctl pulsar admin packages upload function://public/default/mongo-connect-zip@v1 \
    --path /tmp/mongodb-kafka-connect-mongodb-1.12.0.zip \
    --description "mongodb kafka connect in zip format" \
    --properties fileName=mongo-kafka-connect.zip \
    --properties libDir=mongodb-kafka-connect-mongodb-1.12.0/lib
    ```

    You should see the following output:

    ```bash theme={null}
    The package 'function://public/default/mongo-connect-zip@v1' uploaded from path '/tmp/mongodb-kafka-connect-mongodb-1.12.0.zip' successfully
    ```

    <Tip title="Tip">
      the property `libDir` specifies the directory where the third-party libraries are located in the zip file.
    </Tip>
  </Tab>

  <Tab title="Pulsarctl">
    You need to set the context for Pulsarctl first:

    ```bash theme={null}
    # create a context
    pulsarctl context set ${context-name}  \
    --admin-service-url ${admin-service-url} \
    --issuer-endpoint ${issuerUrl} \
    --audience urn:sn:pulsar:${orgName}:${instanceName} \
    --key-file ${privateKey}

    # activate oauth2
    pulsarctl oauth2 activate
    ```

    <Note title="Note">
      Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).

      * `context-name`: any name you want
      * `admin-service-url`: the HTTP service URL of your Pulsar cluster.
      * `privateKey`: the path to the downloaded OAuth2 key file.
      * `issuerUrl`: the URL of the OAuth2 issuer.
      * `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#uniform-resource-name-urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
        * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
        * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
    </Note>

    Upload packages

    ```bash theme={null}
    pulsarctl packages upload function://public/default/mongo-connect-zip@v1 \
    --path /tmp/mongodb-kafka-connect-mongodb-1.12.0.zip \
    --description "mongodb kafka connect in zip format" \
    --properties fileName=mongo-kafka-connect.zip \
    --properties libDir=mongodb-kafka-connect-mongodb-1.12.0/lib
    ```

    You should see the following output:

    ```bash theme={null}
    The package 'function://public/default/mongo-connect-zip@v1' uploaded from path '/tmp/mongodb-kafka-connect-mongodb-1.12.0.zip' successfully
    ```

    <Tip title="Tip">
      the property `libDir` specifies the directory where the third-party libraries are located in the zip file.
    </Tip>
  </Tab>

  <Tab title="Pulsar-admin">
    ```bash theme={null}
    ./bin/pulsar-admin \
        --admin-url "${WEB_SERVICE_URL}" \
        --auth-plugin org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 \
        --auth-params '{"privateKey":"file://${privateKey}","issuerUrl":"${issuerUrl}","audience":"urn:sn:pulsar:${orgName}:${instanceName}}' \
        packages upload function://public/default/mongo-connect-zip@v1 \
        --path /tmp/mongodb-kafka-connect-mongodb-1.12.0.zip \
        --description "mongodb kafka connect in zip format" \
        --properties fileName=mongo-kafka-connect.zip \
        --properties libDir=mongodb-kafka-connect-mongodb-1.12.0/lib
    ```

    <Note title="Note">
      Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).

      * `admin-url`: the HTTP service URL of your Pulsar cluster.
      * `privateKey`: the path to the downloaded OAuth2 key file.
      * `issuerUrl`: the URL of the OAuth2 issuer.
      * `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#uniform-resource-name-urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
        * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
        * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
    </Note>

    You should see the following output:

    ```bash theme={null}
    The package 'function://public/default/mongo-connect-zip@v1' uploaded from path '/tmp/mongodb-kafka-connect-mongodb-1.12.0.zip' successfully
    ```

    <Tip title="Tip">
      the property `libDir` specifies the directory where the third-party libraries are located in the zip file.
    </Tip>
  </Tab>
</Tabs>

The following example shows how to create a custom mongodb source connect named `mongo-source` on StreamNative Cloud using different tools.

<Tabs>
  <Tab title="snctl">
    To create a custom mongodb source connect named `mongo-source`, run the following command.

    ```bash theme={null}
    > cat mongo.json
    {
      "name": "mongo-source",
      "config": {
        "connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",
        "connection.uri": "mongodb://mongo.default.svc.cluster.local:27017/?authSource=admin",
        "database": "kafka-mongo",
        "collection": "source",
        "key.converter.schemas.enable": false,
        "value.converter.schemas.enable": false,
        "sn.pulsar.package.url": "function://public/default/mongo-connect-zip@v1",
        "tasks.max": "1"
      }
    }
    > snctl kafka admin connect apply -f mongo.json --use-service-account
    ```

    <Tip title="Tip">
      The `sn.pulsar.package.url` is the package url you uploaded to the StreamNative Cloud Package service.
    </Tip>

    You should see the following output:

    ```bash theme={null}
    Created connector mongo-source
    ```

    If you want to verify whether the data generator source connect has been created successfully, run the following command:

    ```bash theme={null}
    snctl kafka admin connect get connectors
    ```

    You should see the following output:

    ```bash theme={null}

     NAME                  TYPE     STATE        TASKS
     mongo-source          source   RUNNING      0: RUNNING
    ```
  </Tab>

  <Tab title="kcctl">
    To create a custom mongodb source connect named `mongo-source`, run the following command.

    ```bash theme={null}
    > cat mongo.json
    {
      "name": "mongo-source",
      "config": {
        "connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",
        "connection.uri": "mongodb://mongo.default.svc.cluster.local:27017/?authSource=admin",
        "database": "kafka-mongo",
        "collection": "source",
        "key.converter.schemas.enable": false,
        "value.converter.schemas.enable": false,
        "sn.pulsar.package.url": "function://public/default/mongo-connect-zip@v1",
        "tasks.max": "1"
      }
    }
    > kcctl apply -f mongo.json
    ```

    <Tip title="Tip">
      The `sn.pulsar.package.url` is the package url you uploaded to the StreamNative Cloud Package service.
    </Tip>

    You should see the following output:

    ```bash theme={null}
    Created connector mongo-source
    ```

    If you want to verify whether the data generator source connect has been created successfully, run the following command:

    ```bash theme={null}
    kcctl get connectors
    ```

    You should see the following output:

    ```bash theme={null}

     NAME                  TYPE     STATE        TASKS
     mongo-source          source   RUNNING      0: RUNNING
    ```
  </Tab>

  <Tab title="Rest API">
    To create a custom mongodb source connect named `mongo-source`, run the following command.

    ```bash theme={null}
    > cat mongo.json
    {
      "name": "mongo-source",
      "config": {
        "connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",
        "connection.uri": "mongodb://mongo.default.svc.cluster.local:27017/?authSource=admin",
        "database": "kafka-mongo",
        "collection": "source",
        "key.converter.schemas.enable": false,
        "value.converter.schemas.enable": false,
        "sn.pulsar.package.url": "function://public/default/mongo-connect-zip@v1",
        "tasks.max": "1"
      }
    }
    > curl -X POST --header "Content-Type: application/json" "https://public%2Fdefault:${APIKEY}@${KAFKA-SERVICE-URL}/admin/kafkaconnect/connectors/" --data @mongo.json
    {"name":"mongo-source","config":{"connector.class":"com.mongodb.kafka.connect.MongoSourceConnector","connection.uri":"mongodb://mongo.default.svc.cluster.local:27017/?authSource=admin","database":"kafka-mongo","collection":"source","key.converter.schemas.enable":false,"value.converter.schemas.enable":false,"sn.pulsar.package.url":"function://public/default/mongo-connect-zip","tasks.max":"1"}}
    ```

    If you want to list the submitted connect for a double check, run the following command:

    ```bash theme={null}
    > curl "https://public%2Fdefault:${APIKEY}@${KAFKA-SERVICE-URL}/admin/kafkaconnect/connectors/"

    ["mongo-source"]
    ```
  </Tab>
</Tabs>

## Set resources for kafka connect

You can use below two configs to set the resources for the kafka connect to control the CPU and memory usage of the connector:

* `sn.cpu`: The number of CPU cores to allocate to the connector, default to **0.5**.
* `sn.memory`: The bytes of memory to allocate to the connector, default to **2147483648** (2G).

<Note title="Note">
  You need to upgrade your Pulsar cluster to `v3.0.8.4+`, `v3.3.3.4+` or `v4.0.1.3+` to use the `sn.cpu` and `sn.memory` configs.
</Note>

## Tune Kafka Connect clients

You can tune the Kafka client behavior for an individual connector by adding connector-level override settings. These settings are useful when one connector needs different throughput, latency, or retry behavior than the default Kafka Connect worker settings.

Use the following prefixes in the connector configuration:

* `producer.override.*`: Overrides Kafka producer settings. Use this prefix mainly for source connectors that write records to Kafka topics. Common settings include `compression.type`, `batch.size`, `linger.ms`, and `acks`.
* `consumer.override.*`: Overrides Kafka consumer settings. Use this prefix for sink connectors that read records from Kafka topics. Common settings include `max.poll.records`, `fetch.min.bytes`, `fetch.max.wait.ms`, and `auto.offset.reset`.
* `admin.override.*`: Overrides Kafka administrative client settings. Use this prefix when the connector needs different administrative client behavior, such as for topic creation or dead-letter queue operations. Common settings include `request.timeout.ms`, `retry.backoff.ms`, and `default.api.timeout.ms`.

For example, the following sink connector configuration increases the consumer batch size and adjusts admin client timeouts:

```json theme={null}
{
  "name": "jdbc-sink",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
    "topics": "orders",
    "tasks.max": "1",
    "consumer.override.max.poll.records": "1000",
    "consumer.override.fetch.min.bytes": "1048576",
    "consumer.override.fetch.max.wait.ms": "500",
    "admin.override.request.timeout.ms": "30000",
    "admin.override.retry.backoff.ms": "500"
  }
}
```

The following source connector example tunes the producer used to write records to Kafka:

```json theme={null}
{
  "name": "mongo-source",
  "config": {
    "connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",
    "tasks.max": "1",
    "producer.override.compression.type": "gzip",
    "producer.override.batch.size": "131072",
    "producer.override.linger.ms": "20",
    "producer.override.acks": "all"
  }
}
```

## Schema Registry Support

Kafka Connect supports using schema registry to save Avro/Protobuf/Json schema for the value and key. And StreamNative has an internal schema registry which can be used without complex configurations.
To use it, you just need to set the following configuration in the connector configuration:

* `value.converter.schema.registry.internal: true`: if you want to use the internal schema registry for the value converter.
* `key.converter.schema.registry.internal: true`: if you want to use the internal schema registry for the key converter.


# Set up Your Environment
Source: https://docs.streamnative.io/cloud/connect/kafka-connect/deploy-kafka-connectors/kafka-connect-setup



This section introduces how to set up a new service account to run kafka connectors. To perform the following operations, you need to be the cluster administrator beforehand.

## Create a service account for Pulsar users

1. On the left navigation pane of StreamNative Cloud Console, click **Service Accounts**.

2. Click **Create Service Account**.

3. Enter a name for the service account, and then click **Confirm**.

<Note title="Note">
  You may see the `Role xxxx cannot access topic public/__kafka_connect/__kafka_connect_offset_storage` exception when you create a connector, this is a known issue which uses a wrong namespace `public/__kafka_connect` to create the offset storage topic,
  after v3.3.1.5, this issue has been fixed, and will use namespace `public/default` instead.

  You can create a **Super Admin** service account or create the `public/__kafka_connect` namespace and grant `produce` permission to this service account you created as the workaround.
</Note>

## Authorize the service account

To make the service account work, you need to make the service account granted with proper permissions (`packages`, `produce`, and `consume`).

To grant the service account permissions on the namespace level, follow these steps:

<Tabs>
  <Tab title="Cloud Console">
    1. Navigate to the **Namespace Dashboard** page by [switching to the namespace workspace](/cloud/get-started/cloud-console#switch-a-namespace).

    2. On the **Namespace Dashboard** page, click **Configuration** on the left navigation pane.

    3. On the **Namespace configuration** page, click **ADD ROLE**, and select the service account that you want to authorize.

    4. On the drop-down menu below the service account, select the proper permissions to assign to the service account. There are six permissions in total:

    * `consume`: allow the service account to consume messages.
    * `produce`: allow the service account to publish messages.
    * `functions`: allow the service account to submit and manage functions.
    * `sinks`: allow the service account to create and manage sink connectors.
    * `sources`: allow the service account to create and manage source connectors.
    * `packages`: allow the service account to upload and manage pulsar packages. If you want to submit a customized function/connector, then you will need to upload the function/connector’s JAR/NAR/Python file first, which requires the `packages` permission.

      <img alt="Authorize Service Account" />
  </Tab>
</Tabs>

## Grant access to the service account

To grant the underlying infrastructure with access to the newly created service account's OAuth2 key file, you need to create a service account binding via UI.

Go to the `Service Accounts` tab and choose the service account you want to use for running the connector. Clicking on the right button and there will be a `Edit service account bindings` option.

<img alt="Binding Service Account step-1" />

Click the `Edit service account bindings`, choose the desired pool member and confirm.

<img alt="Binding Service Account step-2" />

You can also enable the `Enable IAM Role Creation` option to create a separate IAM role for the service account.

Now your connector is ready to use the service account in StreamNative environments.

## (Optional) Create a separate IAM role for the service account

StreamNative's I/O components (Pulsar Functions, Pulsar Connectors, and Kafka Connectors) run as cloud-native workloads on AWS, GCP, and Azure infrastructures.

Use the cloud providers' native IAM (Identity and Access Management) services to control access to infrastructure resources such as S3, GCS, and Azure Blob Storage. This removes the need for password-based authentication for the service accounts that run connectors in StreamNative Cloud.

By default, StreamNative uses a single service account per `PulsarCluster` for all I/O components (Pulsar Functions, Pulsar Connectors, and Kafka Connectors) to access underlying infrastructure resources. This means all I/O Components in the same cluster share one service account and the same permissions.

<Warning>
  Using a single service account for all I/O components means all functions and connectors share the same permissions. If one component is compromised, it could access resources intended for other components.

  So it's better to leave the default service account with no permissions and use it for running IO components that do not require access to external resources.

  And create separate service accounts with the minimum required permissions for each IO component that need to access external resources.
</Warning>

To enhance security and improve isolation, you can create a separate IAM role for each service account used to run connectors. This let you grant only the permissions each service account needs.

Create a separate IAM role for your service account:

<Tabs>
  <Tab title="Snctl">
    <Note title="Note">
      This feature is available in [snctl](/tools/cli/snctl/snctl-overview) v1.3.0 or later.
    </Note>

    1. Get the `PoolMember` name and namespace from the `PulsarCluster`

       ```shell theme={null}
       snctl get pulsarcluster <cluster-id> -o yaml
       ```

       In the output, find the `poolMemberRef` block, which looks like:

       ```yaml theme={null}
       poolMemberRef:
         name: <pool-member-name>
         namespace: <namespace>
       ```

       <Note title="Note">
         Multiple clusters may be located in the same `PoolMember`. You do not need to create separate IAM roles for each cluster within the same `PoolMember`.
       </Note>

    2. Create a new `ServiceAccountBinding` that binds the service account to the `PoolMember`

       ```shell theme={null}
       snctl create serviceaccountbinding <binding-name> \
         --pool-member <pool-member-namespace>/<pool-member-name> \
         --enable-iam-account-creation \
         --service-account-name <service-account-name>
       ```

       **AWS only:** You can specify one or more `AWS Assume Role ARNs` that can be assumed by the IAM role created for the service account (repeat the flag for each ARN):

       ```shell theme={null}
       snctl create serviceaccountbinding <binding-name> \
         --pool-member <pool-member-namespace>/<pool-member-name> \
         --enable-iam-account-creation \
         --service-account-name <service-account-name> \
         --aws-assume-role-arns <arn1> \
         --aws-assume-role-arns <arn2>
       ```

       <Note title="Note">
         The IAM role created by StreamNative will include permissions to call `sts:AssumeRole` on `arn1` and `arn2`. You must still add a trust policy on `arn1` and `arn2` to allow the newly created role to assume them, an example likes below:

         ```json theme={null}
         {
             "Version": "2012-10-17",
             "Statement": [
                 {
                     "Effect": "Allow",
                     "Principal": {
                         "AWS": "arn:aws:iam::[aws-account-id]:role/StreamNative/sncloud-role/authorization.streamnative.io/iamaccounts/IamAccount-[org-id]-sab-[binding-name]"
                     },
                     "Action": "sts:AssumeRole"
                 }
             ]
         }
         ```
       </Note>

    3. (Optional) Update an existing `ServiceAccountBinding` to create the IAM role

       ```shell theme={null}
       snctl edit serviceaccountbinding <binding-name>
       ```

    4. Verify the IAM role was created successfully

       ```shell theme={null}
       snctl get serviceaccountbinding <binding-name> -o yaml
       ```

       Expected output (example):

       ```yaml theme={null}
       apiVersion: cloud.streamnative.io/v1alpha1
       kind: ServiceAccountBinding
       metadata:
         creationTimestamp: "2025-06-25T08:45:35Z"
         finalizers:
         - serviceaccountbinding.finalizers.cloud.streamnative.io
         generation: 1
         name: test-admin
         namespace: o-lftqu
         ownerReferences:
         - apiVersion: cloud.streamnative.io/v1alpha1
           kind: ServiceAccount
           name: admin
           uid: 4ef639aa-6278-4863-9a23-f1da50cea448
         resourceVersion: "54707713"
         uid: 918d40ad-551d-416b-a2ae-d41548d6608e
       spec:
         enableIamRoleCreation: true
         poolMemberRef:
           name: azure-eastus-zephyr
           namespace: streamnative
         serviceAccountName: admin
       status:
         conditions:
         - lastTransitionTime: "2025-06-25T08:45:35Z"
           status: "True"
           type: IAMAccountReady
         - lastTransitionTime: "2025-06-25T08:45:35Z"
           status: "True"
           type: ServiceAccountReady
         - lastTransitionTime: "2025-07-16T13:56:17Z"
           status: "True"
           type: ResourceExists
         - lastTransitionTime: "2025-07-16T13:56:17Z"
           status: "True"
           type: PoolMemberReady
         - lastTransitionTime: "2025-07-16T13:56:17Z"
           reason: AllConditionStatusTrue
           status: "True"
           type: Ready
       ```

       In the output, the `status.conditions` array should include a condition with `type: IAMAccountReady` and `status: "True"`, indicating the IAM role was created successfully.<br /><br />

    5. Use the service account when creating I/O components

       <br /><br />You can now select this service account in Console (or use its API key with the CLI) when creating I/O components (Pulsar Functions, Pulsar Connectors, and Kafka Connectors). The components will inherit the permissions granted to the IAM role created in the previous step.
  </Tab>

  <Tab title="Console">
    You can also create a separate IAM role for your service account via StreamNative Cloud Console. Just enable the `Enable IAM Role Creation` option when creating or editing a service account binding.

    <img alt="Binding Service Account step-2" />

    **AWS only:** You can specify one or more `AWS Assume Role ARNs` that can be assumed by the IAM role created for the service account (use one line for each ARN)

    <img alt="Binding Service Account step-3" />

    <Note title="Note">
      The IAM role created by StreamNative will include permissions to call `sts:AssumeRole` on `arn1` and `arn2`. You must still add a trust policy on `arn1` and `arn2` to allow the newly created role to assume them, an example likes below:

      ```json theme={null}
      {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Principal": {
                      "AWS": "arn:aws:iam::[aws-account-id]:role/StreamNative/sncloud-role/authorization.streamnative.io/iamaccounts/IamAccount-[org-id]-sab-[binding-name]"
                  },
                  "Action": "sts:AssumeRole"
              }
          ]
      }
      ```
    </Note>
  </Tab>
</Tabs>

<Note title="Note">
  In Azure, a Managed Identity `sab-[binding-name]-[org-id]` is created;

  In AWS, an IAM role `role/StreamNative/sncloud-role/authorization.streamnative.io/iamaccounts/IamAccount-[org-id]-sab-[binding-name]` is created;

  In GCP, a service account with display name: `IamAccount/[org-id]/sab-[binding-name]` is created.
</Note>

## Set up client tools

### Using StreamNative Cloud CLI tool `snctl`

Starting from v1.0.0 of the StreamNative Cloud CLI tool `snctl`, it supports to manage Kafka Connectors running on StreamNative Cloud.

Follow the steps below to set up `snctl`:

1. Use `snctl config set --organization $ORG` to set your StreamNative Cloud organization.
2. Use `snctl context use` to select your target StreamNative Cloud cluster interactively.
3. To send Kafka Connect admin requests as a service account, use `snctl kafka admin connect --as-service-account $SERVICE_ACCOUNT_NAME ...` or `snctl kafka admin connect --use-service-account ...`.
4. To run a Kafka connector as a service account, use `--sn-service-account $SERVICE_ACCOUNT_NAME` on `snctl kafka admin connect apply` or `snctl kafka admin connect patch connector`. To select the runtime service account interactively, use `--use-sn-service-account`.
5. Verify the setup using `snctl kafka admin connect info`, it should print something like below.

   ```shell theme={null}
   > snctl kafka admin connect info
   URL:               https://${KAFKA-SERVICE-URL}/admin/kafkaconnect/
   Version:           3.7.0
   Commit:            839b886f9b732b15
   Kafka Cluster ID:  connect
   ```

<Note>
  `--as-service-account` and `--sn-service-account` set different identities.

  Use `--as-service-account` to send the request with the specified service account's credentials. That same service account also becomes the runtime identity of the connector. Therefore, the service account must have permissions to create or update connectors, download packages, and produce or consume messages.

  Use `--sn-service-account` to keep the request authenticated as the current caller, but run the connector with the specified service account as its runtime identity. In this case, the caller must have permission to create or update the connector and use the selected service account (see [account-admin](/cloud/security/access/rbac/manage-rbac-roles#account-admin)). The runtime service account only needs the permissions required by the connector itself, such as producing or consuming messages and downloading packages.

  You can use `--as-service-account` on Pulsar clusters of any supported version. The `--sn-service-account` and `--use-sn-service-account` flags require Pulsar 4.0.x version 4.0.10.6 or later, or Pulsar 4.2.x version 4.2.1.4 or later.
</Note>

### Using kcctl

We have tested with the [kcctl](https://github.com/kcctl/kcctl) to manage Kafka Connectors, you can also try other CLI tools you like.

Follow the steps below to set up it:

1. Create an apikey from the service account you created.

   You can follow the instructions to [create an API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#create-an-cluster-api-key) for the service account you choose to use.

2. Set up the kcctl with the apikey.

   ```shell theme={null}
   kcctl config set-context --bootstrap-servers ${KAFKA-SERVICE-URL}:9093 --cluster=https://${KAFKA-SERVICE-URL}/admin/kafkaconnect/ --username public/default --password "token:${APIKEY}" ${NAME}
   kcctl config use-context ${NAME}
   ```

<Note title="Note">
  The KAFKA-SERVICE-URL is the endpoint of the Kafka service, you can find it in the StreamNative Cloud Console.
</Note>

3. Verify the setup using `kcctl info`, it should print something like below.

   ```shell theme={null}
   > kcctl info
   URL:               https://${KAFKA-SERVICE-URL}/admin/kafkaconnect/
   Version:           3.7.0
   Commit:            839b886f9b732b15
   Kafka Cluster ID:  connect
   ```

## What’s next?

* [Check kafka connect availability](/cloud/connect/kafka-connect/deploy-kafka-connectors/kafka-connect-check)

* [Create kafka connectors](/cloud/connect/kafka-connect/deploy-kafka-connectors/kafka-connect-create)


# Manage Kafka Connectors
Source: https://docs.streamnative.io/cloud/connect/kafka-connect/kafka-connect-manage



StreamNative Cloud enables you to manage Pulsar IO Connectors by using a variety of tools, including `snctl`, `kcctl`, `restful-api`, and `Console`.

## Update a connect

When you want to modify configurations or update resources for connects, you can update connectors using multiple tools.

The following example shows how to update the tasks.max of the data generator source connector `test` to `2` using different tools.

<Tabs>
  <Tab title="snctl">
    ```bash theme={null}
    > cat datagen.json
    {
      "name": "test",
      "config": {
        "connector.class": "io.confluent.kafka.connect.datagen.DatagenConnector",
        "kafka.topic": "testusers",
        "quickstart": "users",
        "key.converter": "org.apache.kafka.connect.storage.StringConverter",
        "value.converter": "org.apache.kafka.connect.json.JsonConverter",
        "value.converter.schemas.enable": "false",
        "max.interval": 1000,
        "iterations": 10000000,
        "tasks.max": "2"
      }
    }
    > snctl kafka admin connect apply -f datagen.json
    ```

    You should see the following output:

    ```bash theme={null}
    Updated connector test
    ```

    And you can further check the status:

    ```bash theme={null}
    snctl kafka admin connect describe connector test

    Name:       test
    Type:       source
    State:      RUNNING
    Worker ID:  10.80.161.255:8083
    Config:
      connector.class:                 io.confluent.kafka.connect.datagen.DatagenConnector
      iterations:                      10000000
      kafka.topic:                     testusers
      key.converter:                   org.apache.kafka.connect.storage.StringConverter
      max.interval:                    1000
      name:                            test
      quickstart:                      users
      tasks.max:                       2
      value.converter:                 org.apache.kafka.connect.json.JsonConverter
      value.converter.schemas.enable:  false
    Tasks:
      0:
        State:                             RUNNING
        Worker ID:                         10.80.161.255:8083
        Config:
          connector.class:                 io.confluent.kafka.connect.datagen.DatagenConnector
          quickstart:                      users
          tasks.max:                       2
          max.interval:                    1000
          iterations:                      10000000
          task.class:                      io.confluent.kafka.connect.datagen.DatagenTask
          name:                            test
          value.converter.schemas.enable:  false
          kafka.topic:                     testusers
          task.id:                         0
          value.converter:                 org.apache.kafka.connect.json.JsonConverter
          key.converter:                   org.apache.kafka.connect.storage.StringConverter
      1:
        State:                             RUNNING
        Worker ID:                         10.80.160.255:8083
        Config:
          connector.class:                 io.confluent.kafka.connect.datagen.DatagenConnector
          quickstart:                      users
          tasks.max:                       2
          max.interval:                    1000
          iterations:                      10000000
          task.class:                      io.confluent.kafka.connect.datagen.DatagenTask
          name:                            test
          value.converter.schemas.enable:  false
          kafka.topic:                     testusers
          task.id:                         0
          value.converter:                 org.apache.kafka.connect.json.JsonConverter
          key.converter:                   org.apache.kafka.connect.storage.StringConverter
    Topics:
      testusers
    ```
  </Tab>

  <Tab title="kcctl">
    ```bash theme={null}
    > cat datagen.json
    {
      "name": "test",
      "config": {
        "connector.class": "io.confluent.kafka.connect.datagen.DatagenConnector",
        "kafka.topic": "testusers",
        "quickstart": "users",
        "key.converter": "org.apache.kafka.connect.storage.StringConverter",
        "value.converter": "org.apache.kafka.connect.json.JsonConverter",
        "value.converter.schemas.enable": "false",
        "max.interval": 1000,
        "iterations": 10000000,
        "tasks.max": "2"
      }
    }
    > kcctl apply -f datagen.json
    ```

    You should see the following output:

    ```bash theme={null}
    Updated connector test
    ```

    And you can further check the status:

    ```bash theme={null}
    kcctl describe connectors test --tasks-config

    Name:       test
    Type:       source
    State:      RUNNING
    Worker ID:  10.80.161.255:8083
    Config:
      connector.class:                 io.confluent.kafka.connect.datagen.DatagenConnector
      iterations:                      10000000
      kafka.topic:                     testusers
      key.converter:                   org.apache.kafka.connect.storage.StringConverter
      max.interval:                    1000
      name:                            test
      quickstart:                      users
      tasks.max:                       2
      value.converter:                 org.apache.kafka.connect.json.JsonConverter
      value.converter.schemas.enable:  false
    Tasks:
      0:
        State:                             RUNNING
        Worker ID:                         10.80.161.255:8083
        Config:
          connector.class:                 io.confluent.kafka.connect.datagen.DatagenConnector
          quickstart:                      users
          tasks.max:                       2
          max.interval:                    1000
          iterations:                      10000000
          task.class:                      io.confluent.kafka.connect.datagen.DatagenTask
          name:                            test
          value.converter.schemas.enable:  false
          kafka.topic:                     testusers
          task.id:                         0
          value.converter:                 org.apache.kafka.connect.json.JsonConverter
          key.converter:                   org.apache.kafka.connect.storage.StringConverter
      1:
        State:                             RUNNING
        Worker ID:                         10.80.160.255:8083
        Config:
          connector.class:                 io.confluent.kafka.connect.datagen.DatagenConnector
          quickstart:                      users
          tasks.max:                       2
          max.interval:                    1000
          iterations:                      10000000
          task.class:                      io.confluent.kafka.connect.datagen.DatagenTask
          name:                            test
          value.converter.schemas.enable:  false
          kafka.topic:                     testusers
          task.id:                         0
          value.converter:                 org.apache.kafka.connect.json.JsonConverter
          key.converter:                   org.apache.kafka.connect.storage.StringConverter
    Topics:
      testusers
    ```
  </Tab>

  <Tab title="Rest API">
    ```bash theme={null}
    > cat datagen.json
    {
      "name": "test",
      "config": {
        "connector.class": "io.confluent.kafka.connect.datagen.DatagenConnector",
        "kafka.topic": "testusers",
        "quickstart": "users",
        "key.converter": "org.apache.kafka.connect.storage.StringConverter",
        "value.converter": "org.apache.kafka.connect.json.JsonConverter",
        "value.converter.schemas.enable": "false",
        "max.interval": 1000,
        "iterations": 10000000,
        "tasks.max": "2"
      }
    }
    > curl -X PUT --header "Content-Type: application/json" "https://public%2Fdefault:${APIKEY}@${KAFKA-SERVICE-URL}/admin/kafkaconnect/connectors/test/config" --data @datagen.json
    ```

    If no error is response, the update is successful.

    And you can further check the status:

    ```bash theme={null}
    > curl "https://public%2Fdefault:${APIKEY}@${KAFKA-SERVICE-URL}/admin/kafkaconnect/connectors/test | jq '.'"
    {
      "name": "datagen-test-offset",
      "config": {
        "connector.class": "io.confluent.kafka.connect.datagen.DatagenConnector",
        "iterations": "10000000",
        "kafka.topic": "testusers",
        "key.converter": "org.apache.kafka.connect.storage.StringConverter",
        "max.interval": "1000",
        "name": "datagen-test-offset",
        "quickstart": "users",
        "tasks.max": "2",
        "value.converter": "org.apache.kafka.connect.json.JsonConverter",
        "value.converter.schemas.enable": "false"
      },
      "tasks": [
        {
          "connector": "datagen-test-offset",
          "task": 0
        },
        {
          "connector": "datagen-test-offset",
          "task": 1
        }
      ],
      "type": "source"
    }
    ```
  </Tab>

  <Tab title="Console">
    1. On the left navigation pane of StreamNative Console, under **Resources**, click **Connectors**.

    2. On the Connectors page, select the **Kafka Sources** or **Kafka Sinks** tab.

    3. Click the ellipsis at the end of the row of the connector, and then click **Edit**.

    4. Edit the configuration that you want to change, and click **SUBMIT**.
  </Tab>
</Tabs>

## Delete a connector

The following example shows how to delete the data generator source connector `test` using different tools.

<Tabs>
  <Tab title="snctl">
    To delete the source connector `test`, use the following command.

    ```bash theme={null}
    snctl kafka admin connect delete connector test
    ```

    You should see the following output:

    ```bash theme={null}
    Deleted connector test
    ```

    If you want to verify whether the source connector has been deleted successfully, run the following command.

    ```bash theme={null}
    snctl kafka admin connect get connectors
    ```

    You should see the following output:

    ```bash theme={null}

     NAME                  TYPE     STATE        TASKS

    ```
  </Tab>

  <Tab title="kcctl">
    To delete the source connector `test`, use the following command.

    ```bash theme={null}
    kcctl delete connector test
    ```

    You should see the following output:

    ```bash theme={null}
    Deleted connector test
    ```

    If you want to verify whether the source connector has been deleted successfully, run the following command.

    ```bash theme={null}
    kcctl get connectors
    ```

    You should see the following output:

    ```bash theme={null}

     NAME                  TYPE     STATE        TASKS

    ```
  </Tab>

  <Tab title="Rest API">
    To delete the source connector `test`, use the following command.

    ```bash theme={null}
    > curl -X DELETE "https://public%2Fdefault:${APIKEY}@${KAFKA-SERVICE-URL}/admin/kafkaconnect/connectors/test"
    ```

    If no error is response, the delete is successful.

    If you want to verify whether the source connector has been deleted successfully, run the following command.

    ```bash theme={null}
    curl -X GET "https://public%2Fdefault:${APIKEY}@${KAFKA-SERVICE-URL}/admin/kafkaconnect/connectors/test"
    ```

    You should see the following output:

    ```bash theme={null}
    {"reason":"This resource doesn't exist, please check the name"}
    ```
  </Tab>

  <Tab title="Console">
    1. On the left navigation pane of StreamNative Console, under **Resources**, click **Connectors**.

    2. On the Connectors page, select the **Kafka Sources** or **Kafka Sinks** tab.

    3. Click the ellipsis at the end of the row of the connector, and then click **Delete**.

    4. Enter the connector name and then click **Confirm**.
  </Tab>
</Tabs>

## Stop a connector

You can stop a running connector using different tools, after stopped, all resources of the connector will be released.

After stopped, you can restart the connector.

<Tabs>
  <Tab title="snctl">
    To stop the source connector `test`, use the following command.

    ```bash theme={null}
    snctl kafka admin connect stop test
    ```

    You should see the following output:

    ```bash theme={null}
    Stopped connector test
    ```

    If you want to verify whether the source connector has been stopped successfully, run the following command.

    ```bash theme={null}
    snctl kafka admin connect get connectors
    ```

    You should see the following output:

    ```bash theme={null}
     NAME      TYPE     STATE     TASKS
     datagen   source   STOPPED
    ```
  </Tab>

  <Tab title="kcctl">
    To stop the source connector `test`, use the following command.

    ```bash theme={null}
    kcctl stop connectors test
    ```

    You should see the following output:

    ```bash theme={null}
    Stopped connector test
    ```

    If you want to verify whether the source connector has been stopped successfully, run the following command.

    ```bash theme={null}
    kcctl get connectors
    ```

    You should see the following output:

    ```bash theme={null}
     NAME      TYPE     STATE     TASKS
     datagen   source   STOPPED
    ```
  </Tab>

  <Tab title="Rest API">
    To stop the source connector `test`, use the following command.

    ```bash theme={null}
    > curl -X PUT --header 'content-type: application/json' "https://public%2Fdefault:${APIKEY}@${KAFKA-SERVICE-URL}/admin/kafkaconnect/connectors/test/stop"
    ```

    If no error is response, the stop is successful.

    If you want to verify whether the source connector has been stopped successfully, run the following command.

    ```bash theme={null}
    curl -X GET "https://public%2Fdefault:${APIKEY}@${KAFKA-SERVICE-URL}/admin/kafkaconnect/connectors/test/status | jq '.'"
    ```

    You should see the following output:

    ```bash theme={null}
    {
      "name": "test",
      "connector": {
        "state": "STOPPED",
        "worker_id": "10.80.161.255:8083",
        "trace": "Kafka Connector Resource stopped"
      },
      "tasks": [],
      "type": "source"
    }
    ```
  </Tab>

  <Tab title="Console">
    1. On the left navigation pane of StreamNative Console, under **Resources**, click **Connectors**.

    2. On the Connectors page, select the **Kafka Sources** or **Kafka Sinks** tab.

    3. Click the ellipsis at the end of the row of the connector, and then click **Stop**.
  </Tab>
</Tabs>

## Restart a connector

You can restart a stopped or a running connector using different tools.

<Tabs>
  <Tab title="snctl">
    To restart the source connector `test`, use the following command.

    ```bash theme={null}
    snctl kafka admin connect restart connector test
    ```

    You should see the following output:

    ```bash theme={null}
    Restarted connector test
    ```

    If you want to verify whether the source connector has been restarted successfully, run the following command.

    ```bash theme={null}
    snctl kafka admin connect get connectors
    ```

    You should see the following output:

    ```bash theme={null}
     NAME      TYPE     STATE     TASKS
     datagen   source   RUNNING
    ```
  </Tab>

  <Tab title="kcctl">
    To restart the source connector `test`, use the following command.

    ```bash theme={null}
    kcctl restart connectors test
    ```

    You should see the following output:

    ```bash theme={null}
    Restarted connector test
    ```

    If you want to verify whether the source connector has been restarted successfully, run the following command.

    ```bash theme={null}
    kcctl get connectors
    ```

    You should see the following output:

    ```bash theme={null}
     NAME      TYPE     STATE     TASKS
     datagen   source   RUNNING
    ```
  </Tab>

  <Tab title="Rest API">
    To restart the source connector `test`, use the following command.

    ```bash theme={null}
    > curl -X POST "https://public%2Fdefault:${APIKEY}@${KAFKA-SERVICE-URL}/admin/kafkaconnect/connectors/test/restart"
    ```

    If no error is response, the restart is successful.

    If you want to verify whether the source connector has been restarted successfully, run the following command.

    ```bash theme={null}
    curl -X GET "https://public%2Fdefault:${APIKEY}@${KAFKA-SERVICE-URL}/admin/kafkaconnect/connectors/test/status | jq '.'"
    ```

    You should see the following output:

    ```bash theme={null}
    {
      "name": "test",
      "connector": {
        "state": "RUNNING",
        "worker_id": "10.80.161.255:8083",
        "trace": "",
      },
      "tasks": [
        {
          "id": 0,
          "state": "RUNNING",
          "worker_id": "10.244.0.75:8083"
        }
      ],
      "type": "source"
    }
    ```
  </Tab>

  <Tab title="Console">
    1. On the left navigation pane of StreamNative Console, under **Resources**, click **Connectors**.

    2. On the Connectors page, select the **Kafka Sources** or **Kafka Sinks** tab.

    3. Click the ellipsis at the end of the row of the connector, and then click **Restart**.
  </Tab>
</Tabs>

## Pause a connector

You can pause a running connector using different tools, a paused connector still occupy the resources but just stop processing messages.

<Tabs>
  <Tab title="snctl">
    To pause the source connector `test`, use the following command.

    ```bash theme={null}
    snctl kafka admin connect pause test
    ```

    You should see the following output:

    ```bash theme={null}
    Paused connector test
    ```

    If you want to verify whether the source connector has been paused successfully, run the following command.

    ```bash theme={null}
    snctl kafka admin connect get connectors
    ```

    You should see the following output:

    ```bash theme={null}
     NAME      TYPE     STATE     TASKS
     datagen   source   PAUSED    0: PAUSED
    ```
  </Tab>

  <Tab title="kcctl">
    To pause the source connector `test`, use the following command.

    ```bash theme={null}
    kcctl pause connectors test
    ```

    You should see the following output:

    ```bash theme={null}
    Paused connector test
    ```

    If you want to verify whether the source connector has been paused successfully, run the following command.

    ```bash theme={null}
    kcctl get connectors
    ```

    You should see the following output:

    ```bash theme={null}
     NAME      TYPE     STATE     TASKS
     datagen   source   PAUSED    0: PAUSED
    ```
  </Tab>

  <Tab title="Rest API">
    To pause the source connector `test`, use the following command.

    ```bash theme={null}
    > curl -X PUT --header 'content-type: application/json' "https://public%2Fdefault:${APIKEY}@${KAFKA-SERVICE-URL}/admin/kafkaconnect/connectors/test/pause"
    ```

    If no error is response, the pause is successful.

    If you want to verify whether the source connector has been paused successfully, run the following command.

    ```bash theme={null}
    curl -X GET "https://public%2Fdefault:${APIKEY}@${KAFKA-SERVICE-URL}/admin/kafkaconnect/connectors/test/status | jq '.'"
    ```

    You should see the following output:

    ```bash theme={null}
    {
      "name": "test",
      "connector": {
        "state": "PAUSED",
        "worker_id": "10.80.161.255:8083",
        "trace": "",
      },
      "tasks": [
        {
          "id": 0,
          "state": "PAUSEd",
          "worker_id": "10.244.0.75:8083"
        }
      ],
      "type": "source"
    }
    ```
  </Tab>

  <Tab title="Console">
    1. On the left navigation pane of StreamNative Console, under **Resources**, click **Connectors**.

    2. On the Connectors page, select the **Kafka Sources** or **Kafka Sinks** tab.

    3. Click the ellipsis at the end of the row of the connector, and then click **Pause**.
  </Tab>
</Tabs>

## Resume a connector

You can resume a paused connector using different tools.

<Tabs>
  <Tab title="snctl">
    To resume the source connector `test`, use the following command.

    ```bash theme={null}
    snctl kafka admin connect resume test
    ```

    You should see the following output:

    ```bash theme={null}
    Resumed connector test
    ```

    If you want to verify whether the source connector has been resumed successfully, run the following command.

    ```bash theme={null}
    snctl kafka admin connect get connectors
    ```

    You should see the following output:

    ```bash theme={null}
     NAME      TYPE     STATE     TASKS
     datagen   source   RUNNING   0: RUNNING
    ```
  </Tab>

  <Tab title="kcctl">
    To resume the source connector `test`, use the following command.

    ```bash theme={null}
    kcctl resume connectors test
    ```

    You should see the following output:

    ```bash theme={null}
    Resumed connector test
    ```

    If you want to verify whether the source connector has been resumed successfully, run the following command.

    ```bash theme={null}
    kcctl get connectors
    ```

    You should see the following output:

    ```bash theme={null}
     NAME      TYPE     STATE     TASKS
     datagen   source   RUNNING   0: RUNNING
    ```
  </Tab>

  <Tab title="Rest API">
    To resume the source connector `test`, use the following command.

    ```bash theme={null}
    > curl -X PUT --header 'content-type: application/json' "https://public%2Fdefault:${APIKEY}@${KAFKA-SERVICE-URL}/admin/kafkaconnect/connectors/test/resume"
    ```

    If no error is response, the pause is successful.

    If you want to verify whether the source connector has been resumed successfully, run the following command.

    ```bash theme={null}
    curl -X GET "https://public%2Fdefault:${APIKEY}@${KAFKA-SERVICE-URL}/admin/kafkaconnect/connectors/test/status | jq '.'"
    ```

    You should see the following output:

    ```bash theme={null}
    {
      "name": "test",
      "connector": {
        "state": "RESUMED",
        "worker_id": "10.80.161.255:8083",
        "trace": "",
      },
      "tasks": [
        {
          "id": 0,
          "state": "RESUMED",
          "worker_id": "10.244.0.75:8083"
        }
      ],
      "type": "source"
    }
    ```
  </Tab>

  <Tab title="Console">
    1. On the left navigation pane of StreamNative Console, under **Resources**, click **Connectors**.

    2. On the Connectors page, select the **Kafka Sources** or **Kafka Sinks** tab.

    3. Click the ellipsis at the end of the row of the connector, and then click **Resume**.
  </Tab>
</Tabs>

## What’s next?

* [Monitor and troubleshoot kafka connectors](/cloud/connect/kafka-connect/kafka-connect-monitoring)

* [SMTs(Single Message Transformations)](/cloud/connect/kafka-connect/kafka-connect-smt)

* Discover kafka Connect Ecosystem on [StreamNative Hub](/connect/overview).


# Monitor and Troubleshoot Kafka Connectors
Source: https://docs.streamnative.io/cloud/connect/kafka-connect/kafka-connect-monitoring



StreamNative Cloud allows you to monitor kafka connectors' status and logs.

## View connector status

This section describes how to view connector status using `snctl`, `kcctl`, `Rest API`, and console.

The following example introduces how to view the status of the data generator source connector named `test`.

<Tabs>
  <Tab title="snctl">
    To check the status of the source connector `test`, run the following command:

    ```bash theme={null}
    snctl kafka admin connect describe connector test
    ```

    ```bash theme={null}
    Name:       test
    Type:       source
    State:      RUNNING
    Worker ID:  10.80.161.252:8083
    Config:
      config.action.reload:               restart
      connector.class:                    io.confluent.kafka.connect.datagen.DatagenConnector
      errors.log.enable:                  false
      errors.log.include.messages:        false
      errors.retry.delay.max.ms:          60000
      errors.retry.timeout:               0
      errors.tolerance:                   none
      exactly.once.source.support:        disabled
      exactly.once.support:               requested
      header.converter:                   org.apache.kafka.connect.storage.SimpleHeaderConverter
      iterations:                         -1
      kafka.topic:                        testgen
      key.converter:                      org.apache.kafka.connect.json.JsonConverter
      max.interval:                       500
      name:                               test
      offset.storage.partitions:          25
      offset.storage.replication.factor:  3
      offset.storage.topic:               kafka-connect-offset-storage
      quickstart:                         users
      tasks.max:                          1
      tasks.max.enforce:                  true
      transaction.boundary:               poll
      value.converter:                    org.apache.kafka.connect.json.JsonConverter
    Tasks:
      0:
        State:      RUNNING
        Worker ID:  10.80.161.252:8083
    Topics:
      testgen
    ```
  </Tab>

  <Tab title="kcctl">
    To check the status of the source connector `test`, run the following command:

    ```bash theme={null}
    kcctl describe connectors test
    ```

    ```bash theme={null}
    Name:       test
    Type:       source
    State:      RUNNING
    Worker ID:  10.80.161.252:8083
    Config:
      config.action.reload:               restart
      connector.class:                    io.confluent.kafka.connect.datagen.DatagenConnector
      errors.log.enable:                  false
      errors.log.include.messages:        false
      errors.retry.delay.max.ms:          60000
      errors.retry.timeout:               0
      errors.tolerance:                   none
      exactly.once.source.support:        disabled
      exactly.once.support:               requested
      header.converter:                   org.apache.kafka.connect.storage.SimpleHeaderConverter
      iterations:                         -1
      kafka.topic:                        testgen
      key.converter:                      org.apache.kafka.connect.json.JsonConverter
      max.interval:                       500
      name:                               test
      offset.storage.partitions:          25
      offset.storage.replication.factor:  3
      offset.storage.topic:               kafka-connect-offset-storage
      quickstart:                         users
      tasks.max:                          1
      tasks.max.enforce:                  true
      transaction.boundary:               poll
      value.converter:                    org.apache.kafka.connect.json.JsonConverter
    Tasks:
      0:
        State:      RUNNING
        Worker ID:  10.80.161.252:8083
    Topics:
      testgen
    ```
  </Tab>

  <Tab title="Rest API">
    To check the status of the source connector `test`, run the following command:

    ```bash theme={null}
    curl -X GET "https://public%2Fdefault:${APIKEY}@${KAFKA-SERVICE-URL}/admin/kafkaconnect/connectors/test"
    ```

    You should see the following output:

    ```bash theme={null}
    {
      "name": "test",
      "config": {
        "config.action.reload": "restart",
        "connector.class": "io.confluent.kafka.connect.datagen.DatagenConnector",
        "errors.log.enable": "false",
        "errors.log.include.messages": "false",
        "errors.retry.delay.max.ms": "60000",
        "errors.retry.timeout": "0",
        "errors.tolerance": "none",
        "exactly.once.source.support": "disabled",
        "exactly.once.support": "requested",
        "header.converter": "org.apache.kafka.connect.storage.SimpleHeaderConverter",
        "iterations": "-1",
        "kafka.topic": "testgen",
        "key.converter": "org.apache.kafka.connect.json.JsonConverter",
        "max.interval": "500",
        "name": "test",
        "offset.storage.partitions": "25",
        "offset.storage.replication.factor": "3",
        "offset.storage.topic": "kafka-connect-offset-storage",
        "quickstart": "users",
        "tasks.max": "1",
        "tasks.max.enforce": "true",
        "transaction.boundary": "poll",
        "value.converter": "org.apache.kafka.connect.json.JsonConverter"
      },
      "tasks": [
        {
          "connector": "test",
          "task": 0
        }
      ],
      "type": "source"
    }
    ```
  </Tab>

  <Tab title="Console">
    1. From the left navigation pane, under **Resources**, click **Connectors**.

    2. Select the **Kafka Sources** or **Kafka Sinks** tab and click the connector you want to check, you will see the status of this connector and its tasks, and also the logs and metrics.

       <img alt="image of kafka connect detail overview" />
  </Tab>
</Tabs>

## View connector logs

### View the connector logs using `snctl`

This section describes how to view connector logs using `snctl`.

<Note title="Note">
  This example assumes you have [installed snctl](/tools/cli/snctl/snctl-overview#install-snctl) and [initialized snctl configurations](/tools/cli/snctl/snctl-overview#initialize-snctl-configuration).
</Note>

You can run the `snctl logs` command to view logs for a specific connector. This table outlines the configuration options that are used for viewing connector logs. For details about all supported fields, you can use the `snctl logs -h` command to list more information.

| Option                   | Descriptions                                                                                                     |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------- |
| `-c` or `--cluster`      | The name of your Pulsar cluster where the connector is created.                                                  |
| `-p` or `--component`    | The type of component to monitor. Available options are `function`, `sink`, `source` and `kafka-connect`.        |
| `-f` or `--follow`       | Continuously list the connector log history.                                                                     |
| `-h` or `--help`         | Show usage information about the `snctl logs` command.                                                           |
| `-i` or `--instance`     | The name of your Pulsar instance where the connector is created.                                                 |
| `--name`                 | The name of your connector.                                                                                      |
| `-o` or `--organization` | The name of your organization where the connector is created.                                                    |
| `--previous`             | Print the logs that are generated before the configured timestamp.                                               |
| `--pulsar-tenant`        | The name of your Pulsar tenant where the connector is created.                                                   |
| `--pulsar-namespace`     | The name of your Pulsar namespace where the connector is created.                                                |
| `--since`                | List logs more recent than the specific time. Available units are `second`, `minute`, and `hour`, such as `24h`. |
| `-s` or `--size`         | Specify how many lines of recent logs to display.                                                                |
| `--timestamp`            | Include timestamps on each line in the log output.                                                               |

The following command example shows how to view up to 60 lines of the `data1` sink connector’s logs within the last 5 hours.

```bash theme={null}
snctl logs --since 5h --organization sndev --instance aws --cluster aws --name data1 --pulsar-tenant public --pulsar-namespace default -p kafka-connect -f -s 60
```

You should see the following output:

```bash theme={null}
2024-08-26T04:37:21,290+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 241 acknowledged messages
2024-08-26T04:38:21,306+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 229 acknowledged messages
2024-08-26T04:39:13,572+0000 [kafka-admin-client-thread | datagen-ui-73c8b112-mesh-admin] INFO  org.apache.kafka.clients.NetworkClient - [AdminClient clientId=datagen-ui-73c8b112-mesh-admin] Node 1753251735 disconnected.
2024-08-26T04:39:21,320+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 233 acknowledged messages
2024-08-26T04:40:18,606+0000 [kafka-producer-network-thread | connector-producer-datagen-ui-0] INFO  org.apache.kafka.clients.NetworkClient - [Producer clientId=connector-producer-datagen-ui-0] Node 1261922880 disconnected.
2024-08-26T04:40:21,334+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 261 acknowledged messages
2024-08-26T04:41:21,349+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 235 acknowledged messages
2024-08-26T04:42:21,362+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 270 acknowledged messages
2024-08-26T04:43:21,381+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 235 acknowledged messages
2024-08-26T04:44:13,738+0000 [kafka-admin-client-thread | datagen-ui-73c8b112-mesh-admin] INFO  org.apache.kafka.clients.NetworkClient - [AdminClient clientId=datagen-ui-73c8b112-mesh-admin] Node 1261922880 disconnected.
2024-08-26T04:44:21,393+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 240 acknowledged messages
2024-08-26T04:45:21,407+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 227 acknowledged messages
2024-08-26T04:46:21,421+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 234 acknowledged messages
2024-08-26T04:47:21,436+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 226 acknowledged messages
2024-08-26T04:48:21,449+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 248 acknowledged messages
2024-08-26T04:49:13,883+0000 [kafka-admin-client-thread | datagen-ui-73c8b112-mesh-admin] INFO  org.apache.kafka.clients.NetworkClient - [AdminClient clientId=datagen-ui-73c8b112-mesh-admin] Node 1753251735 disconnected.
2024-08-26T04:49:21,462+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 231 acknowledged messages
2024-08-26T04:50:21,479+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 218 acknowledged messages
2024-08-26T04:51:21,492+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 241 acknowledged messages
2024-08-26T04:52:21,509+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 247 acknowledged messages
2024-08-26T04:53:21,530+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 230 acknowledged messages
2024-08-26T04:54:14,033+0000 [kafka-admin-client-thread | datagen-ui-73c8b112-mesh-admin] INFO  org.apache.kafka.clients.NetworkClient - [AdminClient clientId=datagen-ui-73c8b112-mesh-admin] Node 1261922880 disconnected.
2024-08-26T04:54:18,658+0000 [kafka-producer-network-thread | connector-producer-datagen-ui-0] INFO  org.apache.kafka.clients.NetworkClient - [Producer clientId=connector-producer-datagen-ui-0] Node 1261922880 disconnected.
2024-08-26T04:54:21,559+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 221 acknowledged messages
2024-08-26T04:55:21,585+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 238 acknowledged messages
2024-08-26T04:56:21,614+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 224 acknowledged messages
2024-08-26T04:57:21,643+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 226 acknowledged messages
2024-08-26T04:58:21,655+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 236 acknowledged messages
2024-08-26T04:59:14,185+0000 [kafka-admin-client-thread | datagen-ui-73c8b112-mesh-admin] INFO  org.apache.kafka.clients.NetworkClient - [AdminClient clientId=datagen-ui-73c8b112-mesh-admin] Node 1753251735 disconnected.
2024-08-26T04:59:21,673+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 242 acknowledged messages
2024-08-26T05:00:21,690+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 233 acknowledged messages
2024-08-26T05:01:21,715+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 244 acknowledged messages
2024-08-26T05:02:21,728+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 242 acknowledged messages
2024-08-26T05:03:21,743+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 249 acknowledged messages
2024-08-26T05:04:14,338+0000 [kafka-admin-client-thread | datagen-ui-73c8b112-mesh-admin] INFO  org.apache.kafka.clients.NetworkClient - [AdminClient clientId=datagen-ui-73c8b112-mesh-admin] Node 1261922880 disconnected.
2024-08-26T05:04:21,758+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 226 acknowledged messages
2024-08-26T05:05:21,779+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 249 acknowledged messages
2024-08-26T05:06:21,800+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 234 acknowledged messages
2024-08-26T05:07:21,812+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 249 acknowledged messages
2024-08-26T05:08:21,827+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 239 acknowledged messages
2024-08-26T05:09:14,476+0000 [kafka-admin-client-thread | datagen-ui-73c8b112-mesh-admin] INFO  org.apache.kafka.clients.NetworkClient - [AdminClient clientId=datagen-ui-73c8b112-mesh-admin] Node 1753251735 disconnected.
2024-08-26T05:09:21,841+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 232 acknowledged messages
2024-08-26T05:10:21,855+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 252 acknowledged messages
2024-08-26T05:11:21,870+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 242 acknowledged messages
2024-08-26T05:12:21,884+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 234 acknowledged messages
2024-08-26T05:13:21,897+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 237 acknowledged messages
2024-08-26T05:14:14,648+0000 [kafka-admin-client-thread | datagen-ui-73c8b112-mesh-admin] INFO  org.apache.kafka.clients.NetworkClient - [AdminClient clientId=datagen-ui-73c8b112-mesh-admin] Node 1261922880 disconnected.
2024-08-26T05:14:21,909+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 237 acknowledged messages
2024-08-26T05:15:21,920+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 234 acknowledged messages
2024-08-26T05:16:21,934+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 248 acknowledged messages
2024-08-26T05:17:21,953+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 232 acknowledged messages
2024-08-26T05:18:21,972+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 238 acknowledged messages
2024-08-26T05:19:14,802+0000 [kafka-admin-client-thread | datagen-ui-73c8b112-mesh-admin] INFO  org.apache.kafka.clients.NetworkClient - [AdminClient clientId=datagen-ui-73c8b112-mesh-admin] Node 1753251735 disconnected.
2024-08-26T05:19:21,991+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 226 acknowledged messages
2024-08-26T05:20:22,004+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 235 acknowledged messages
2024-08-26T05:21:22,018+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 251 acknowledged messages
2024-08-26T05:22:22,033+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 238 acknowledged messages
2024-08-26T05:23:22,047+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 227 acknowledged messages
2024-08-26T05:24:14,938+0000 [kafka-admin-client-thread | datagen-ui-73c8b112-mesh-admin] INFO  org.apache.kafka.clients.NetworkClient - [AdminClient clientId=datagen-ui-73c8b112-mesh-admin] Node 1261922880 disconnected.
2024-08-26T05:24:22,061+0000 [SourceTaskOffsetCommitter-1] INFO  org.apache.kafka.connect.runtime.WorkerSourceTask - WorkerSourceTask{id=datagen-ui-0} Committing offsets for 236 acknowledged messages
```

### View the connector logs from log topics

StreamNative Cloud provide a way to send your logs to a Pulsar topic, you can just use config `sn.log.topic` to set it.

Once the connector is deployed with this config, you can configure consumers to consume messages from the log topic.

## What’s next?

* [SMTs(Single Message Transformations)](/cloud/connect/kafka-connect/kafka-connect-smt)

* Discover kafka Connect Ecosystem on [StreamNative Hub](/connect/overview).


# Kafka Connect Overview
Source: https://docs.streamnative.io/cloud/connect/kafka-connect/kafka-connect-overview



<Tip title="Tip">
  The Kafka Connect functionality requires Apache Pulsar version 3.3.1.4 or higher on your StreamNative cluster. If your current version is below 3.3.1.4, please [contact our support team](https://support.streamnative.io/hc/en-us) to upgrade your cluster. Alternatively, you may create a new StreamNative cluster to access Kafka Connect functionality.

  Additionally, ensure that the 'Kafka Protocol' and 'Pulsar Function' options are enabled at the cluster level. Without this, the Kafka Source and Kafka Sink tabs will not appear on the Connectors page.
</Tip>

## Introduction to Kafka Connect

### Concept

[Kafka Connect](https://kafka.apache.org/documentation/#connect) is an integration tool that is released with the Apache Kafka project. It provides reliable data streaming between Apache Kafka and external systems and is both scalable and flexible. Kafka Connect works with Kafka on Pulsar (KoP), which is compatible with the Kafka API.

Kafka Connect uses Source and Sink connectors for integration. Source connectors stream data from an external system to Kafka, while Sink connectors stream data from Kafka to an external system.

The following diagram illustrates the data movement among source connectors, Pulsar Kop, sink connectors, and external systems.

<img alt="Pulsar KoP and elasticsearch with kafka connect" />

### Benefits

* Seamless & Simplified Data Integration: Kafka Connect provide a unified interface for connecting Kop to various external systems and data sources, allowing developers to easily integrate Kop with their existing infrastructure without the need for custom integration code.

* Extensibility: Kafka Connect are designed to be easily extensible, allowing developers to create custom connectors for specific use cases and data sources not covered by the [built-in connectors](#built-in-connectors-on-stream-native-cloud).

* Reduced Development Effort: By leveraging pre-built connectors, developers can save time and effort. They don't have to write and maintain complex integration code from scratch for each external system.

* Reliable and Scalable: Kafka Connect are built to be reliable and scalable, ensuring the data transfers between Kop and external systems are efficient and fault-tolerant.

### Use cases

* Data Ingestion to Pulsar Kop: If you have data coming from various external sources, such as databases, message queues, or cloud storage systems like Amazon S3, you can use Kafka Connect to ingest that data into Kop topics.

* Data Export from Pulsar Kop: Kafka Connect also enable you to export data from Kop topics to other systems or storage solutions. This helps you to synchronize data across different environments, replicate data, or stream data to external services or databases.

* Real-time Data Processing: Kafka Connect facilitate real-time data processing by enabling the seamless movement of data between Kop and other systems. This is particularly useful in event-driven architectures, streaming applications, and microservices-based solutions.

* Extending Kop's Functionality: If you have specific use cases or data sources not directly supported by Kop, you can deploy custom Kafka Connect to extend Kop's functionality and integrate with those systems.

### Connectors Shared Responsibility

StreamNative and our customers have a shared responsibility for maintaining and keeping connectors properly functioning. Outlined below, StreamNative has a responsibility for custom connectors that are maintained by uploading to StreamNative Cloud to maintain connectivity to the Pulsar Kop cluster and for logging and monitoring. Customers who upload their connectors are responsible for all other operations such as configurations, updates, development, support and plugin installation. Partner connectors have similar support but the partners with StreamNative will be responsible for development and connector support. For built in connectors, StreamNative is responsible for everything except connector configuration.

For more details, see below.

<img alt="Connectors Shared Responsibility Model" />

## Built-in connectors on StreamNative Cloud

To further reduce the development overhead and time, StreamNative has pre-built a variety of Kafka connect on StreamNative Cloud. With proper configurations, you can integrate the data between your Pulsar Kop cluster on StreamNative Cloud and your data systems effortlessly.

### Built-in source connectors

Currently, StreamNative Cloud supports the following kafka source connectors:

* [MongoDB source connector](/connect/connectors/kafka-connect-mongo-source/current/kafka-connect-mongodb-source)
* [YugabyteDB CDC source connector](/connect/connectors/kafka-connect-yugabyte-cdc-source/current/kafka-connect-yugabyte-cdc-source)
* [Datagen source connector](/connect/connectors/kafka-connect-datagen/current/kafka-connect-datagen-source)
* [Cosmos DB source connector](/connect/connectors/kafka-connect-cosmosdb-source/current/kafka-connect-cosmosdb-source)
* [Debezium MongoDB source connector](/connect/connectors/kafka-connect-debezium-mongodb/current/kafka-connect-debezium-mongodb)
* [Debezium Mysql source connector](/connect/connectors/kafka-connect-debezium-mysql/current/kafka-connect-debezium-mysql)
* [Debezium PostgreSql source connector](/connect/connectors/kafka-connect-debezium-postgresql/current/kafka-connect-debezium-postgresql)
* [Debezium Spanner source connector](/connect/connectors/kafka-connect-debezium-spanner/current/kafka-connect-debezium-spanner)
* [Google Pub/Sub source connector](/connect/connectors/kafka-connect-google-pubsub-source/current/kafka-connect-google-pubsub-source)
* [Google Pub/Sub Lite source connector](/connect/connectors/kafka-connect-google-pubsub-lite-source/current/kafka-connect-google-pubsub-lite-source)
* [Debezium SqlServer source connector](/connect/connectors/kafka-connect-debezium-sqlserver/current/kafka-connect-debezium-sqlserver)
* [JDBC source connector](/connect/connectors/kafka-connect-jdbc-source/current/kafka-connect-jdbc-source)
* [JR source connector](/connect/connectors/kafka-connect-jr-source/current/kafka-connect-jr-source)

### Built-in sink connectors

Currently, StreamNative Cloud supports the following kafka sink connectors.

* [Iceberg sink connector](/connect/connectors/kafka-connect-iceberg/current/kafka-connect-iceberg-sink)
* [Milvus sink connector](/connect/connectors/kafka-connect-milvus-sink/current/kafka-connect-milvus-sink)
* [MongoDB sink connector](/connect/connectors/kafka-connect-mongo-sink/current/kafka-connect-mongodb-sink)
* [BigQuery sink connector](/connect/connectors/kafka-connect-bigquery/current/kafka-connect-bigquery)
* [Cosmos DB sink connector](/connect/connectors/kafka-connect-cosmosdb-sink/current/kafka-connect-cosmosdb-sink)
* [ElasticSearch sink connector](/connect/connectors/kafka-connect-elasticsearch-sink/current/kafka-connect-elasticsearch-sink)
* [Snowflake sink connector](/connect/connectors/kafka-connect-snowflake-sink/current/kafka-connect-snowflake-sink)
* [JDBC sink connector](/connect/connectors/kafka-connect-jdbc-sink/current/kafka-connect-jdbc-sink)
* [Debezium JDBC sink connector](/connect/connectors/kafka-connect-debezium-jdbc-sink/current/kafka-connect-debezium-jdbc-sink)
* [Google Pub/Sub sink connector](/connect/connectors/kafka-connect-google-pubsub-sink/current/kafka-connect-google-pubsub-sink)
* [Google Pub/Sub Lite sink connector](/connect/connectors/kafka-connect-google-pubsub-lite-sink/current/kafka-connect-google-pubsub-lite-sink)
* [Google Cloud Storage sink connector](/connect/connectors/kafka-connect-google-cloud-storage-sink/current/kafka-connect-google-cloud-storage-sink)
* [Google Bigtable sink connector](/connect/connectors/kafka-connect-google-bigtable-sink/current/kafka-connect-google-bigtable-sink)

<Tip title="Tip">
  Currently, the Kafka Connect doesn't support multi-tenancy. All Kafka Connects are deployed to the `public/default` namespace.

  And all of configured topics are also under the `public/default` namespace by default, unless you specified topics with the prefix: `${tenant}.${namespace}.`.

  For more details about the multi-tenancy support for Kafka topics in StreamNative cloud, please refer to the [Kafka Multi-Tenancy](/cloud/build/kafka-clients/advanced-features/kafka-multi-tenancy).
</Tip>

## Self-hosted Kafka Connect

Despite the fact that StreamNative Cloud supports fully managed Kafka Connect connectors, you can still self-host Kafka Connect connectors in your own environment. See the [Kafka Connect QuickStart](id:cloud-connect-kafka-connect) for how to configure your own Kafka Connect connectors to connect to your StreamNative cluster.

## What’s next?

* [Deploy kafka connectors](/cloud/connect/kafka-connect/deploy-kafka-connectors/deploy-kafka-connect-index)

* [Manage kafka connectors](/cloud/connect/kafka-connect/kafka-connect-manage)

* [Monitor and troubleshoot kafka connectors](/cloud/connect/kafka-connect/kafka-connect-monitoring)

* [SMTs(Single Message Transformations)](/cloud/connect/kafka-connect/kafka-connect-smt)

* Discover kafka Connect Ecosystem on [StreamNative Hub](/connect/overview).


# Kafka Connect SMTs(Single Message Transformations)
Source: https://docs.streamnative.io/cloud/connect/kafka-connect/kafka-connect-smt



## Transformations

StreamNative cloud supports Kafka Connect transformations. You can use the following transformations in your Kafka Connect configurations:

* [InsertField](https://kafka.apache.org/37/documentation.html#org.apache.kafka.connect.transforms.InsertField) - Add a field using either static data or record metadata
* [ReplaceField](https://kafka.apache.org/37/documentation.html#org.apache.kafka.connect.transforms.ReplaceField) - Filter or rename fields
* [MaskField](https://kafka.apache.org/37/documentation.html#org.apache.kafka.connect.transforms.MaskField) - Replace field with valid null value for the type (0, empty string, etc) or custom replacement (non-empty string or numeric value only)
* [ValueToKey](https://kafka.apache.org/37/documentation.html#org.apache.kafka.connect.transforms.ValueToKey) - Replace the record key with a new key formed from a subset of fields in the record value
* [HoistField](https://kafka.apache.org/37/documentation.html#org.apache.kafka.connect.transforms.HoistField) - Wrap the entire event as a single field inside a Struct or a Map
* [ExtractField](https://kafka.apache.org/37/documentation.html#org.apache.kafka.connect.transforms.ExtractField) - Extract a specific field from Struct and Map and include only this field in results
* [SetSchemaMetadata](https://kafka.apache.org/37/documentation.html#org.apache.kafka.connect.transforms.SetSchemaMetadata) - modify the schema name or version
* [TimestampRouter](https://kafka.apache.org/37/documentation.html#org.apache.kafka.connect.transforms.TimestampRouter) - Modify the topic of a record based on original topic and timestamp. Useful when using a sink that needs to write to different tables or indexes based on timestamps
* [RegexRouter](https://kafka.apache.org/37/documentation.html#org.apache.kafka.connect.transforms.RegexRouter) - modify the topic of a record based on original topic, replacement string and a regular expression
* [Flatten](https://kafka.apache.org/37/documentation.html#org.apache.kafka.connect.transforms.Flatten): Flatten a nested data structure, generating names for each field by concatenating the field names at each level with a configurable delimiter character.
* [Cast](https://kafka.apache.org/37/documentation.html#org.apache.kafka.connect.transforms.Cast): Cast fields or the entire key or value to a specific type, e.g. to force an integer field to a smaller width.
* [TimestampConverter](https://kafka.apache.org/37/documentation.html#org.apache.kafka.connect.transforms.TimestampConverter): Convert timestamps between different formats such as Unix epoch, strings, and Connect Date/Timestamp types.
* [Filter](https://kafka.apache.org/37/documentation.html#org.apache.kafka.connect.transforms.Filter) - Removes messages from all further processing. This is used with a predicate to selectively filter certain messages.
* [InsertHeader](https://kafka.apache.org/37/documentation.html#org.apache.kafka.connect.transforms.InsertHeader) - Add a header using static data
* [HeadersFrom](https://kafka.apache.org/37/documentation.html#org.apache.kafka.connect.transforms.HeaderFrom) - Copy or move fields in the key or value to the record headers
* [DropHeaders](https://kafka.apache.org/37/documentation.html#org.apache.kafka.connect.transforms.DropHeaders) - Remove headers by name

Some built-in connectors also has specified transformations:

### YugabyteDB CDC Source

* [YBExtractNewRecordState SMT](https://docs.yugabyte.com/preview/explore/change-data-capture/using-yugabytedb-grpc-replication/debezium-connector-yugabytedb/#ybextractnewrecordstate-smt)
* [ExtractTopic](https://docs.yugabyte.com/preview/explore/change-data-capture/using-yugabytedb-grpc-replication/debezium-connector-yugabytedb/#ybextractnewrecordstate-smt)
* [PGCompatible SMT](https://docs.yugabyte.com/preview/explore/change-data-capture/using-yugabytedb-grpc-replication/debezium-connector-yugabytedb/#ybextractnewrecordstate-smt)

### Iceberg Sink

* [CopyValue](https://github.com/tabular-io/iceberg-kafka-connect/blob/main/kafka-connect-transforms/README.md#copyvalue)
* [DmsTransform](https://github.com/tabular-io/iceberg-kafka-connect/blob/main/kafka-connect-transforms/README.md#dmstransform)
* [DebeziumTransform](https://github.com/tabular-io/iceberg-kafka-connect/blob/main/kafka-connect-transforms/README.md#debeziumtransform)
* [JsonToMapTransform](https://github.com/tabular-io/iceberg-kafka-connect/blob/main/kafka-connect-transforms/README.md#jsontomaptransform)
* [KafkaMetadataTransform](https://github.com/tabular-io/iceberg-kafka-connect/blob/main/kafka-connect-transforms/README.md#kafkametadatatransform)
* [MongoDebeziumTransform](https://github.com/tabular-io/iceberg-kafka-connect/blob/main/kafka-connect-transforms/README.md#mongodebeziumtransform)

## Predicates

StreamNative cloud supports Kafka Connect predicates. You can use the following predicates in your Kafka Connect configurations:

* [TopicNameMatches](https://kafka.apache.org/37/documentation.html#org.apache.kafka.connect.transforms.predicates.TopicNameMatches): matches records in a topic with a name matching a particular Java regular expression.
* [HasHeaderKey](https://kafka.apache.org/37/documentation.html#org.apache.kafka.connect.transforms.predicates.HasHeaderKey): matches records which have a header with the given key.
* [RecordIsTombstone](https://kafka.apache.org/37/documentation.html#org.apache.kafka.connect.transforms.predicates.RecordIsTombstone): matches tombstone records, that is records with a null value.

## What’s next?

* Discover kafka Connect Ecosystem on [StreamNative Hub](/connect/overview).


# Configuration Reference
Source: https://docs.streamnative.io/cloud/connect/pulsar-io/connector-config



This section lists all the common configuration options for the built-in source and sink connectors. For connector-specific configurations, see [StreamNative Hub](/connect/overview).

## Source connector configurations

This table lists all the common configurations for a source connector.

| Field                         | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `-a`, `--archive`             | The path to the NAR archive for the source. <br /> It supports the file-URL-path (file://) which assumes that the NAR file already exists on the worker host from which the worker can download the package. <br /> For a built-in connector, it should be set to `builtin//<connector_name>`.                                                                                                                                                                                                                                                                                           |
| `--classname`                 | The source's class name if the `archive` is set to a file-URL-path (file://).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `--cpu`                       | The CPU (in cores) that needs to be allocated per source instance (applicable only to Docker runtime).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `--deserialization-classname` | The SerDe classname for the source.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `--destination-topic-name`    | The Pulsar topic to which data is sent.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `--disk`                      | The disk (in bytes) that needs to be allocated per source instance (applicable only to Docker runtime).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `--name`                      | The source's name.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `--namespace`                 | The source's namespace.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `--parallelism`               | The source's parallelism factor, that is, the number of source instances to run.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `--processing-guarantees`     | The processing guarantees (also named as delivery semantics) applied to the source. A source connector receives messages from the external system and writes messages to a Pulsar topic. The `--processing-guarantees` ensures the processing guarantees for writing messages to the Pulsar topic. <br />The available values are `ATLEAST_ONCE`, `ATMOST_ONCE`, `EFFECTIVELY_ONCE`.                                                                                                                                                                                                     |
| `--ram`                       | The RAM (in bytes) that needs to be allocated per source instance (applicable only to the process and Docker runtimes).                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `-st`, `--schema-type`        | The schema type.<br /> Either a built-in schema (for example, AVRO and JSON) or a custom schema class name to be used to encode messages emitted from source.                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `--source-config`             | The key/values configurations of the source. For example: '`{"sleepBetweenMessages": 60}`'; For configuration details, refer to the [documentation](/connect/overview) of each connector                                                                                                                                                                                                                                                                                                                                                                                                 |
| `--source-config-file`        | The path to a YAML config file that specifies the source's configuration.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `--sn-service-account`        | The StreamNative Cloud service account that the source uses as its runtime identity. This `snctl` flag writes `snServiceAccount` to `custom-runtime-options`. You can use `--use-sn-service-account` to select a runtime service account interactively. You must have the `cloud.serviceaccounts.describe` permission on the specified service account (see [account-admin](/cloud/security/access/rbac/manage-rbac-roles#account-admin)). Only Pulsar 4.0.x clusters running version 4.0.10.6 or later, or Pulsar 4.2.x clusters running version 4.2.1.4 or later, support this config. |
| `-t`, `--source-type`         | The source's connector provider.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `--tenant`                    | The source's tenant.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `--producer-config`           | The custom producer configuration (as a JSON string).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |

## Sink connector configurations

This table lists all the common configurations for a sink connector.

| Field                     | Description                                                                                                                                                                                                                                                                                   |
| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `-a`, `--archive`         | The path to the archive file for the sink. <br /> It supports the file-URL-path (file://) which assumes that the NAR file already exists on the worker host from which the worker can download the package. <br /> For a built-in connector, it should be set to `builtin//<connector_name>`. |
| `--classname`             | The sink's class name if the `archive` is set to a file-URL-path (file://).                                                                                                                                                                                                                   |
| `--cpu`                   | The CPU (in cores) that needs to be allocated per sink instance (applicable only to Docker runtime).                                                                                                                                                                                          |
| `--custom-schema-inputs`  | The map of input topics to schema types or class names (as a JSON string).                                                                                                                                                                                                                    |
| `--custom-serde-inputs`   | The map of input topics to SerDe class names (as a JSON string).                                                                                                                                                                                                                              |
| `--disk`                  | The disk (in bytes) that needs to be allocated per sink instance (applicable only to Docker runtime).                                                                                                                                                                                         |
| `-i, --inputs`            | The sink's input topic or topics (multiple topics can be specified as a comma-separated list).                                                                                                                                                                                                |
| `--name`                  | The sink's name.                                                                                                                                                                                                                                                                              |
| `--namespace`             | The sink's namespace.                                                                                                                                                                                                                                                                         |
| `--parallelism`           | The sink's parallelism factor, that is, the number of sink instances to run.                                                                                                                                                                                                                  |
| `--processing-guarantees` | The processing guarantees (also known as delivery semantics) applied to the sink. The `--processing-guarantees` implementation in Pulsar also relies on sink implementation. <br />The available values are `ATLEAST_ONCE`, `ATMOST_ONCE`, `EFFECTIVELY_ONCE`.                                |
| `--ram`                   | The RAM (in bytes) that needs to be allocated per sink instance (applicable only to the process and Docker runtimes).                                                                                                                                                                         |
| `--retain-ordering`       | Sink consumes messages in order.                                                                                                                                                                                                                                                              |
| `--sink-config`           | The key/values configurations of the sink. For example: '`{"sleepBetweenMessages": 60}`'; For configuration details, refer to the [documentation](/connect/overview) of each connector                                                                                                        |
| `--sink-config-file`      | The path to a YAML config file specifying the sink's configuration.                                                                                                                                                                                                                           |
| `--sn-service-account`    | The StreamNative Cloud service account that the sink uses as its runtime identity. This `snctl` flag writes `snServiceAccount` to `custom-runtime-options`. You can use `--use-sn-service-account` to select a runtime service account interactively.                                         |
| `-t`, `--sink-type`       | The sink's connector provider. The `sink-type` parameter of the currently built-in connectors is determined by the setting of the `name` parameter. You can use the `pulsar-admin sinks available-sinks` command to get all built-in sink connectors.                                         |
| `--subs-name`             | Pulsar source subscription name if you want to specify a subscription name for the input-topic consumer.                                                                                                                                                                                      |
| `--tenant`                | The sink's tenant.                                                                                                                                                                                                                                                                            |
| `--timeout-ms`            | The message timeout in milliseconds.                                                                                                                                                                                                                                                          |
| `--topics-pattern`        | The topic pattern to consume from a list of topics under a namespace that matches the pattern. <br />`--input` and `--topics-Pattern` are mutually exclusive. <br />Add SerDe class name for a pattern in `--customSerdeInputs`.                                                              |

## StreamNative Cloud custom runtime options

To facilitate submitting Pulsar functions based on your requirements, Function on Cloud service provides some custom options via `custom-runtime-options`.

<Note title="Note">
  When update functions/sinks/sources with custom runtime options, the original custom runtime options will be replaced by the new ones, so make sure all the wanted fields are passed in the custom runtime options when you do the update.
</Note>

This table lists all fields available for custom options.

| Name                            | Type                  | Default | Description                                                                                                                                                                                                                                                                                                                                                                                                                              |
| ------------------------------- | --------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `clusterName`                   | String                | N/A     | The Pulsar cluster of a Pulsar function, source, or sink.                                                                                                                                                                                                                                                                                                                                                                                |
| `inputTypeClassName`            | String                | `[B`    | The map of input topics to Java class names.                                                                                                                                                                                                                                                                                                                                                                                             |
| `outputTypeClassName`           | String                | `[B`    | The map of output topics to Java class names.                                                                                                                                                                                                                                                                                                                                                                                            |
| `maxReplicas`                   | Integer               | `0`     | The maximum number of Pulsar instances that you want to run for this Pulsar Function. When the value of the `maxReplicas` parameter is greater than the value of `replicas`, it indicates that the Functions controller automatically scales the Pulsar Functions based on the CPU usage. By default, `maxReplicas` is set to 0, which indicates that auto-scaling is disabled.                                                          |
| `env`                           | `Map<String, String>` | N/A     | The environment variables being attached to a Pod that is created by the Function Mesh Operator for the cluster.                                                                                                                                                                                                                                                                                                                         |
| `imagePullSecrets`              | `List<String>`        | N/A     | A list of references to secrets in the same namespace for pulling any of the images used by a Pod.                                                                                                                                                                                                                                                                                                                                       |
| `logLevel`                      | String                | info    | The log levels for Pulsar functions. For details, see [log levels](https://functionmesh.io/docs/next/reference/crd-config/function-crd#log-levels).                                                                                                                                                                                                                                                                                      |
| `logRotationPolicy`             | String                | N/A     | The log rotation policies for Pulsar functions. You can set the log rotation policies based on the time or the log file size. For details, see [log rotation policies](https://functionmesh.io/docs/next/reference/crd-config/function-crd#log-rotation-policies).                                                                                                                                                                       |
| `runnerImageTag`                | String                | N/A     | The tag of the [runner image](https://functionmesh.io/docs/next/reference/crd-config/function-crd#runner-images) that is used to submit a function, source, or sink.                                                                                                                                                                                                                                                                     |
| `hpaSpec` (Preview)             | HPASpec               | N/A     | The Kubernetes HorizontalPodAutoscaler settings. For details, see [Kubernetes documentation](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/). This feature may not ready for your cluster enviroment, please file a ticket if you want this feature enabled.                                                                                                                                                 |
| `logFormat`                     | String                | text    | The log format that defines how the content of a log file should be interpreted. Available options are ` json` and `text`. The log format configurations are only available for the Java and Python runtimes.                                                                                                                                                                                                                            |
| `logTopic`                      | String                | N/A     | Used for sinks/sources since they don't have a log topic argument like functions                                                                                                                                                                                                                                                                                                                                                         |
| `logTopicAgent`                 | String                | runtime | The log agent that defines how StreamNative cloud redirects your functions / connectors log into a Pulsar topic. Available options are `runtime` and `sidecar`. When use `sidecar` all logs (including instance logs) will be sent to the log topic by filebeat (better performance than `runtime`), else will use the Pulsar Functions runtime's log-topic implementation instead.                                                      |
| `genericKind`                   | String                | N/A     | Used for functions written in languages other than Java and Python, available values are `executable`, `nodejs`, `wasm`                                                                                                                                                                                                                                                                                                                  |
| `snServiceAccount`              | String                | N/A     | The StreamNative Cloud service account that the function, source, or sink uses as its runtime identity. You must have the `cloud.serviceaccounts.describe` permission on the specified service account (see [account-admin](/cloud/security/access/rbac/manage-rbac-roles#account-admin)). Only Pulsar 4.0.x clusters running version 4.0.10.6 or later, or Pulsar 4.2.x clusters running version 4.2.1.4 or later, support this config. |
| `terminationGracePeriodSeconds` | Long                  | N/A     | The amount of time that kubernetes will give for a pod before terminating it.                                                                                                                                                                                                                                                                                                                                                            |
| `pauseRollout`                  | Boolean               | N/A     | Whether to pause the rollout of functions/connectors, when set to true, running functions\&connectors will not restart during the upgrade of backend function-mesh operator.                                                                                                                                                                                                                                                             |
| `enableStateStore`              | Boolean               | N/A     | Whether to enable stateful backend for this function, default to false, only available for **Java** function                                                                                                                                                                                                                                                                                                                             |

You can compose the custom runtime options as a JSON string and pass it to the `custom-runtime-options` field. For example:

```json theme={null}
{
  "inputTypeClassName": "java.lang.String",
  "logTopic": "log-topic-name"
}
```

Then it could be passed to `custom-runtime-options` field as follows:

```bash theme={null}
snctl pulsar admin sinks create --custom-runtime-options '{"inputTypeClassName":"java.lang.String","logTopic":"log-topic-name"}' --as-service-account $SERVICE_ACCOUNT ...
```

```bash theme={null}
pulsarctl sinks create --custom-runtime-options '{"inputTypeClassName":"java.lang.String","logTopic":"log-topic-name"}' ...
```


# Manage Connectors
Source: https://docs.streamnative.io/cloud/connect/pulsar-io/connector-manage



StreamNative Cloud enables you to manage Pulsar IO Connectors by using a variety of tools, including `snctl`, `pulsarctl`, `pulsar-admin`, and `Terraform`.

<Note title="Note">
  If you want to update or delete connectors using `snctl`, `pulsarctl` or `pulsar-admin`, make sure you have set up your client tool. For more information, see [set up client tools](/cloud/connect/pulsar-io/deploy-connectors/connector-setup#set-up-client-tools).
</Note>

## Update a connector

When you want to modify configurations or update resources for connectors, you can update connectors using multiple tools.

The following example shows how to update the parallelism of the data generator source connector `test` to `2` using different tools.

<Tabs>
  <Tab title="snctl">
    ```bash theme={null}
    snctl pulsar admin sources update \
    --name test \
    --parallelism 2
    ```

    You should see the following output:

    ```bash theme={null}
    Updated successfully
    ```

    And you can further check the status:

    ```bash theme={null}
    snctl pulsar admin sources status --name test

    {
      "numInstances": 2,
      "numRunning": 2,
      "instances": [
        {
          "instanceId": 0,
          "status": {
            "running": true,
            "error": "",
            "numRestarts": 0,
            "numReceivedFromSource": 1799,
            "numSystemExceptions": 0,
            "latestSystemExceptions": [],
            "numSourceExceptions": 0,
            "latestSourceExceptions": [],
            "numWritten": 1799,
            "lastReceivedTime": 1693946327331,
            "workerId": "test"
          }
        },
        {
          "instanceId": 1,
          "status": {
            "running": true,
            "error": "",
            "numRestarts": 0,
            "numReceivedFromSource": 689,
            "numSystemExceptions": 0,
            "latestSystemExceptions": [],
            "numSourceExceptions": 0,
            "latestSourceExceptions": [],
            "numWritten": 689,
            "lastReceivedTime": 1693946327129,
            "workerId": "test"
          }
        }
      ]
    }

    ```
  </Tab>

  <Tab title="pulsarctl">
    ```bash theme={null}
    pulsarctl sources update \
    --name test \
    --parallelism 2
    ```

    You should see the following output:

    ```bash theme={null}
    Updated successfully
    ```

    And you can further check the status:

    ```bash theme={null}
    pulsarctl sources status --name test

    {
      "numInstances": 2,
      "numRunning": 2,
      "instances": [
        {
          "instanceId": 0,
          "status": {
            "running": true,
            "error": "",
            "numRestarts": 0,
            "numReceivedFromSource": 1799,
            "numSystemExceptions": 0,
            "latestSystemExceptions": [],
            "numSourceExceptions": 0,
            "latestSourceExceptions": [],
            "numWritten": 1799,
            "lastReceivedTime": 1693946327331,
            "workerId": "test"
          }
        },
        {
          "instanceId": 1,
          "status": {
            "running": true,
            "error": "",
            "numRestarts": 0,
            "numReceivedFromSource": 689,
            "numSystemExceptions": 0,
            "latestSystemExceptions": [],
            "numSourceExceptions": 0,
            "latestSourceExceptions": [],
            "numWritten": 689,
            "lastReceivedTime": 1693946327129,
            "workerId": "test"
          }
        }
      ]
    }

    ```
  </Tab>

  <Tab title="pulsar-admin">
    ```bash theme={null}
    pulsar-admin \
        --admin-url "${WEB_SERVICE_URL}" \
        --auth-plugin org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 \
        --auth-params '{"privateKey":"file:///YOUR-KEY-FILE-PATH",
            "issuerUrl":"https://auth.streamnative.cloud/",
            "audience":"urn:sn:pulsar:${orgName}:${instanceName}}'
    sources update \
    --name test \
    --parallelism 2
    ```

    Replace the placeholder variables with the actual values, which you can get when you set up client tools.

    * `admin-url`: the HTTP service URL of your Pulsar cluster.
    * `private_key`: the path to the downloaded OAuth2 key file.
    * `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#uniform-resource-name-urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
      * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
      * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).

    You should see the following output:

    ```bash theme={null}
    Updated successfully
    ```

    And you can further check the status:

    ```bash theme={null}
    pulsar-admin sources status --name test
    {
      "numInstances" : 2,
      "numRunning" : 2,
      "instances" : [ {
        "instanceId" : 0,
        "status" : {
          "running" : true,
          "error" : "",
          "numRestarts" : 0,
          "numReceivedFromSource" : 1287,
          "numSystemExceptions" : 0,
          "latestSystemExceptions" : [ ],
          "numSourceExceptions" : 0,
          "latestSourceExceptions" : [ ],
          "numWritten" : 1287,
          "lastReceivedTime" : 1693946605095,
          "workerId" : "test"
        }
      }, {
        "instanceId" : 1,
        "status" : {
          "running" : true,
          "error" : "",
          "numRestarts" : 0,
          "numReceivedFromSource" : 909,
          "numSystemExceptions" : 0,
          "latestSystemExceptions" : [ ],
          "numSourceExceptions" : 0,
          "latestSourceExceptions" : [ ],
          "numWritten" : 909,
          "lastReceivedTime" : 1693946605023,
          "workerId" : "test"
        }
      } ]
    }
    ```
  </Tab>

  <Tab title="Terraform">
    To update the submitted connector, you only need to update the Terraform file and then call the following command.

    ```bash theme={null}
    # update the main.tf file
    terraform apply

    # output
    pulsar_source.test: Refreshing state... [id=public/default/dg-test-tf]

    Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
      ~ update in-place

    Terraform will perform the following actions:

      # pulsar_source.test will be updated in-place
      ~ resource "pulsar_source" "test" {
          ~ archive                    = "connectors/pulsar-io-data-generator-2.9.2.17.nar" -> "builtin://data-generator"
          - custom_runtime_options     = jsonencode(
                {
                  - clusterName         = "test"
                  - managed             = true
                  - maxReplicas         = 0
                  - outputTypeClassName = "org.apache.pulsar.io.datagenerator.Person"
                  - serviceAccountName  = "test-function-pulsarcluster"
                }
            ) -> null
            id                         = "public/default/dg-test-tf"
            name                       = "dg-test-tf"
          ~ parallelism                = 1 -> 2
            # (10 unchanged attributes hidden)
        }

    Plan: 0 to add, 1 to change, 0 to destroy.

    Do you want to perform these actions?
      Terraform will perform the actions described above.
      Only 'yes' will be accepted to approve.

      Enter a value: yes

    pulsar_source.test: Modifying... [id=public/default/dg-test-tf]
    pulsar_source.test: Modifications complete after 1s [id=public/default/dg-test-tf]

    Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
    ```
  </Tab>

  <Tab title="Console">
    1. On the left navigation pane of StreamNative Console, under **Resources**, click **Connectors**.

    2. On the Connectors page, select the **Created Sources** tab.

    3. Click the ellipsis at the end of the row of the connector, and then click **Edit**.

    4. Edit the configuration that you want to change, and click **SUBMIT**.
  </Tab>

  <Tab title="Rest API">
    ```bash theme={null}
    curl -X PUT https://${WEB_SERVICE_URL}/admin/v3/sources/{tenant}/{namespace}/test \
      -H 'Authorization: Bearer <API Key>' \
      -H "Content-Type: multipart/form-data" \
      -F 'sourceConfig={"name": "test", "tenant": "public", "namespace": "default", "parallelism": 2};type=application/json'
    ```

    If no error is response, the update is successful.

    And you can further check the status:

    ```bash theme={null}
    curl -X GET https://${WEB_SERVICE_URL}/admin/v3/sources/public/default/test/status \
      --header 'Authorization: Bearer <API Key>' | jq '.'
    {
      "numInstances": 2,
      "numRunning": 2,
      "instances": [
        {
          "instanceId": 0,
          "status": {
            "running": true,
            "error": "",
            "numRestarts": 0,
            "numReceivedFromSource": 1799,
            "numSystemExceptions": 0,
            "latestSystemExceptions": [],
            "numSourceExceptions": 0,
            "latestSourceExceptions": [],
            "numWritten": 1799,
            "lastReceivedTime": 1693946327331,
            "workerId": "test"
          }
        },
        {
          "instanceId": 1,
          "status": {
            "running": true,
            "error": "",
            "numRestarts": 0,
            "numReceivedFromSource": 689,
            "numSystemExceptions": 0,
            "latestSystemExceptions": [],
            "numSourceExceptions": 0,
            "latestSourceExceptions": [],
            "numWritten": 689,
            "lastReceivedTime": 1693946327129,
            "workerId": "test"
          }
        }
      ]
    }
    ```
  </Tab>
</Tabs>

## Delete a connector

The following example shows how to delete the data generator source connector `test` using different tools.

<Tabs>
  <Tab title="snctl">
    To delete the source connector `test`, use the following command.

    ```bash theme={null}
    snctl pulsar admin sources delete --tenant public --namespace default --name test
    ```

    You should see the following output:

    ```bash theme={null}
    Deleted test successfully
    ```

    If you want to verify whether the source connector has been deleted successfully, run the following command.

    ```bash theme={null}
    snctl pulsar admin sources get --tenant public --namespace default --name test
    ```

    You should see the following output:

    ```bash theme={null}
    [✖]  code: 500 reason: failed to perform the request: responseCode: 404, responseMessage: sources.compute.functionmesh.io "test-XXXXX" not found
    ```
  </Tab>

  <Tab title="pulsarctl">
    To delete the source connector `test`, use the following command.

    ```bash theme={null}
    pulsarctl sources delete --tenant public --namespace default --name test
    ```

    You should see the following output:

    ```bash theme={null}
    Deleted test successfully
    ```

    If you want to verify whether the source connector has been deleted successfully, run the following command.

    ```bash theme={null}
    pulsarctl sources get --tenant public --namespace default --name test
    ```

    You should see the following output:

    ```bash theme={null}
    [✖]  code: 500 reason: failed to perform the request: responseCode: 404, responseMessage: sources.compute.functionmesh.io "test-XXXXX" not found
    ```
  </Tab>

  <Tab title="pulsar-admin">
    To delete the source connector `test`, run the following command.

    ```bash theme={null}
    ./bin/pulsar-admin sources delete  --tenant public --namespace default --name test
    ```

    You should see the following output:

    ```bash theme={null}
    Delete source successfully
    ```

    To verify the source connector has been deleted, run the following command.

    ```bash theme={null}
    ./bin/pulsar-admin sources get --tenant public --namespace default --name test
    ```

    You should see the following output:

    ```bash theme={null}
    failed to perform the request: responseCode: 404, responseMessage: sources.compute.functionmesh.io "test-e9ef0ca6" not found
    ```
  </Tab>

  <Tab title="Terraform">
    To delete the source connector `test` with terraform, run the following command and type `yes` on the prompt.

    ```bash theme={null}
    terraform destroy

    # output
    pulsar_source.test: Refreshing state... [id=public/default/dg-test-tf]

    Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
      - destroy

    Terraform will perform the following actions:

      # pulsar_source.test will be destroyed
      - resource "pulsar_source" "test" {
          - archive                    = "connectors/pulsar-io-data-generator-2.9.2.17.nar" -> null
          - classname                  = "org.apache.pulsar.io.datagenerator.DataGeneratorSource" -> null
          - configs                    = jsonencode(
                {
                  - sleepBetweenMessages = "60"
                }
            ) -> null
          - cpu                        = 1 -> null
          - custom_runtime_options     = jsonencode(
                {
                  - clusterName         = "test"
                  - managed             = true
                  - maxReplicas         = 0
                  - outputTypeClassName = "org.apache.pulsar.io.datagenerator.Person"
                  - serviceAccountName  = "test-function-pulsarcluster"
                }
            ) -> null
          - destination_topic_name     = "public/default/dg-test" -> null
          - disk_mb                    = 10240 -> null
          - id                         = "public/default/dg-test-tf" -> null
          - name                       = "dg-test-tf" -> null
          - namespace                  = "default" -> null
          - parallelism                = 2 -> null
          - processing_guarantees      = "ATMOST_ONCE" -> null
          - ram_mb                     = 1024 -> null
          - tenant                     = "public" -> null
          - use_thread_local_producers = false -> null
        }

    Plan: 0 to add, 0 to change, 1 to destroy.

    Do you really want to destroy all resources?
      Terraform will destroy all your managed infrastructure, as shown above.
      There is no undo. Only 'yes' will be accepted to confirm.

      Enter a value: yes

    pulsar_source.test: Destroying... [id=public/default/dg-test-tf]
    pulsar_source.test: Destruction complete after 1s

    Destroy complete! Resources: 1 destroyed.
    ```
  </Tab>

  <Tab title="Console">
    1. On the left navigation pane of StreamNative Console, under **Resources**, click **Connectors**.

    2. On the Connectors page, select the **Created Sources** tab.

    3. Click the ellipsis at the end of the row of the connector, and then click **Delete**.

    4. Enter the connector name and then click **Confirm**.
  </Tab>

  <Tab title="Rest API">
    To delete the source connector `test`, use the following command.

    ```bash theme={null}
    curl -X DELETE https://${WEB_SERVICE_URL}/admin/v3/sources/public/default/test \
      --header 'Authorization: Bearer <API Key>'
    ```

    If no error is response, the delete is successful.

    If you want to verify whether the source connector has been deleted successfully, run the following command.

    ```bash theme={null}
    curl -X GET https://${WEB_SERVICE_URL}/admin/v3/sources/public/default/test/status \
      --header 'Authorization: Bearer <API Key>' | jq '.'
    ```

    You should see the following output:

    ```bash theme={null}
    {"reason":"failed to perform the request: responseCode: 404, responseMessage: sources.compute.functionmesh.io \"test-6b51d8ef\" not found"}%
    ```
  </Tab>
</Tabs>


# Monitor and Troubleshoot Connectors
Source: https://docs.streamnative.io/cloud/connect/pulsar-io/connector-monitoring



StreamNative Cloud allows you to monitor connectors status, logs, and exceptions that are thrown when a source or sink connector fails to be created, updated, or cannot work.

## View connector status

This section describes how to view connector status using `snctl`, `pulsarctl`, `pulsar-admin`, `Rest API`, and console.

<Note title="Note">
  If you want to monitor connectors using `snctl`, `pulsarctl` or `pulsar-admin`, `Rest API`, make sure you have set up your client tool. For more information, see set up client tools.
</Note>

The following example introduces how to view the status of the data generator source connector named `test`.

<Tabs>
  <Tab title="snctl">
    To check the status of the source connector `test`, run the following command:

    ```bash theme={null}
    snctl pulsar admin sources status --tenant public --namespace default --name test
    ```

    ```bash theme={null}
    {
      "numInstances": 1,
      "numRunning": 1,
      "instances": [
        {
          "instanceId": 0,
          "status": {
            "running": true,
            "error": "",
            "numRestarts": 0,
            "numReceivedFromSource": 2622,
            "numSystemExceptions": 0,
            "latestSystemExceptions": [],
            "numSourceExceptions": 0,
            "latestSourceExceptions": [],
            "numWritten": 2622,
            "lastReceivedTime": 1691532145625,
            "workerId": "test"
          }
      ]
    }
    ```
  </Tab>

  <Tab title="pulsarctl">
    To check the status of the source connector `test`, run the following command:

    ```bash theme={null}
    pulsarctl sources status --tenant public --namespace default --name test
    ```

    ```bash theme={null}
    {
      "numInstances": 1,
      "numRunning": 1,
      "instances": [
        {
          "instanceId": 0,
          "status": {
            "running": true,
            "error": "",
            "numRestarts": 0,
            "numReceivedFromSource": 2622,
            "numSystemExceptions": 0,
            "latestSystemExceptions": [],
            "numSourceExceptions": 0,
            "latestSourceExceptions": [],
            "numWritten": 2622,
            "lastReceivedTime": 1691532145625,
            "workerId": "test"
          }
      ]
    }
    ```
  </Tab>

  <Tab title="pulsar-admin">
    To check the status of the source connector `test`, run the following command:

    ```bash theme={null}
    ./bin/pulsar-admin sources status --tenant public --namespace default --name test
    ```

    You should see the following output:

    ```bash theme={null}
    {
      "numInstances" : 1,
      "numRunning" : 1,
      "instances" : [ {
        "instanceId" : 0,
        "status" : {
          "running" : true,
          "error" : "",
          "numRestarts" : 0,
          "numReceivedFromSource" : 433,
          "numSystemExceptions" : 0,
          "latestSystemExceptions" : [ ],
          "numSourceExceptions" : 0,
          "latestSourceExceptions" : [ ],
          "numWritten" : 433,
          "lastReceivedTime" : 1691452485845,
          "workerId" : "test"
        }
      } ]
    }
    ```
  </Tab>

  <Tab title="Console">
    1. From the left navigation pane, under **Resources**, click **Connectors**.

    2. Select the **Created Sinks** tab or the **Created Sources** tab to view the status and exceptions of the connector, as well as the system exceptions.
  </Tab>

  <Tab title="Rest API">
    To check the status of the source connector `test`, run the following command:

    ```bash theme={null}
    curl -X GET https://${WEB_SERVICE_URL}/admin/v3/sources/public/default/test/status \
      --header 'Authorization: Bearer <API Key>' | jq '.'
    ```

    You should see the following output:

    ```bash theme={null}
    {
      "numInstances" : 1,
      "numRunning" : 1,
      "instances" : [ {
        "instanceId" : 0,
        "status" : {
          "running" : true,
          "error" : "",
          "numRestarts" : 0,
          "numReceivedFromSource" : 433,
          "numSystemExceptions" : 0,
          "latestSystemExceptions" : [ ],
          "numSourceExceptions" : 0,
          "latestSourceExceptions" : [ ],
          "numWritten" : 433,
          "lastReceivedTime" : 1691452485845,
          "workerId" : "test"
        }
      } ]
    }
    ```
  </Tab>
</Tabs>

## View connector logs

### View the connector logs using `snctl`

This section describes how to view connector logs using `snctl`.

<Note title="Note">
  This example assumes you have [installed snctl](/tools/cli/snctl/snctl-overview#install-snctl) and [initialized snctl configurations](/tools/cli/snctl/snctl-overview#initialize-snctl-configuration).
</Note>

You can run the `snctl logs` command to view logs for a specific connector. This table outlines the configuration options that are used for viewing connector logs. For details about all supported fields, you can use the `snctl logs -h` command to list more information.

| Option                   | Descriptions                                                                                                     |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------- |
| `-c` or `--cluster`      | The name of your Pulsar cluster where the connector is created.                                                  |
| `-p` or `--component`    | The type of component to monitor. Available options are `function`, `sink`, and `source`.                        |
| `-f` or `--follow`       | Continuously list the connector log history.                                                                     |
| `-h` or `--help`         | Show usage information about the `snctl logs` command.                                                           |
| `-i` or `--instance`     | The name of your Pulsar instance where the connector is created.                                                 |
| `--name`                 | The name of your connector.                                                                                      |
| `-o` or `--organization` | The name of your organization where the connector is created.                                                    |
| `--previous`             | Print the logs that are generated before the configured timestamp.                                               |
| `--pulsar-tenant`        | The name of your Pulsar tenant where the connector is created.                                                   |
| `--pulsar-namespace`     | The name of your Pulsar namespace where the connector is created.                                                |
| `--since`                | List logs more recent than the specific time. Available units are `second`, `minute`, and `hour`, such as `24h`. |
| `-s` or `--size`         | Specify how many lines of recent logs to display.                                                                |
| `--timestamp`            | Include timestamps on each line in the log output.                                                               |

The following command example shows how to view up to 60 lines of the `data1` sink connector’s logs within the last 5 hours.

```bash theme={null}
snctl logs --since 5h --organization sndev --instance aws --cluster aws --name data1 --pulsar-tenant public --pulsar-namespace default -p sink -f -s 60
```

You should see the following output:

```bash theme={null}
2023-03-23T02:18:43,075+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [public/default/my-topic][11] Subscribed to topic on aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650 -- consumer: 0
2023-03-23T02:18:43,075+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [public/default/my-topic][11] Subscribed to topic on aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650 -- consumer: 0
2023-03-23T02:18:43,028+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [public/default/my-topic][11] Subscribing to topic on cnx [id: 0xd4f91768, L:/172.16.136.145:53364 - R:aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650], consumerId 0
2023-03-23T02:18:43,028+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [public/default/my-topic][11] Subscribing to topic on cnx [id: 0xd4f91768, L:/172.16.136.145:53364 - R:aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650], consumerId 0
2023-03-23T02:18:43,026+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xd4f91768, L:/172.16.136.145:53364 - R:aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650]] Connected to server
2023-03-23T02:18:43,026+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xd4f91768, L:/172.16.136.145:53364 - R:aws-broker-0.aws-broker-headless.sndev.svc.cluster.local/172.16.239.82:6650]] Connected to server
2023-03-23T02:18:42,951+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Pulsar client config: {"serviceUrl":"pulsar://aws-broker.sndev.svc.cluster.local:6650","authPluginClassName":"org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2","authParams":"*****","authParamMap":null,"operationTimeoutMs":30000,"lookupTimeoutMs":30000,"statsIntervalSeconds":60,"numIoThreads":1,"numListenerThreads":1,"connectionsPerBroker":1,"useTcpNoDelay":true,"useTls":false,"tlsTrustCertsFilePath":null,"tlsAllowInsecureConnection":true,"tlsHostnameVerificationEnable":false,"concurrentLookupRequest":5000,"maxLookupRequest":50000,"maxLookupRedirects":20,"maxNumberOfRejectedRequestPerConnection":50,"keepAliveIntervalSeconds":30,"connectionTimeoutMs":10000,"requestTimeoutMs":60000,"initialBackoffIntervalNanos":100000000,"maxBackoffIntervalNanos":60000000000,"enableBusyWait":false,"listenerName":null,"useKeyStoreTls":false,"sslProvider":null,"tlsTrustStoreType":"JKS","tlsTrustStorePath":null,"tlsTrustStorePassword":null,"tlsCiphers":[],"tlsProtocols":[],"memoryLimitBytes":0,"proxyServiceUrl":null,"proxyProtocol":null,"enableTransaction":false,"socks5ProxyAddress":null,"socks5ProxyUsername":null,"socks5ProxyPassword":null}
2023-03-23T02:18:42,951+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Pulsar client config: {"serviceUrl":"pulsar://aws-broker.sndev.svc.cluster.local:6650","authPluginClassName":"org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2","authParams":"*****","authParamMap":null,"operationTimeoutMs":30000,"lookupTimeoutMs":30000,"statsIntervalSeconds":60,"numIoThreads":1,"numListenerThreads":1,"connectionsPerBroker":1,"useTcpNoDelay":true,"useTls":false,"tlsTrustCertsFilePath":null,"tlsAllowInsecureConnection":true,"tlsHostnameVerificationEnable":false,"concurrentLookupRequest":5000,"maxLookupRequest":50000,"maxLookupRedirects":20,"maxNumberOfRejectedRequestPerConnection":50,"keepAliveIntervalSeconds":30,"connectionTimeoutMs":10000,"requestTimeoutMs":60000,"initialBackoffIntervalNanos":100000000,"maxBackoffIntervalNanos":60000000000,"enableBusyWait":false,"listenerName":null,"useKeyStoreTls":false,"sslProvider":null,"tlsTrustStoreType":"JKS","tlsTrustStorePath":null,"tlsTrustStorePassword":null,"tlsCiphers":[],"tlsProtocols":[],"memoryLimitBytes":0,"proxyServiceUrl":null,"proxyProtocol":null,"enableTransaction":false,"socks5ProxyAddress":null,"socks5ProxyUsername":null,"socks5ProxyPassword":null}
2023-03-23T02:18:42,939+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Starting Pulsar consumer status recorder with config: {"topicNames":["public/default/my-topic"],"topicsPattern":null,"subscriptionName":"11","subscriptionType":"Shared","subscriptionMode":"Durable","receiverQueueSize":1000,"acknowledgementsGroupTimeMicros":100000,"negativeAckRedeliveryDelayMicros":60000000,"maxTotalReceiverQueueSizeAcrossPartitions":50000,"consumerName":null,"ackTimeoutMillis":0,"tickDurationMillis":1000,"priorityLevel":0,"maxPendingChunkedMessage":10,"autoAckOldestChunkedMessageOnQueueFull":false,"expireTimeOfIncompleteChunkedMessageMillis":60000,"cryptoFailureAction":"FAIL","properties":{"application":"pulsar-sink","id":"public/default/data1-cc47863d","instance_hostname":"data1-cc47863d-sink-0","instance_id":"0"},"readCompacted":false,"subscriptionInitialPosition":"Latest","patternAutoDiscoveryPeriod":60,"regexSubscriptionMode":"PersistentOnly","deadLetterPolicy":null,"retryEnable":false,"autoUpdatePartitions":true,"autoUpdatePartitionsIntervalSeconds":60,"replicateSubscriptionState":false,"resetIncludeHead":false,"keySharedPolicy":null,"batchIndexAckEnabled":false,"ackReceiptEnabled":false,"poolMessages":false,"maxPendingChuckedMessage":10}
2023-03-23T02:18:42,939+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Starting Pulsar consumer status recorder with config: {"topicNames":["public/default/my-topic"],"topicsPattern":null,"subscriptionName":"11","subscriptionType":"Shared","subscriptionMode":"Durable","receiverQueueSize":1000,"acknowledgementsGroupTimeMicros":100000,"negativeAckRedeliveryDelayMicros":60000000,"maxTotalReceiverQueueSizeAcrossPartitions":50000,"consumerName":null,"ackTimeoutMillis":0,"tickDurationMillis":1000,"priorityLevel":0,"maxPendingChunkedMessage":10,"autoAckOldestChunkedMessageOnQueueFull":false,"expireTimeOfIncompleteChunkedMessageMillis":60000,"cryptoFailureAction":"FAIL","properties":{"application":"pulsar-sink","id":"public/default/data1-cc47863d","instance_hostname":"data1-cc47863d-sink-0","instance_id":"0"},"readCompacted":false,"subscriptionInitialPosition":"Latest","patternAutoDiscoveryPeriod":60,"regexSubscriptionMode":"PersistentOnly","deadLetterPolicy":null,"retryEnable":false,"autoUpdatePartitions":true,"autoUpdatePartitionsIntervalSeconds":60,"replicateSubscriptionState":false,"resetIncludeHead":false,"keySharedPolicy":null,"batchIndexAckEnabled":false,"ackReceiptEnabled":false,"poolMessages":false,"maxPendingChuckedMessage":10}
```

### View the connector logs from log topics

StreamNative Cloud provide an alternative implementation of the Pulsar Functions log topic(see [Pulsar Functions log topic](https://pulsar.apache.org/docs/en/functions-debug-log-topic)) feature called `sidecar` mode, which supports using such feature in Connectors.

To enable the `sidecar` mode, user should provide the following configurations when deploying the connector:

```yaml theme={null}
pulsarctl sinks/sources create --custom-runtime-options '{"logTopic":"LOG-TOPIC", "logTopicAgent":"sidecar"}' ...
```

For more details about the `custom-runtime-options`, see [Custom runtime options](/cloud/connect/pulsar-io/connector-config#stream-native-cloud-custom-runtime-options).

Once the connector is deployed with the `sidecar` mode, you can configure consumers to consume messages from the log topic.


# Connector Overview
Source: https://docs.streamnative.io/cloud/connect/pulsar-io/connector-overview



## Introduction to Pulsar IO Connectors

### Concept

Pulsar IO connectors allow you to integrate Pulsar with various external systems and data sources for both inbound and outbound data movement. For example, you can use Pulsar IO Connectors to ingest data from external sources into Pulsar topics or export data from Pulsar topics to other external systems.

Pulsar IO Connectors can be further categorized as source connectors or sink connectors:

* A source connector ingests data from an external system into Pulsar.
* A sink connector egresses data from Pulsar to an external system.

The following diagram illustrates the data movement among source connectors, Pulsar, sink connectors, and external systems.

<img alt="Pulsar IO connector" />

### Benefits

* Seamless & Simplified Data Integration: Pulsar IO Connectors provide a unified interface for connecting Pulsar to various external systems and data sources, allowing developers to easily integrate Pulsar with their existing infrastructure without the need for custom integration code.

* Extensibility: Pulsar IO Connectors are designed to be easily extensible, allowing developers to create custom connectors for specific use cases and data sources not covered by the [built-in connectors](#built-in-connectors-on-streamnative-cloud).

* Reduced Development Effort: By leveraging pre-built connectors, developers can save time and effort. They don't have to write and maintain complex integration code from scratch for each external system.

* Reliable and Scalable: Pulsar IO Connectors are built to be reliable and scalable, ensuring the data transfers between Pulsar and external systems are efficient and fault-tolerant.

### Use cases

* Data Ingestion to Pulsar: If you have data coming from various external sources, such as Apache Kafka, databases, message queues, or cloud storage systems like Amazon S3, you can use Pulsar IO Connectors to ingest that data into Pulsar topics.

* Data Export from Pulsar: Pulsar IO Connectors also enable you to export data from Pulsar topics to other systems or storage solutions. This helps you to synchronize data across different environments, replicate data, or stream data to external services or databases.

* Real-time Data Processing: Pulsar IO Connectors facilitate real-time data processing by enabling the seamless movement of data between Pulsar and other systems. This is particularly useful in event-driven architectures, streaming applications, and microservices-based solutions.

* Extending Pulsar's Functionality: If you have specific use cases or data sources not directly supported by Pulsar, you can deploy custom Pulsar IO Connectors to extend Pulsar's functionality and integrate with those systems.

### Connectors Shared Responsibility

StreamNative and our customers have a shared responsibility for maintaining and keeping connectors properly functioning. Outlined below, StreamNative has a responsibility for custom connectors that are maintained by uploading to StreamNative Cloud to maintain connectivity to the Pulsar cluster and for logging and monitoring. Customers who upload their connectors are responsible for all other operations such as configurations, updates, development, support and plugin installation. Partner connectors have similar support but the partners with StreamNative will be responsible for development and connector support. For built in connectors, StreamNative is responsible for everything except connector configuration.

For more details, see below.

<img alt="Connectors Shared Responsibility Model" />

## Built-in connectors on StreamNative Cloud

To further reduce the development overhead and time, StreamNative has pre-built a variety of Pulsar IO connectors on StreamNative Cloud. With proper configurations, you can integrate the data between your Pulsar cluster on StreamNative Cloud and your data systems effortlessly.

### Built-in source connectors

Currently, StreamNative Cloud supports the following source connectors:

* [AMQP1\_0 source connector](/connect/connectors/amqp-1-0-source/current/amqp-1-0-source)
* [AWS SQS source connector](/connect/connectors/sqs-source/current/sqs-source)
* [Google BigQuery source connector](/connect/connectors/google-bigquery-source/current/google-bigquery-source)
* [Debezium Microsoft SQL Server source connector](/connect/connectors/debezium-mssql-source/current/debezium-mssql-source)
* [Debezium MongoDB source connector](/connect/connectors/debezium-mongodb-source/current/debezium-mongodb-source)
* [Debezium MySQL source connector](/connect/connectors/debezium-mysql-source/current/debezium-MySQL-source)
* [Debezium PostgreSQL source connector](/connect/connectors/debezium-postgres-source/current/debezium-postgres-source)
* [Kafka source connector](/connect/connectors/kafka-source/current/kafka-source)
* [Kinesis source connector](/connect/connectors/kinesis-source/current/kinesis-source)
* Data generator source connector (for internal testing purposes only)

### Built-in sink connectors

Currently, StreamNative Cloud supports the following sink connectors.

* [AMQP1\_0 sink connector](/connect/connectors/amqp-1-0-sink/current/amqp-1-0-sink)
* [AWS EventBridge sink connector](/connect/connectors/aws-eventbridge-sink/current/aws-eventbridge-sink)
* [AWS Lambda sink connector](/connect/connectors/aws-lambda-sink/current/aws-lambda-sink)
* [AWS SQS sink connector](/connect/connectors/sqs-sink/current/sqs-sink)
* [AWS S3 Sink Connector](/connect/connectors/aws-s3-sink/current/aws-s3-sink)
* [Google Cloud Storage Sink Connector](/connect/connectors/google-cloud-storage-sink/current/google-cloud-storage-sink)
* [Azure Blob Storage Sink Connector](/connect/connectors/azure-blob-storage-sink/current/azure-blob-storage-sink)
* [Elasticsearch sink connector](/connect/connectors/elasticsearch-sink/current/elasticsearch-sink)
* [Google BigQuery sink connector](/connect/connectors/google-bigquery-sink/current/google-bigquery-sink)
* [Kinesis sink connector](/connect/connectors/kinesis-sink/current/kinesis-sink)
* [Snowflake Streaming sink connector](/connect/connectors/snowflake-streaming-sink/current/snowflake-streaming)
* [Snowflake sink connector](/connect/connectors/snowflake-sink/current/snowflake-sink)
* Data generator sink connector (for internal testing purposes only)

## What’s next?

* Learn how to [deploy connectors](/cloud/connect/pulsar-io/deploy-connectors/deploy-connector-index).
* Learn how to [manage connectors](/cloud/connect/pulsar-io/connector-manage).
* Learn how to [monitor connectors](/cloud/connect/pulsar-io/connector-monitoring).
* Reference [common configurations](/cloud/connect/pulsar-io/connector-config).
* Discover Pulsar Ecosystem on [StreamNative Hub](/connect/overview).
* Watch the [playlist for deploying connectors on StreamNative Cloud](https://www.youtube.com/playlist?list=PL7-BmxsE3q4WIhP4EISejpJ12YaTd6Y3h).


# Check Connector Availability
Source: https://docs.streamnative.io/cloud/connect/pulsar-io/deploy-connectors/connector-check



You can check the latest availabiltiy of built-in connectors on StreamNative Cloud using `snctl`, `pulsarctl` or `pulsar-admin`.

## Check Source Connector Availability

<Tabs>
  <Tab title="snctl">
    ```
     snctl pulsar admin sources available-sources

    +-------------------+-----------------------------------------+--------------------------------------------------------------+
    |       NAME        |             DESCRIPTION                 |                           CLASSNAME                          |
    +-------------------+-----------------------------------------+--------------------------------------------------------------+
    | amqp1_0           | AMQP1_0 connectors@2.8.2.4              | org.apache.pulsar.ecosystem.io.amqp.AmqpSource               |
    | bigquery          | Google BigQuery connectors@2.10.1.12    | org.apache.pulsar.ecosystem.io.bigquery.BigQuerySource       |
    | data-generator    | Test data generator connector@2.9.2.17  | org.apache.pulsar.io.datagenerator.DataGeneratorSource       |
    | debezium-mongodb  | Debezium MongoDb Source@2.9.2.17        | org.apache.pulsar.io.debezium.mongodb.DebeziumMongoDbSource  |
    | debezium-mssql    | Debezium Microsoft SQL Server@2.9.2.17  | org.apache.pulsar.io.debezium.mssql.DebeziumMsSqlSource      |
    | debezium-mysql    | Debezium MySql Source@2.9.2.17          | org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource      |
    | debezium-postgres | Debezium Postgres Source@2.9.2.17       | org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource|
    | kafka             | Kafka Source@2.9.2.17                   | org.apache.pulsar.io.kafka.KafkaBytesSource                  |
    | kinesis           | Kinesis connectors@2.9.2.17             | org.apache.pulsar.io.kinesis.KinesisSource                   |
    | sqs               | SQS connectors@2.9.2.18                 | org.apache.pulsar.ecosystem.io.sqs.SQSSource                 |
    +-------------------+-----------------------------------------+--------------------------------------------------------------+

    ```
  </Tab>

  <Tab title="pulsarctl">
    ```
     pulsarctl sources available-sources

    +-------------------+-----------------------------------------+--------------------------------------------------------------+
    |       NAME        |             DESCRIPTION                 |                           CLASSNAME                          |
    +-------------------+-----------------------------------------+--------------------------------------------------------------+
    | amqp1_0           | AMQP1_0 connectors@2.8.2.4              | org.apache.pulsar.ecosystem.io.amqp.AmqpSource               |
    | bigquery          | Google BigQuery connectors@2.10.1.12    | org.apache.pulsar.ecosystem.io.bigquery.BigQuerySource       |
    | data-generator    | Test data generator connector@2.9.2.17  | org.apache.pulsar.io.datagenerator.DataGeneratorSource       |
    | debezium-mongodb  | Debezium MongoDb Source@2.9.2.17        | org.apache.pulsar.io.debezium.mongodb.DebeziumMongoDbSource  |
    | debezium-mssql    | Debezium Microsoft SQL Server@2.9.2.17  | org.apache.pulsar.io.debezium.mssql.DebeziumMsSqlSource      |
    | debezium-mysql    | Debezium MySql Source@2.9.2.17          | org.apache.pulsar.io.debezium.mysql.DebeziumMysqlSource      |
    | debezium-postgres | Debezium Postgres Source@2.9.2.17       | org.apache.pulsar.io.debezium.postgres.DebeziumPostgresSource|
    | kafka             | Kafka Source@2.9.2.17                   | org.apache.pulsar.io.kafka.KafkaBytesSource                  |
    | kinesis           | Kinesis connectors@2.9.2.17             | org.apache.pulsar.io.kinesis.KinesisSource                   |
    | sqs               | SQS connectors@2.9.2.18                 | org.apache.pulsar.ecosystem.io.sqs.SQSSource                 |
    +-------------------+-----------------------------------------+--------------------------------------------------------------+

    ```
  </Tab>

  <Tab title="pulsar-admin">
    ```
     ./bin/pulsar-admin \
          --admin-url "${WEB_SERVICE_URL}" \
          --auth-plugin org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 \
          --auth-params '{"privateKey":"file:///YOUR-KEY-FILE-PATH",
              "issuerUrl":"https://auth.streamnative.cloud/",
              "audience":"urn:sn:pulsar:${orgName}:${instanceName}}'
          sources available-sources

    amqp1_0
    AMQP1_0 connectors@2.8.2.4
    ----------------------------------------
    bigquery
    Google BigQuery connectors@2.10.1.12
    sidebarTitle: Check Connector Availability
    ----------------------------------------
    data-generator
    Test data generator connector@2.9.2.17
    ----------------------------------------
    debezium-mongodb
    Debezium MongoDb Source@2.9.2.17
    sidebarTitle: Check Connector Availability
    ----------------------------------------
    debezium-mssql
    Debezium Microsoft SQL Server Source@2.9.2.17
    ----------------------------------------
    debezium-mysql
    Debezium MySql Source@2.9.2.17
    sidebarTitle: Check Connector Availability
    ----------------------------------------
    debezium-postgres
    Debezium Postgres Source@2.9.2.17
    ----------------------------------------
    kafka
    Kafka Source@2.9.2.17
    sidebarTitle: Check Connector Availability
    ----------------------------------------
    kinesis
    Kinesis connectors@2.9.2.17
    ----------------------------------------
    sqs
    SQS connectors@2.9.2.18
    sidebarTitle: Check Connector Availability
    ----------------------------------------
    ```
  </Tab>

  <Tab title="Rest API">
    ```
    curl -X GET https://${WEB_SERVICE_URL}/admin/v3/sources/builtinsources \
      --header 'Authorization: Bearer <API Key>' | jq '.'


    [
      {
        "name": "amqp1_0",
        "description": "AMQP1_0 connectors",
        "sourceClass": "org.apache.pulsar.ecosystem.io.amqp.AmqpSource",
        "sinkClass": "org.apache.pulsar.ecosystem.io.amqp.AmqpSink",
        "sourceConfigClass": "org.apache.pulsar.ecosystem.io.amqp.AmqpSourceConfig",
        "sinkConfigClass": "org.apache.pulsar.ecosystem.io.amqp.AmqpSinkConfig",
        "id": "pulsar-io-amqp1_0",
        "version": "2.8.2.4",
        "imageRepository": "streamnative/pulsar-io-amqp-1-0",
        "imageTag": "2.8.2.4",
        "typeClassName": "java.nio.ByteBuffer",
        "sourceTypeClassName": "java.nio.ByteBuffer",
        "sinkTypeClassName": "java.nio.ByteBuffer",
        "defaultSchemaType": "org.apache.pulsar.client.impl.schema.ByteBufferSchema",
        "jar": "connectors/pulsar-io-amqp1_0-2.8.2.4.nar"
      },
      {
        "name": "bigquery",
        "description": "Google BigQuery connectors",
        "sourceClass": "org.apache.pulsar.ecosystem.io.bigquery.BigQuerySource",
        "sinkClass": "org.apache.pulsar.ecosystem.io.bigquery.BigQuerySink",
        "sourceConfigClass": "org.apache.pulsar.ecosystem.io.bigquery.BigQuerySourceConfig",
        "sinkConfigClass": "org.apache.pulsar.ecosystem.io.bigquery.BigQueryConfig",
        "id": "pulsar-io-bigquery",
        "version": "3.0.1.6",
        "imageRepository": "streamnative/pulsar-io-bigquery",
        "imageTag": "3.0.1.6",
        "sourceTypeClassName": "org.apache.pulsar.client.api.schema.GenericRecord",
        "sinkTypeClassName": "org.apache.pulsar.client.api.schema.GenericObject",
        "jarFullName": "pulsar-io-bigquery-3.0.1.6.jar",
        "jar": "connectors/pulsar-io-bigquery-3.0.1.6.jar"
      },
      {
        "name": "data-generator",
        "description": "Test data generator connector",
        "sourceClass": "org.apache.pulsar.io.datagenerator.DataGeneratorSource",
        "sinkClass": "org.apache.pulsar.io.datagenerator.DataGeneratorPrintSink",
        "sourceConfigClass": "org.apache.pulsar.io.datagenerator.DataGeneratorSourceConfig",
        "id": "pulsar-io-data-generator",
        "version": "2.9.2.17",
        "imageRepository": "streamnative/pulsar-io-data-generator",
        "imageTag": "2.9.2.17",
        "typeClassName": "org.apache.pulsar.io.datagenerator.Person",
        "sourceTypeClassName": "org.apache.pulsar.io.datagenerator.Person",
        "sinkTypeClassName": "org.apache.pulsar.io.datagenerator.Person",
        "jar": "connectors/pulsar-io-data-generator-2.9.2.17.nar"
      },
      {
        "name": "debezium-mongodb",
        "description": "Debezium MongoDb Source",
        "sourceClass": "org.apache.pulsar.io.debezium.mongodb.DebeziumMongoDbSource",
        "id": "pulsar-io-debezium-mongodb",
        "version": "2.9.2.17",
        "imageRepository": "streamnative/pulsar-io-debezium-mongodb",
        "imageTag": "2.9.2.17",
        "typeClassName": "org.apache.pulsar.common.schema.KeyValue",
        "sourceTypeClassName": "org.apache.pulsar.common.schema.KeyValue",
        "sinkTypeClassName": "org.apache.pulsar.common.schema.KeyValue",
        "jar": "connectors/pulsar-io-debezium-mongodb-2.9.2.17.nar"
      }
      // omitting...
    ]

    ```
  </Tab>
</Tabs>

## Check Sink Connector Availability

<Tabs>
  <Tab title="snctl">
    ```
     snctl pulsar admin sinks available-sinks
    +-----------------+----------------------------------------+------------------------------------------------------------------+
    | NAME            | DESCRIPTION                            | CLASSNAME                                                        |
    +-----------------+----------------------------------------+------------------------------------------------------------------+
    | amqp1_0         | AMQP1_0 connectors @2.8.2.4            | org.apache.pulsar.ecosystem.io.amqp.AmqpSink                     |
    | aws-eventbridge | AWS EventBridge Sink @2.10.4.3         | org.apache.pulsar.io.eventbridge.sink.EventBridgeSink            |
    | aws-lambda      | AWS Lambda Sink @2.9.2.17              | org.apache.pulsar.ecosystem.io.aws.lambda.AWSLambdaBytesSink     |
    | bigquery        | Google BigQuery connectors @2.10.1.12  | org.apache.pulsar.ecosystem.io.bigquery.BigQuerySink             |
    | cloud-storage   | Cloud storage Sink @2.9.4.3            | org.apache.pulsar.io.jcloud.sink.CloudStorageGenericRecordSink   |
    | data-generator  | Test data generator connector @2.9.2.17| org.apache.pulsar.io.datagenerator.DataGeneratorPrintSink        |
    | elasticsearch   | Elasticsearch Sink @2.10.0.3           | org.apache.pulsar.io.elasticsearch.ElasticSearchSink             |
    | kinesis         | Kinesis connectors @2.9.2.17           | org.apache.pulsar.io.kinesis.KinesisSink                         |
    | snowflake       | Snowflake Sink @2.10.3.4               | org.apache.pulsar.ecosystem.io.snowflake.SnowflakeSinkConnector  |
    | sqs             | SQS connectors @2.9.2.18               | org.apache.pulsar.ecosystem.io.sqs.SQSSink                       |
    +-----------------+----------------------------------------+------------------------------------------------------------------+
    ```
  </Tab>

  <Tab title="pulsarctl">
    ```
     pulsarctl sinks available-sinks
    +-----------------+----------------------------------------+------------------------------------------------------------------+
    | NAME            | DESCRIPTION                            | CLASSNAME                                                        |
    +-----------------+----------------------------------------+------------------------------------------------------------------+
    | amqp1_0         | AMQP1_0 connectors @2.8.2.4            | org.apache.pulsar.ecosystem.io.amqp.AmqpSink                     |
    | aws-eventbridge | AWS EventBridge Sink @2.10.4.3         | org.apache.pulsar.io.eventbridge.sink.EventBridgeSink            |
    | aws-lambda      | AWS Lambda Sink @2.9.2.17              | org.apache.pulsar.ecosystem.io.aws.lambda.AWSLambdaBytesSink     |
    | bigquery        | Google BigQuery connectors @2.10.1.12  | org.apache.pulsar.ecosystem.io.bigquery.BigQuerySink             |
    | cloud-storage   | Cloud storage Sink @2.9.4.3            | org.apache.pulsar.io.jcloud.sink.CloudStorageGenericRecordSink   |
    | data-generator  | Test data generator connector @2.9.2.17| org.apache.pulsar.io.datagenerator.DataGeneratorPrintSink        |
    | elasticsearch   | Elasticsearch Sink @2.10.0.3           | org.apache.pulsar.io.elasticsearch.ElasticSearchSink             |
    | kinesis         | Kinesis connectors @2.9.2.17           | org.apache.pulsar.io.kinesis.KinesisSink                         |
    | snowflake       | Snowflake Sink @2.10.3.4               | org.apache.pulsar.ecosystem.io.snowflake.SnowflakeSinkConnector  |
    | sqs             | SQS connectors @2.9.2.18               | org.apache.pulsar.ecosystem.io.sqs.SQSSink                       |
    +-----------------+----------------------------------------+------------------------------------------------------------------+
    ```
  </Tab>

  <Tab title="pulsar-admin">
    ```
     ./bin/pulsar-admin \
          --admin-url "${WEB_SERVICE_URL}" \
          --auth-plugin org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 \
          --auth-params '{"privateKey":"file:///YOUR-KEY-FILE-PATH",
              "issuerUrl":"https://auth.streamnative.cloud/",
              "audience":"urn:sn:pulsar:${orgName}:${instanceName}}'
          sinks available-sinks

    amqp1_0
    AMQP1_0 connectors@2.8.2.4
    ----------------------------------------
    aws-eventbridge
    AWS EventBridge Sink@2.10.4.3
    sidebarTitle: Check Connector Availability
    ----------------------------------------
    aws-lambda
    AWS Lambda Sink@2.9.2.17
    ----------------------------------------
    bigquery
    Google BigQuery connectors@2.10.1.12
    sidebarTitle: Check Connector Availability
    ----------------------------------------
    cloud-storage
    Cloud storage Sink@2.8.4.3
    ----------------------------------------
    data-generator
    Test data generator connector@2.9.2.17
    sidebarTitle: Check Connector Availability
    ----------------------------------------
    elasticsearch
    Elasticsearch Sink@2.10.0.3
    ----------------------------------------
    kinesis
    Kinesis connectors@2.9.2.17
    sidebarTitle: Check Connector Availability
    ----------------------------------------
    snowflake
    Snowflake Sink@2.10.3.4
    ----------------------------------------
    sqs
    SQS connectors@2.9.2.18
    sidebarTitle: Check Connector Availability
    ----------------------------------------
    ```
  </Tab>

  <Tab title="Rest API">
    ```
    curl -X GET https://${WEB_SERVICE_URL}/admin/v3/sinks/builtinsinks \
      --header 'Authorization: Bearer <API Key>' | jq '.'


    [
      {
        "name": "aws-eventbridge",
        "description": "AWS EventBridge Sink",
        "sinkClass": "org.apache.pulsar.io.eventbridge.sink.EventBridgeSink",
        "sinkConfigClass": "org.apache.pulsar.io.eventbridge.sink.EventBridgeConfig",
        "id": "pulsar-io-aws-eventbridge",
        "version": "2.10.4.3",
        "imageRepository": "streamnative/pulsar-io-aws-eventbridge",
        "imageTag": "2.10.4.3",
        "typeClassName": "org.apache.pulsar.client.api.schema.GenericObject",
        "sourceTypeClassName": "org.apache.pulsar.client.api.schema.GenericObject",
        "sinkTypeClassName": "org.apache.pulsar.client.api.schema.GenericObject",
        "jar": "connectors/pulsar-io-aws-eventbridge-2.10.4.3.nar"
      },
      {
        "name": "aws-lambda",
        "description": "AWS Lambda Sink",
        "sinkClass": "org.apache.pulsar.ecosystem.io.aws.lambda.AWSLambdaBytesSink",
        "sinkConfigClass": "org.apache.pulsar.ecosystem.io.aws.lambda.AWSLambdaConnectorConfig",
        "id": "pulsar-io-aws-lambda",
        "version": "2.9.2.17",
        "imageRepository": "streamnative/pulsar-io-aws-lambda",
        "imageTag": "2.9.2.17",
        "sourceTypeClassName": "[B",
        "sinkTypeClassName": "[B",
        "jar": "connectors/pulsar-io-aws-lambda-2.9.2.17.nar"
      },
      {
        "name": "bigquery",
        "description": "Google BigQuery connectors",
        "sourceClass": "org.apache.pulsar.ecosystem.io.bigquery.BigQuerySource",
        "sinkClass": "org.apache.pulsar.ecosystem.io.bigquery.BigQuerySink",
        "sourceConfigClass": "org.apache.pulsar.ecosystem.io.bigquery.BigQuerySourceConfig",
        "sinkConfigClass": "org.apache.pulsar.ecosystem.io.bigquery.BigQueryConfig",
        "id": "pulsar-io-bigquery",
        "version": "3.0.1.6",
        "imageRepository": "streamnative/pulsar-io-bigquery",
        "imageTag": "3.0.1.6",
        "sourceTypeClassName": "org.apache.pulsar.client.api.schema.GenericRecord",
        "sinkTypeClassName": "org.apache.pulsar.client.api.schema.GenericObject",
        "jarFullName": "pulsar-io-bigquery-3.0.1.6.jar",
        "jar": "connectors/pulsar-io-bigquery-3.0.1.6.jar"
      },
      {
        "name": "cloud-storage",
        "description": "Cloud storage Sink",
        "sinkClass": "org.apache.pulsar.io.jcloud.sink.CloudStorageGenericRecordSink",
        "sinkConfigClass": "org.apache.pulsar.io.jcloud.sink.CloudStorageSinkConfig",
        "id": "pulsar-io-cloud-storage",
        "version": "2.10.2.4",
        "imageRepository": "streamnative/pulsar-io-cloud-storage",
        "imageTag": "2.10.2.4",
        "typeClassName": "org.apache.pulsar.client.api.schema.GenericRecord",
        "sourceTypeClassName": "org.apache.pulsar.client.api.schema.GenericRecord",
        "sinkTypeClassName": "org.apache.pulsar.client.api.schema.GenericRecord",
        "jar": "connectors/pulsar-io-cloud-storage-2.10.2.4.nar"
      }
      // omitting...
    ]

    ```
  </Tab>
</Tabs>

## What’s next?

* [Create connectors](/cloud/connect/pulsar-io/deploy-connectors/connector-create)


# Create Connectors
Source: https://docs.streamnative.io/cloud/connect/pulsar-io/deploy-connectors/connector-create



## Prerequisites

Before deploying a connector to StreamNative Cloud, make sure the following prerequisites have been met:

* A running external data system service.

* A running [Pulsar Cluster](/cloud/clusters/manage-clusters/cluster#create-a-cluster) on StreamNative Cloud and the [required environment](/cloud/connect/pulsar-io/deploy-connectors/connector-setup) has been set up.

* At least one of the required tools: [snctl](/tools/cli/snctl/snctl-overview), [pulsarctl](/tools/cli/pulsarctl/pulsarctl-overview), `pulsar-admin` , or the Terraform module. For a quickstart of setting up `pulsarctl` and `pulsar-admin`, see [set up client tools](/cloud/connect/pulsar-io/deploy-connectors/connector-setup#set-up-client-tools).

## Create a built-in connector

<Tip title="Tip">
  Before creating a connector, it's highly recommended to do the following:

  1. [Check connector availability](/cloud/connect/pulsar-io/deploy-connectors/connector-check) to ensure the version number of the connector you want to create is supported on StreamNative Cloud.
  2. Go to [StreamNative Hub](/connect/overview) and find the connector-specific docs of your version for configuration reference.
</Tip>

The following example shows how to create a data generator source connector named `test` on StreamNative Cloud using different tools.
The `builtin://` is followed by the name of the built-in connector, such as `builtin://data-generator`.

<Tabs>
  <Tab title="snctl">
    To create a data generator source connector named `test`, run the following command.

    ```bash theme={null}
    snctl pulsar admin sources create --archive builtin://data-generator --destination-topic-name public/default/dg-test --source-config '{"sleepBetweenMessages": 60}' --name test --sn-service-account $SERVICE_ACCOUNT
    ```

    Replace `$SERVICE_ACCOUNT` with the StreamNative Cloud service account that the connector uses as its runtime identity. To select the runtime service account interactively, use `--use-sn-service-account`.

    <Note>
      Only Pulsar 4.0.x clusters running version 4.0.10.6 or later, or Pulsar 4.2.x clusters running version 4.2.1.4 or later, support the `--sn-service-account` flag.

      You can use `--as-service-account` instead of `--sn-service-account` if you are using other versions of Pulsar clusters.
    </Note>

    You should see the following output:

    ```bash theme={null}
    Created test successfully
    ```

    If you want to verify whether the data generator source connector has been created successfully, run the following command:

    ```bash theme={null}
    snctl pulsar admin sources list
    ```

    You should see the following output:

    ```bash theme={null}
    +---------------------+
    | PULSAR SOURCES NAME |
    +---------------------+
    | test                |
    +---------------------+
    ```

    <Tip title="Tip">
      If you want to create a sink connector, use the `snctl pulsar admin sinks create` command.
    </Tip>
  </Tab>

  <Tab title="pulsarctl">
    To create a data generator source connector named `test`, run the following command.

    ```bash theme={null}
    pulsarctl sources create --archive builtin://data-generator --destination-topic-name public/default/dg-test --source-config '{"sleepBetweenMessages": 60}' --name test
    ```

    You should see the following output:

    ```bash theme={null}
    Created test successfully
    ```

    If you want to verify whether the data generator source connector has been created successfully, run the following command:

    ```bash theme={null}
    pulsarctl sources list
    ```

    You should see the following output:

    ```bash theme={null}
    +---------------------+
    | PULSAR SOURCES NAME |
    +---------------------+
    | test                |
    +---------------------+
    ```

    <Tip title="Tip">
      If you want to create a sink connector, use the `pulsarctl sinks create` command.
    </Tip>
  </Tab>

  <Tab title="pulsar-admin">
    To create a data generator source connector named `test`, run the following command.

    ```bash theme={null}
    ./bin/pulsar-admin \
        --admin-url "${WEB_SERVICE_URL}" \
        --auth-plugin org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 \
        --auth-params '{"privateKey":"file:///YOUR-KEY-FILE-PATH",
            "issuerUrl":"https://auth.streamnative.cloud/",
            "audience":"urn:sn:pulsar:${orgName}:${instanceName}}'
     sources create --archive builtin://data-generator --destination-topic-name public/default/dg-test --source-config '{"sleepBetweenMessages": 60}' --name test
    ```

    Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/connect/pulsar-io/deploy-connectors/connector-setup#set-up-client-tools).

    * `admin-url`: the HTTP service URL of your Pulsar cluster.
    * `private_key`: the path to the downloaded OAuth2 key file.
    * `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#uniform-resource-name-urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
      * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
      * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).

    If you want to list the submitted connector for a double check, run the following command:

    ```bash theme={null}
    ./bin/pulsar-admin sources list
    ```

    You should see the following output:

    ```bash theme={null}
    [
      "test"
    ]
    ```

    <Tip title="Tip">
      To reduce the complexity of your command, you can add the above parameters with values into the `conf/client.conf` file under the downloaded Pulsar release. Once it's configured, you can run a simple command instead:

      ```bash theme={null}
      ./bin/pulsar-admin sources create --archive builtin://data-generator --destination-topic-name public/default/dg-test --source-config '{"sleepBetweenMessages": 60}' --name test
      ```
    </Tip>
  </Tab>

  <Tab title="Terraform">
    1. Add the following content to your Terraform file:

    ```bash theme={null}
    terraform {
      required_providers {
        pulsar = {
          version = "0.2.0"
          source = "registry.terraform.io/streamnative/pulsar"
        }
      }
    }

    provider "pulsar" {
      web_service_url = "${WEB_SERVICE_URL}"
      audience        = "urn:sn:pulsar:${orgName}:${instanceName}"
      issuer_url      = "https://auth.streamnative.cloud/"
      key_file_path   = "file:///YOUR-KEY-FILE-PATH"
      api_version     = 3
    }

    resource "pulsar_source" "test" {
      provider = pulsar
      name      = "dg-test-tf"
      tenant    = "public"
      namespace = "default"
      archive = "builtin://data-generator"
      destination_topic_name = "public/default/dg-test"
      processing_guarantees = "ATMOST_ONCE"
      configs = "{\"sleepBetweenMessages\":\"60\"}"
    }
    ```

    2. Call the following commands in your Terraform file directory:

    ```bash theme={null}
    terraform init

    # output
    Terraform has been successfully initialized!
    ```

    ```bash theme={null}
    terraform apply

    # output
    Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
      + create

    Terraform will perform the following actions:

      # pulsar_source.test will be created
      + resource "pulsar_source" "test" {
          + archive                = "builtin://data-generator"
          + classname              = (known after apply)
          + configs                = jsonencode(
                {
                  + sleepBetweenMessages = "60"
                }
            )
          + cpu                    = 1
          + destination_topic_name = "public/default/dg-test"
          + disk_mb                = 10240
          + id                     = (known after apply)
          + name                   = "dg-test-tf"
          + namespace              = "default"
          + parallelism            = 1
          + processing_guarantees  = "ATMOST_ONCE"
          + ram_mb                 = 1024
          + tenant                 = "public"
        }

    Plan: 1 to add, 0 to change, 0 to destroy.

    Do you want to perform these actions?
      Terraform will perform the actions described above.
      Only 'yes' will be accepted to approve.

      Enter a value: yes

    pulsar_source.test: Creating...
    pulsar_source.test: Creation complete after 1s [id=public/default/dg-test-tf]

    Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
    ```

    <Tip title="Tip">
      You can use `snctl`, `pulsarctl` or `pulsar-admin` to list the source connector submitted using Terraform.
    </Tip>
  </Tab>

  <Tab title="Rest API">
    To create a data generator source connector named `test`, run the following command.

    ```bash theme={null}
    curl -X POST https://${WEB_SERVICE_URL}/admin/v3/sources/{tenant}/{namespace}/test \
      -H 'Authorization: Bearer <API Key>' \
      -H "Content-Type: multipart/form-data" \
      -F 'sourceConfig={"name": "test", "tenant": "public", "namespace": "default", "archive": "builtin://data-generator", "topicName": "public/default/dg-test", "configs": {"sleepBetweenMessages": "60"}};type=application/json'
    ```

    If you want to list the submitted connector for a double check, run the following command:

    ```bash theme={null}
    curl -X GET https://pc-ae474868.aws-use2-dixie-snc.streamnative.test.aws.sn2.dev/admin/v3/sources/public/default \
      --header 'Authorization: Bearer <API Key>'

    ["test"]
    ```
  </Tab>
</Tabs>

For all the common configurations of built-in connectors, see [Configuration reference](/cloud/connect/pulsar-io/connector-config).

## Pass sensitive configs to connector

Some connectors require sensitive information, such as passwords, token, to be passed to the connector. And you may not want to expose these sensitive information in the connector configuration.
To solve this problem, you can use the following methods to pass sensitive information to the connector:

* **Create a secret**

  For example, the AWS lambda sink connector requires the AWS access key and secret key to be passed to the connector.
  You can create a secret in the console UI and pass the secret name to the connector configuration.

  <img alt="screenshot of creating secret for AWS lambda sink" />

<Tip title="Tip">
  The `location` should be the same as the region of your Pulsar cluster.

  The `awsAccessKey` and `awsSecretKey` is the field name, and the `lambda-sink-secret` can be any unique name you want to give to the secret.
</Tip>

<Tip title="Tip">
  Only "sensitive" fields are able to load from secrets. You can get the list of sensitive fields from the connector configuration reference.
  E.g. [AWS lambda sink configurations](https://docs.streamnative.io/hub/connector-aws-lambda-sink-v3.1#configuration-properties)
</Tip>

* **Pass secrets to the connector configuration**

  The following example shows how to create an AWS lambda sink connector named `test` on StreamNative Cloud using different tools.
  The `builtin://` is followed by the name of the built-in connector, such as `builtin://data-generator`.

<Tabs>
  <Tab title="snctl">
    To create an AWS lambda sink connector named `test`, run the following command.

    ```bash theme={null}
    snctl pulsar admin sinks create --archive builtin://aws-lambda --inputs public/default/lambda-sink-test --sink-config '{"awsRegion": "us-west-2","lambdaFunctionName": "test-hello","payloadFormat": "V2"}' --secrets '{"awsAccessKey":{"path":"lambda-sink-secret","key":"awsAccessKey"},"awsSecretKey":{"path":"lambda-sink-secret","key":"awsSecretKey"}}' --name test --sn-service-account $SERVICE_ACCOUNT
    ```

    You should see the following output:

    ```bash theme={null}
    Created test successfully
    ```

    <Tip title="Tip">
      The `awsAccessKey` and `awsSecretKey` is the field name, and the `lambda-sink-secret` is the secret name you created in UI.
    </Tip>

    If you want to verify whether the AWS lambda sink connector has been created successfully, run the following command:

    ```bash theme={null}
    snctl pulsar admin sinks list
    ```

    You should see the following output:

    ```bash theme={null}
    +---------------------+
    | PULSAR SINKS NAME |
    +---------------------+
    | test                |
    +---------------------+
    ```
  </Tab>

  <Tab title="pulsarctl">
    To create an AWS lambda sink connector named `test`, run the following command.

    ```bash theme={null}
    pulsarctl sinks create --archive builtin://aws-lambda --inputs public/default/lambda-sink-test --sink-config '{"awsRegion": "us-west-2","lambdaFunctionName": "test-hello","payloadFormat": "V2"}' --secrets '{"awsAccessKey":{"path":"lambda-sink-secret","key":"awsAccessKey"},"awsSecretKey":{"path":"lambda-sink-secret","key":"awsSecretKey"}}' --name test
    ```

    You should see the following output:

    ```bash theme={null}
    Created test successfully
    ```

    <Tip title="Tip">
      The `awsAccessKey` and `awsSecretKey` is the field name, and the `lambda-sink-secret` is the secret name you created in UI.
    </Tip>

    If you want to verify whether the AWS lambda sink connector has been created successfully, run the following command:

    ```bash theme={null}
    pulsarctl sinks list
    ```

    You should see the following output:

    ```bash theme={null}
    +---------------------+
    | PULSAR SINKS NAME |
    +---------------------+
    | test                |
    +---------------------+
    ```
  </Tab>

  <Tab title="pulsar-admin">
    To create an AWS lambda sink connector named `test`, run the following command.

    ```bash theme={null}
    ./bin/pulsar-admin \
        --admin-url "${WEB_SERVICE_URL}" \
        --auth-plugin org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 \
        --auth-params '{"privateKey":"file:///YOUR-KEY-FILE-PATH",
            "issuerUrl":"https://auth.streamnative.cloud/",
            "audience":"urn:sn:pulsar:${orgName}:${instanceName}}'
     sinks create --archive builtin://aws-lambda --inputs public/default/lambda-sink-test --sink-config '{"awsRegion": "us-west-2","lambdaFunctionName": "test-hello","payloadFormat": "V2"}' --secrets '{"awsAccessKey":{"path":"lambda-sink-secret","key":"awsAccessKey"},"awsSecretKey":{"path":"lambda-sink-secret","key":"awsSecretKey"}}' --name test
    ```

    Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/connect/pulsar-io/deploy-connectors/connector-setup#set-up-client-tools).

    * `admin-url`: the HTTP service URL of your Pulsar cluster.
    * `private_key`: the path to the downloaded OAuth2 key file.
    * `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#uniform-resource-name-urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
      * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
      * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).

    If you want to list the submitted connector for a double check, run the following command:

    ```bash theme={null}
    ./bin/pulsar-admin sinks list
    ```

    You should see the following output:

    ```bash theme={null}
    [
      "test"
    ]
    ```

    <Tip title="Tip">
      The `awsAccessKey` and `awsSecretKey` is the field name, and the `lambda-sink-secret` is the secret name you created in UI.
    </Tip>

    <Tip title="Tip">
      To reduce the complexity of your command, you can add the above parameters with values into the `conf/client.conf` file under the downloaded Pulsar release. Once it's configured, you can run a simple command instead:

      ```bash theme={null}
      ./bin/pulsar-admin sinks create --archive builtin://aws-lambda --inputs public/default/lambda-sink-test --sink-config '{"awsRegion": "us-west-2","lambdaFunctionName": "test-hello","payloadFormat": "V2"}' --secrets '{"awsAccessKey":{"path":"lambda-sink-secret","key":"awsAccessKey"},"awsSecretKey":{"path":"lambda-sink-secret","key":"awsSecretKey"}}' --name test
      ```
    </Tip>
  </Tab>

  <Tab title="Terraform">
    * **Add the following content to your Terraform file:**

    ```bash theme={null}
    terraform {
      required_providers {
        pulsar = {
          version = "0.2.0"
          source = "registry.terraform.io/streamnative/pulsar"
        }
      }
    }

    provider "pulsar" {
      web_service_url = "${WEB_SERVICE_URL}"
      audience        = "urn:sn:pulsar:${orgName}:${instanceName}"
      issuer_url      = "https://auth.streamnative.cloud/"
      key_file_path   = "file:///YOUR-KEY-FILE-PATH"
      api_version     = 3
    }

    resource "pulsar_sink" "test" {
      provider = pulsar
      name      = "lambda-sink-test-tf"
      tenant    = "public"
      namespace = "default"
      archive = "builtin://aws-lambda"
      auto_ack = true
      cleanup_subscription = true
      destination_topic_name = "public/default/lambda-sink-test"
      processing_guarantees = "ATMOST_ONCE"
      configs = "{\"awsRegion\":\"us-west-2\",\"lambdaFunctionName\":\"test-hello\",\"payloadFormat\":\"V2\"}"
      secrets = "{\"awsAccessKey\":{\"path\":\"lambda-sink-secret\",\"key\":\"awsAccessKey\"},\"awsSecretKey\":{\"path\":\"lambda-sink-secret\",\"key\":\"awsSecretKey\"}}"
    }
    ```

    * **Call the following commands in your Terraform file directory:**

    ```bash theme={null}
    terraform init

    # output
    Terraform has been successfully initialized!
    ```

    ```bash theme={null}
    terraform apply

    # output
    Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
      + create

    Terraform will perform the following actions:

      # pulsar_sink.test will be created
      + resource "pulsar_sink" "test" {
          + archive               = "builtin://aws-lambda"
          + auto_ack              = true
          + classname             = (known after apply)
          + cleanup_subscription  = true
          + configs               = jsonencode(
                {
                  + awsRegion          = "us-west-2"
                  + lambdaFunctionName = "test-hello"
                  + payloadFormat      = "V2"
                }
            )
          + cpu                   = 1
          + disk_mb               = 10240
          + id                    = (known after apply)
          + inputs                = [
              + "public/default/lambda-sink-test",
            ]
          + name                  = "lambda-sink-test-tf"
          + namespace             = "default"
          + parallelism           = 1
          + processing_guarantees = "ATMOST_ONCE"
          + ram_mb                = 1024
          + retain_ordering       = true
          + secrets               = jsonencode(
                {
                  + awsAccessKey = {
                      + key  = "awsAccessKey"
                      + path = "lambda-sink-secret"
                    }
                  + awsSecretKey = {
                      + key  = "awsSecretKey"
                      + path = "lambda-sink-secret"
                    }
                }
            )
          + subscription_position = "Earliest"
          + tenant                = "public"
        }

    Plan: 1 to add, 0 to change, 0 to destroy.

    Do you want to perform these actions?
      Terraform will perform the actions described above.
      Only 'yes' will be accepted to approve.

      Enter a value: yes

    pulsar_sink.test: Creating...
    pulsar_sink.test: Creation complete after 5s [id=public/default/lambda-sink-test-tf]

    Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
    ```

    <Tip title="Tip">
      The `awsAccessKey` and `awsSecretKey` is the field name, and the `lambda-sink-secret` is the secret name you created in UI.
    </Tip>

    <Tip title="Tip">
      You can use `snctl`, `pulsarctl` or `pulsar-admin` to list the source connector submitted using Terraform.
    </Tip>
  </Tab>

  <Tab title="Rest API">
    To create an AWS lambda sink connector named `test`, run the following command.

    ```bash theme={null}
    curl -X POST https://${WEB_SERVICE_URL}/admin/v3/sinks/{tenant}/{namespace}/test \
      -H 'Authorization: Bearer <API Key>' \
      -H "Content-Type: multipart/form-data" \
      -F 'sinkConfig={"name": "test", "tenant": "public", "namespace": "default", "archive": "builtin://aws-lambda", "inputs": ["public/default/lambda-sink-test"], "configs": {"awsRegion": "us-west-2","lambdaFunctionName": "test-hello", "payloadFormat": "V2"}, "secrets": {"awsAccessKey":{"path":"lambda-sink-secret","key":"awsAccessKey"},"awsSecretKey":{"path":"lambda-sink-secret","key":"awsSecretKey"}}};type=application/json'
    ```

    If you want to list the submitted connector for a double check, run the following command:

    ```bash theme={null}
    curl -X GET https://pc-ae474868.aws-use2-dixie-snc.streamnative.test.aws.sn2.dev/admin/v3/sinks/public/default \
      --header 'Authorization: Bearer <API Key>'

    ["test"]
    ```

    <Tip title="Tip">
      The `awsAccessKey` and `awsSecretKey` is the field name, and the `lambda-sink-secret` is the secret name you created in UI.
    </Tip>
  </Tab>
</Tabs>

For all the common configurations of built-in connectors, see [Configuration reference](/cloud/connect/pulsar-io/connector-config).

## Create a custom connector

<Tip title="Tip">
  Before creating a connector, it's highly recommended to do the following:

  1. [Check connector availability](/cloud/connect/pulsar-io/deploy-connectors/connector-check) to ensure the version number of the connector you want to create is supported on StreamNative Cloud.
  2. Go to [StreamNative Hub](/connect/overview) and find the connector-specific docs of your version for configuration reference.
</Tip>

To create a custom Pulsar Connector, you need to upload the connector jar/nar file to the StreamNative Cloud Package service first. Below are the steps:

### Upload your connector file to Pulsar

<Tabs>
  <Tab title="snctl">
    Upload packages

    ```bash theme={null}
    snctl pulsar admin packages upload sink://public/default/custom-connect@v1 \
    --path /tmp/your-connector.jar \
    --description "custom connector" \
    --properties fileName=your-connector.jar
    ```

    You should see the following output:

    ```bash theme={null}
    The package 'sink://public/default/custom-connect@v1' uploaded from path '/tmp/your-connector.jar' successfully
    ```

    <Tip title="Tip">
      You can also upload your package to `source://${tenant}/${namespace}/${name}@{$version}`, currently Pulsar Package Service supports below protocols:

      * `source://`
      * `sink://`
      * `function://`
    </Tip>
  </Tab>

  <Tab title="Pulsarctl">
    You need to set the context for Pulsarctl first:

    ```bash theme={null}
    # create a context
    pulsarctl context set ${context-name}  \
    --admin-service-url ${admin-service-url} \
    --issuer-endpoint ${issuerUrl} \
    --audience urn:sn:pulsar:${orgName}:${instanceName} \
    --key-file ${privateKey}

    # activate oauth2
    pulsarctl oauth2 activate
    ```

    <Note title="Note">
      Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).

      * `context-name`: any name you want
      * `admin-service-url`: the HTTP service URL of your Pulsar cluster.
      * `privateKey`: the path to the downloaded OAuth2 key file.
      * `issuerUrl`: the URL of the OAuth2 issuer.
      * `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#uniform-resource-name-urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
        * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
        * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
    </Note>

    Upload packages

    ```bash theme={null}
    pulsarctl packages upload sink://public/default/custom-connect@v1 \
    --path /tmp/your-connector.jar \
    --description "custom connector" \
    --properties fileName=your-connector.jar
    ```

    You should see the following output:

    ```bash theme={null}
    The package 'sink://public/default/custom-connect@v1' uploaded from path '/tmp/your-connector.jar' successfully
    ```

    <Tip title="Tip">
      You can also upload your package to `source://${tenant}/${namespace}/${name}@{$version}`, currently Pulsar Package Service supports below protocols:

      * `source://`
      * `sink://`
      * `function://`
    </Tip>
  </Tab>

  <Tab title="Pulsar-admin">
    ```bash theme={null}
    ./bin/pulsar-admin \
        --admin-url "${WEB_SERVICE_URL}" \
        --auth-plugin org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 \
        --auth-params '{"privateKey":"file://${privateKey}","issuerUrl":"${issuerUrl}","audience":"urn:sn:pulsar:${orgName}:${instanceName}}' \
        packages upload sink://public/default/custom-connect@v1 \
        --path /tmp/your-connector.jar \
        --description "custom connector" \
        --properties fileName=your-connector.jar
    ```

    <Note title="Note">
      Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).

      * `admin-url`: the HTTP service URL of your Pulsar cluster.
      * `privateKey`: the path to the downloaded OAuth2 key file.
      * `issuerUrl`: the URL of the OAuth2 issuer.
      * `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#uniform-resource-name-urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
        * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
        * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
    </Note>

    You should see the following output:

    ```bash theme={null}
    The package 'sink://public/default/custom-connect@v1' uploaded from path '/tmp/your-connector.jar' successfully
    ```

    <Tip title="Tip">
      You can also upload your package to `source://${tenant}/${namespace}/${name}@{$version}`, currently Pulsar Package Service supports below protocols:

      * `source://`
      * `sink://`
      * `function://`
    </Tip>
  </Tab>
</Tabs>

To create a custom connector, just replace the `archive` argument to the package URL(like `sink://public/default/custom-connect@v1`) you uploaded.


# Set up Your Environment
Source: https://docs.streamnative.io/cloud/connect/pulsar-io/deploy-connectors/connector-setup



This section introduces how to set up a new service account with the minimum permissions to run connectors. To perform the following operations, you need to be the cluster administrator beforehand.

## Create a service account for Pulsar users

1. On the left navigation pane of StreamNative Cloud Console, click **Service Accounts**.

2. Click **Create Service Account**.

3. Enter a name for the service account, and then click **Confirm**.

<Note title="Note">
  Do **NOT** check the **Super Admin** option when creating this service account.
</Note>

## Authorize the service account

To make the service account work, you need to make the service account granted with proper permissions (`sinks`, `sources`, `packages`, `produce`, and `consume`).

To grant the service account permissions on the namespace level, follow these steps:

<Tabs>
  <Tab title="Cloud Console">
    1. Navigate to the **Namespace Dashboard** page by [switching to the namespace workspace](/cloud/get-started/cloud-console#switch-a-namespace).

    2. On the **Namespace Dashboard** page, click **Configuration** on the left navigation pane.

    3. On the **Namespace configuration** page, click **ADD ROLE**, and select the service account that you want to authorize.

    4. On the drop-down menu below the service account, select the proper permissions to assign to the service account. There are six permissions in total:

    * `consume`: allow the service account to consume messages.
    * `produce`: allow the service account to publish messages.
    * `functions`: allow the service account to submit and manage functions.
    * `sinks`: allow the service account to create and manage sink connectors.
    * `sources`: allow the service account to create and manage source connectors.
    * `packages`: allow the service account to upload and manage pulsar packages. If you want to submit a customized function/connector, then you will need to upload the function/connector’s JAR/NAR/Python file first, which requires the `packages` permission.

      <img alt="Authorize Service Account" />
  </Tab>
</Tabs>

## Grant access to the service account

To grant the underlying infrastructure with access to the newly created service account's OAuth2 key file, you need to create a service account binding via UI.

Go to the `Service Accounts` tab and choose the service account you want to use for running the connector. Clicking on the right button and there will be a `Edit service account bindings` option.

<img alt="Binding Service Account step-1" />

Click the `Edit service account bindings`, choose the desired pool member and confirm.

<img alt="Binding Service Account step-2" />

You can also enable the `Enable IAM Role Creation` option to create a separate IAM role for the service account.

Now your connector is ready to use the service account in StreamNative environments.

## (Optional) Create a separate IAM role for the service account

StreamNative's I/O components (Pulsar Functions, Pulsar Connectors, and Kafka Connectors) run as cloud-native workloads on AWS, GCP, and Azure infrastructures.

Use the cloud providers' native IAM (Identity and Access Management) services to control access to infrastructure resources such as S3, GCS, and Azure Blob Storage. This removes the need for password-based authentication for the service accounts that run connectors in StreamNative Cloud.

By default, StreamNative uses a single service account per `PulsarCluster` for all I/O components (Pulsar Functions, Pulsar Connectors, and Kafka Connectors) to access underlying infrastructure resources. This means all I/O Components in the same cluster share one service account and the same permissions.

<Warning>
  Using a single service account for all I/O components means all functions and connectors share the same permissions. If one component is compromised, it could access resources intended for other components.

  So it's better to leave the default service account with no permissions and use it for running IO components that do not require access to external resources.

  And create separate service accounts with the minimum required permissions for each IO component that need to access external resources.
</Warning>

To enhance security and improve isolation, you can create a separate IAM role for each service account used to run connectors. This let you grant only the permissions each service account needs.

Create a separate IAM role for your service account:

<Tabs>
  <Tab title="Snctl">
    <Note title="Note">
      This feature is available in [snctl](/tools/cli/snctl/snctl-overview) v1.3.0 or later.
    </Note>

    1. Get the `PoolMember` name and namespace from the `PulsarCluster`

       ```shell theme={null}
       snctl get pulsarcluster <cluster-id> -o yaml
       ```

       In the output, find the `poolMemberRef` block, which looks like:

       ```yaml theme={null}
       poolMemberRef:
         name: <pool-member-name>
         namespace: <namespace>
       ```

       <Note title="Note">
         Multiple clusters may be located in the same `PoolMember`. You do not need to create separate IAM roles for each cluster within the same `PoolMember`.
       </Note>

    2. Create a new `ServiceAccountBinding` that binds the service account to the `PoolMember`

       ```shell theme={null}
       snctl create serviceaccountbinding <binding-name> \
         --pool-member <pool-member-namespace>/<pool-member-name> \
         --enable-iam-account-creation \
         --service-account-name <service-account-name>
       ```

       **AWS only:** You can specify one or more `AWS Assume Role ARNs` that can be assumed by the IAM role created for the service account (repeat the flag for each ARN):

       ```shell theme={null}
       snctl create serviceaccountbinding <binding-name> \
         --pool-member <pool-member-namespace>/<pool-member-name> \
         --enable-iam-account-creation \
         --service-account-name <service-account-name> \
         --aws-assume-role-arns <arn1> \
         --aws-assume-role-arns <arn2>
       ```

       <Note title="Note">
         The IAM role created by StreamNative will include permissions to call `sts:AssumeRole` on `arn1` and `arn2`. You must still add a trust policy on `arn1` and `arn2` to allow the newly created role to assume them, an example likes below:

         ```json theme={null}
         {
             "Version": "2012-10-17",
             "Statement": [
                 {
                     "Effect": "Allow",
                     "Principal": {
                         "AWS": "arn:aws:iam::[aws-account-id]:role/StreamNative/sncloud-role/authorization.streamnative.io/iamaccounts/IamAccount-[org-id]-sab-[binding-name]"
                     },
                     "Action": "sts:AssumeRole"
                 }
             ]
         }
         ```
       </Note>

    3. (Optional) Update an existing `ServiceAccountBinding` to create the IAM role

       ```shell theme={null}
       snctl edit serviceaccountbinding <binding-name>
       ```

    4. Verify the IAM role was created successfully

       ```shell theme={null}
       snctl get serviceaccountbinding <binding-name> -o yaml
       ```

       Expected output (example):

       ```yaml theme={null}
       apiVersion: cloud.streamnative.io/v1alpha1
       kind: ServiceAccountBinding
       metadata:
         creationTimestamp: "2025-06-25T08:45:35Z"
         finalizers:
         - serviceaccountbinding.finalizers.cloud.streamnative.io
         generation: 1
         name: test-admin
         namespace: o-lftqu
         ownerReferences:
         - apiVersion: cloud.streamnative.io/v1alpha1
           kind: ServiceAccount
           name: admin
           uid: 4ef639aa-6278-4863-9a23-f1da50cea448
         resourceVersion: "54707713"
         uid: 918d40ad-551d-416b-a2ae-d41548d6608e
       spec:
         enableIamRoleCreation: true
         poolMemberRef:
           name: azure-eastus-zephyr
           namespace: streamnative
         serviceAccountName: admin
       status:
         conditions:
         - lastTransitionTime: "2025-06-25T08:45:35Z"
           status: "True"
           type: IAMAccountReady
         - lastTransitionTime: "2025-06-25T08:45:35Z"
           status: "True"
           type: ServiceAccountReady
         - lastTransitionTime: "2025-07-16T13:56:17Z"
           status: "True"
           type: ResourceExists
         - lastTransitionTime: "2025-07-16T13:56:17Z"
           status: "True"
           type: PoolMemberReady
         - lastTransitionTime: "2025-07-16T13:56:17Z"
           reason: AllConditionStatusTrue
           status: "True"
           type: Ready
       ```

       In the output, the `status.conditions` array should include a condition with `type: IAMAccountReady` and `status: "True"`, indicating the IAM role was created successfully.<br /><br />

    5. Use the service account when creating I/O components

       <br /><br />You can now select this service account in Console (or use its API key with the CLI) when creating I/O components (Pulsar Functions, Pulsar Connectors, and Kafka Connectors). The components will inherit the permissions granted to the IAM role created in the previous step.
  </Tab>

  <Tab title="Console">
    You can also create a separate IAM role for your service account via StreamNative Cloud Console. Just enable the `Enable IAM Role Creation` option when creating or editing a service account binding.

    <img alt="Binding Service Account step-2" />

    **AWS only:** You can specify one or more `AWS Assume Role ARNs` that can be assumed by the IAM role created for the service account (use one line for each ARN)

    <img alt="Binding Service Account step-3" />

    <Note title="Note">
      The IAM role created by StreamNative will include permissions to call `sts:AssumeRole` on `arn1` and `arn2`. You must still add a trust policy on `arn1` and `arn2` to allow the newly created role to assume them, an example likes below:

      ```json theme={null}
      {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Principal": {
                      "AWS": "arn:aws:iam::[aws-account-id]:role/StreamNative/sncloud-role/authorization.streamnative.io/iamaccounts/IamAccount-[org-id]-sab-[binding-name]"
                  },
                  "Action": "sts:AssumeRole"
              }
          ]
      }
      ```
    </Note>
  </Tab>
</Tabs>

<Note title="Note">
  In Azure, a Managed Identity `sab-[binding-name]-[org-id]` is created;

  In AWS, an IAM role `role/StreamNative/sncloud-role/authorization.streamnative.io/iamaccounts/IamAccount-[org-id]-sab-[binding-name]` is created;

  In GCP, a service account with display name: `IamAccount/[org-id]/sab-[binding-name]` is created.
</Note>

## Set up client tools

### Use `snctl` to manage Pulsar IO connectors with a service account

The StreamNative CLI `snctl` includes Pulsar admin commands that connect directly to your StreamNative Cloud cluster. You can use a service account for the admin request, the connector runtime identity, or both.

1. Use `snctl config set --organization $ORG` to set your StreamNative Cloud organization.
2. Use `snctl context use` to select your target StreamNative Cloud cluster interactively.
3. To send the admin request as a service account, use `snctl pulsar admin --as-service-account $SERVICE_ACCOUNT_NAME ...` or `snctl pulsar admin --use-service-account ...`.
4. To run a connector as a service account, use `--sn-service-account $SERVICE_ACCOUNT_NAME` on `snctl pulsar admin sources create`, `snctl pulsar admin sources update`, `snctl pulsar admin sinks create`, or `snctl pulsar admin sinks update`. To select the runtime service account interactively, use `--use-sn-service-account`.

<Note title="Tip">
  `--as-service-account` and `--sn-service-account` set different identities.

  Use `--as-service-account` to send the request with the specified service account's credentials. That same service account also becomes the runtime identity of the connector. Therefore, the service account must have permissions to create or update connectors, download packages, and produce or consume messages.

  Use `--sn-service-account` to keep the request authenticated as the current caller, but run the connector with the specified service account as its runtime identity. In this case, the caller must have permission to create or update the connector and use the selected service account (see [account-admin](/cloud/security/access/rbac/manage-rbac-roles#account-admin)). The runtime service account only needs the permissions required by the connector itself, such as producing or consuming messages and downloading packages.

  You can use `--as-service-account` on Pulsar clusters of any supported version. The `--sn-service-account` and `--use-sn-service-account` flags require Pulsar 4.0.x version 4.0.10.6 or later, or Pulsar 4.2.x version 4.2.1.4 or later.
</Note>

### Use `pulsarctl`, `pulsar-admin`, or the REST API to manage Pulsar IO connectors with a service account

StreamNative Cloud Console provides a step-by-step wizard to walk you through the basic client setup process. You can connect your Pulsar client that uses the previously created service account to interact with your Pulsar cluster.

1. On the left navigation pane of StreamNative Cloud Console, in the **Admin** section, click **Pulsar Clients**.

2. Select the **CLI Tools** tab and follow the wizard to generate the sample code you need for connecting to your Pulsar cluster. The steps may vary depending on the tool you use.

   <img alt="Set up client tools" />

   a. Select `pulsarctl`, `pulsar-admin`, or the REST API.

   b. Download the selected CLI tool.

   c. Select the service account you created.

   d. Select **OAuth2** as the authentication type and download the key file to your local machine.

   e. Set up your CLI tool with that key file, and the steps vary depending on the CLI tool you use.

<Tabs>
  <Tab title="pulsarctl">
    f. Copy the command for setting client configurations to your terminal, update the path of the OAuth2 key file, and run it.

    g. Select the target tenant, namespace and topic, and copy the sample command to run.
  </Tab>

  <Tab title="pulsar-admin">
    f. Select the target tenant, namespace and topic, and copy the sample command to your terminal and update the path of the OAuth2 key file before running.
  </Tab>

  <Tab title="Rest API">
    f. Select the target tenant, namespace and topic, and copy the sample `curl` command to your terminal and update the path of the OAuth2 key file before running.
  </Tab>
</Tabs>

## What’s next?

* [Check connector availability](/cloud/connect/pulsar-io/deploy-connectors/connector-check)


# Deploy connectors
Source: https://docs.streamnative.io/cloud/connect/pulsar-io/deploy-connectors/deploy-connector-index



Pulsar provides a pluggable architecture for connectors, allowing you to deploy pre-built connectors and develop custom connectors for external systems, such as databases, messaging systems, block storage, cloud services, or any other system capable of producing or consuming data.

To deploy connectors on StreamNative Cloud, follow the following instructions.

* [Set up your environment](/cloud/connect/pulsar-io/deploy-connectors/connector-setup)
* [Check connector availability](/cloud/connect/pulsar-io/deploy-connectors/connector-check)
* [Create connectors](/cloud/connect/pulsar-io/deploy-connectors/connector-create)


# Connect to your cluster using Kafka Connect
Source: https://docs.streamnative.io/cloud/connect/self-host-connectors/cloud-connect-elasticsearch



<Note title="Note">
  * This QuickStart assumes that you have created a StreamNative cluster with the Kafka protocol enabled, created a service account, and granted the service account `produce` and `consume` permissions to a namespace for the target topic.
  * This QuickStart is developed on a MAC Operating System (OS) environment. If you choose another OS, the commands might vary.
</Note>

This document shows how to connect to your StreamNative cluster using [Kafka Connect](https://docs.confluent.io/platform/current/connect/index.html) with OAuth2 authentication.

## Before you begin

* Get the OAuth2 credential file.

  1. On the left navigation pane, click **Service Accounts**.
  2. In the row of the service account you want to use, in the **Key File** column, click the **Download** icon to download the OAuth2 credential file to your local directory.

* Get the service URL of your StreamNative cluster.

  1. On the left navigation pane, in the **Admin** area, click **Pulsar Clusters**.
  2. Select the **Details** tab, and in the **Access Points** area, click **Copy** at the end of the row of the **Kafka Service URL (TCP)**.

- Install a recent version of [Docker](https://docs.docker.com/get-docker/).
- [Download the `jq` CLI JSON processor](https://stedolan.github.io/jq/download/), which provides a JSON-format output.

## Configure Elasticsearch

1. Open a terminal, and run the command below to start an Elasticsearch instance with ports `9200` and `9300`.

   ```bash theme={null}
   docker run --name elastic-1 \
     -p 9200:9200 -p 9300:9300 -it \
     -e "discovery.type=single-node" \
     -e "xpack.security.enabled=false" \
     docker.elastic.co/elasticsearch/elasticsearch:7.17.2
   ```

   You should see the following output:

   ```bash theme={null}
   ...output omitted...
   {"@timestamp":"2022-04-15T15:13:17.067Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-Country.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[efb7b3360ba3][generic][T#7]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"ocHgh5mAQROAlUofYHE3Cg","elasticsearch.node.id":"0aWiWmaBTgC0vdp6Zw_ZnQ","elasticsearch.node.name":"efb7b3360ba3","elasticsearch.cluster.name":"docker-cluster"}
   {"@timestamp":"2022-04-15T15:13:17.118Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-City.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[efb7b3360ba3][generic][T#13]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"ocHgh5mAQROAlUofYHE3Cg","elasticsearch.node.id":"0aWiWmaBTgC0vdp6Zw_ZnQ","elasticsearch.node.name":"efb7b3360ba3","elasticsearch.cluster.name":"docker-cluster"}
   ```

2. Verify that the Elasticsearch instance is started successfully.

   ```bash theme={null}
   curl 'http://localhost:9200'
   ```

   You should see the following output:

   ```json theme={null}
   {
     "name": "eaf2be7fe2d6",
     "cluster_name": "docker-cluster",
     "cluster_uuid": "jolOS3_VRGq2-LpZbkT1kw",
     "version": {
       "number": "7.17.2",
       "build_flavor": "default",
       "build_type": "docker",
       "build_hash": "de7261de50d90919ae53b0eff9413fd7e5307301",
       "build_date": "2022-03-28T15:12:21.446567561Z",
       "build_snapshot": false,
       "lucene_version": "8.11.1",
       "minimum_wire_compatibility_version": "6.8.0",
       "minimum_index_compatibility_version": "6.0.0-beta1"
     },
     "tagline": "You Know, for Search"
   }
   ```

## Configure Kafka Connect

[Kafka Connect](https://docs.confluent.io/platform/current/connect/index.html) is an integration tool that is released with the Apache Kafka project. It provides reliable data streaming between Apache Kafka and external systems and is both scalable and flexible. Kafka Connect works with Kafka on Pulsar (KoP), which is compatible with the Kafka API.

Kafka Connect uses Source and Sink connectors for integration. Source connectors stream data from an external system to Kafka, while Sink connectors stream data from Kafka to an external system.

<img alt="Pulsar KoP and elasticsearch with kafka connect" />

1. Navigate to the [Apache downloads page for Kafka](https://www.apache.org/dyn/closer.cgi?path=/kafka/3.1.0/kafka_2.13-3.1.0.tgz), and click the suggested download link for the Kafka 3.1.0 binary package.

2. Extract the Kafka binaries folder in the `_YOUR_HOME_DIRECTORY_/kafka_2.13-3.1.0` directory that you created earlier.

### Download the StreamNative OAuth dependency

```bash theme={null}
# download supplementary libraries
curl -O https://repo1.maven.org/maven2/io/streamnative/pulsar/handlers/oauth-client/3.1.0.1/oauth-client-3.1.0.1.jar --output-dir ./libs
```

### Download the Elasticsearch Sink connector

1. Create a `connectors` folder to store all Kafka connectors.

   ```bash theme={null}
   # switch to your kafka folder
   cd kafka_2.13-3.1.0
   # create a connector folder to store all Kafka connectors
   mkdir connectors
   ```

2. Navigate to the [Elasticsearch Sink connector](https://www.confluent.io/hub/confluentinc/kafka-connect-elasticsearch) and click **Download** to download the archived binaries.

3. Extract the file and copy the unzipped folder into the `connectors` directory.

### Configure Kafka Connect

1. Create a configuration file (named `connect-sn-kop.properties`).

   ```bash theme={null}
   # switch to your kafka folder
   cd kafka_2.13-3.1.0
   # create a connect configuration file
   # which contains information of Kafka server (StreamNative KoP cluster)
   vim conf/connect-sn-kop.properties
   ```

2. Add the following content to the `connect-sn-kop.properties` file.

   ```conf theme={null}
   # add the information of StreamNative KoP cluster
   bootstrap.servers="SERVER-URL"

   sasl.login.callback.handler.class=io.streamnative.pulsar.handlers.kop.security.oauth.OauthLoginCallbackHandler
   security.protocol=SASL_SSL
   sasl.mechanism=OAUTHBEARER
   sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule \
     required oauth.issuer.url="https://auth.streamnative.cloud/"\
     oauth.credentials.url="file://YOUR-KEY-FILE-PATH"\
     oauth.audience="YOUR-AUDIENCE-STRING";

   producer.sasl.login.callback.handler.class=io.streamnative.pulsar.handlers.kop.security.oauth.OauthLoginCallbackHandler
   producer.security.protocol=SASL_SSL
   producer.sasl.mechanism=OAUTHBEARER
   producer.sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule \
     required oauth.issuer.url="https://auth.streamnative.cloud/"\
     oauth.credentials.url="file://YOUR-KEY-FILE-PATH"\
     oauth.audience="YOUR-AUDIENCE-STRING";

   consumer.sasl.login.callback.handler.class=io.streamnative.pulsar.handlers.kop.security.oauth.OauthLoginCallbackHandler
   consumer.security.protocol=SASL_SSL
   consumer.sasl.mechanism=OAUTHBEARER
   consumer.sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule \
     required oauth.issuer.url="https://auth.streamnative.cloud/"\
     oauth.credentials.url="file://YOUR-KEY-FILE-PATH"\
     oauth.audience="YOUR-AUDIENCE-STRING";

   #Cluster level converters
   #These apply when the connectors don't define any converter
   key.converter=org.apache.kafka.connect.json.JsonConverter
   value.converter=org.apache.kafka.connect.json.JsonConverter

   #JSON schemas enabled to false in cluster level
   key.converter.schemas.enable=true
   value.converter.schemas.enable=true

   #Where to keep the Connect topic offset configurations
   offset.storage.file.filename=/tmp/connect.offsets
   offset.flush.interval.ms=10000

   #Plugin path to put the connector binaries
   plugin.path=YOUR-FULL-PATH/connectors/
   ```

   * `oauth.credentials.url`: the path to your downloaded OAuth2 credential file.
   * `bootstrap.servers`: the Kafka service URL of your StreamNative cluster.
   * `oauth.audience`: the `audience` parameter is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
   * `key.converter` and `value.converter`: the converter that sends JSON-format messages to Kafka.
   * `plugin.path`: the full path of the `connectors` folder that is created in the previous step.

   For details about the security of Kafka Connect, see the [Confluent documentation](https://docs.confluent.io/platform/current/kafka/authentication_sasl/authentication_sasl_scram.html#kconnect-long).

### Configure the Elasticsearch Sink connector

Add the following content to the `elasticsearch-sink-connector.properties` file.

```conf theme={null}
name=elasticsearch-sink
connector.class=io.confluent.connect.elasticsearch.ElasticsearchSinkConnector
tasks.max=1
# Topic name to get data from
topics=test-elasticsearch-sink
key.ignore=true
# The key converter for this connector
key.converter=org.apache.kafka.connect.storage.StringConverter
# The value converter for this connector
value.converter=org.apache.kafka.connect.json.JsonConverter
# Identify if the value contains a schema.
# Required value converter is `org.apache.kafka.connect.json.JsonConverter`.
value.converter.schemas.enable=false

schema.ignore=true
# Elasticsearch server url
connection.url=http://localhost:9200
type.name=kafka-connect
```

### Run Kafka Connect

Open a new terminal and navigate to the Kafka folder, run the following command in the directory:

```bash theme={null}
cd kafka_2.13-3.1.0
bin/connect-standalone.sh config/connect-sn-kop.properties config/elasticsearch-sink-connector.properties
```

You should see the following output:

```bash theme={null}
...output omitted...
[2023-02-07 23:11:37,102] INFO [elasticsearch-sink|task-0] [Consumer clientId=connector-consumer-elasticsearch-sink-0, groupId=connect-elasticsearch-sink] Discovered group coordinator kopyhshen-broker-0-3d0a2d7c-2875-4caf-b74e-7d3260027a9a.gcp-shared-gcp-usce1-martin.streamnative.g.snio.cloud:9093 (id: 1865975191 rack: null) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:853)
[2023-02-07 23:11:37,106] INFO [elasticsearch-sink|task-0] [Consumer clientId=connector-consumer-elasticsearch-sink-0, groupId=connect-elasticsearch-sink] (Re-)joining group (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:535)
[2023-02-07 23:11:42,233] INFO [elasticsearch-sink|task-0] [Consumer clientId=connector-consumer-elasticsearch-sink-0, groupId=connect-elasticsearch-sink] Successfully joined group with generation Generation{generationId=3, memberId='connector-consumer-elasticsearch-sink-0-1718e62d-b638-419b-8376-2ee14a19d23d', protocol='range'} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:595)
[2023-02-07 23:11:42,237] INFO [elasticsearch-sink|task-0] [Consumer clientId=connector-consumer-elasticsearch-sink-0, groupId=connect-elasticsearch-sink] Finished assignment for group at generation 3: {connector-consumer-elasticsearch-sink-0-1718e62d-b638-419b-8376-2ee14a19d23d=Assignment(partitions=[test-elasticsearch-sink-0])} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:652)
```

## Index data on the Elasticsearch server

This example shows how to send five JSON-format messages to the `test-elasticsearch-sink` topic. These messages will be forwarded to the local Elasticsearch server.

1. Start a Kafka producer and send five JSON-format messages to the `test-elasticsearch-sink` topic.

   ```bash theme={null}
   echo '{"reporterId": 8824, "reportId": 10000, "content": "Was argued independent 2002 film, The Slaughter Rule.", "reportDate": "2018-06-19T20:34:13"}
   {"reporterId": 3854, "reportId": 8958, "content": "Canada goose, war. Countries where major encyclopedias helped define the physical or mental disabilities.", "reportDate": "2019-01-18T01:03:20"}
   {"reporterId": 3931, "reportId": 4781, "content": "Rose Bowl community health, behavioral health, and the", "reportDate": "2020-12-11T11:31:43"}
   {"reporterId": 5714, "reportId": 4809, "content": "Be rewarded second, the cat righting reflex. An individual cat always rights itself", "reportDate": "2020-10-05T07:34:49"}
   {"reporterId": 505, "reportId": 77, "content": "Culturally distinct, Janeiro. In spite of the crust is subducted", "reportDate": "2018-01-19T01:53:09"}' | ./bin/kafka-console-producer.sh \
       --bootstrap-server "your-pulsar-service-url" \
       --producer.config ./kafka.properties \
       --topic test-elasticsearch-sink
   ```

   If everything goes well, the command will exit normally and output nothing. The records are sent to an Elasticsearch index called `test-elasticsearch-sink`. The name of the index is the same as the name of the Kafka topic. By default, the Elasticsearch Sink connector creates the index with the same name.

   For details about how to configure the Kafka configuration properties file, see [get started with Kafka protocol](/cloud/get-started/quickstart-kafka).

2. Verify that data is indexed on the Elasticsearch server.

   ```bash theme={null}
   curl 'http://localhost:9200/test-elasticsearch-sink/_search' | jq
   ```

   Ten records should be returned.

   ```json theme={null}
   {
     "took": 10,
     "timed_out": false,
     "_shards": {
       "total": 1,
       "successful": 1,
       "skipped": 0,
       "failed": 0
     },
     "hits": {
       "total": {
         "value": 5,
         "relation": "eq"
       },
   ...output omitted...
         {
           "_index": "test-elasticsearch-sink",
           "_type": "_doc",
           "_id": "test-elasticsearch-sink+0+9",
           "_score": 1,
           "_source": {
             "reportId": 4781,
             "reportDate": "2020-12-11T11:31:43",
             "reporterId": 3931,
             "content": "Rose Bowl community health, behavioral health, and the"
           }
         }
       ]
     }
   }
   ```

3. Check the record count in the `test-elasticsearch-sink` index.

   ```bash theme={null}
   curl 'http://localhost:9200/test-elasticsearch-sink/_count'
   ```

   You should see the following output:

   ```
   {"count":5,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}
   ```

   You can see that the number of received records is identical to the number of sent messages.

4. Verify the connectivity between Kafka and Pulsar by searching a keyword in the `test-elasticsearch-sink` index.

   This example shows how to search the keyword `health` in the `test-elasticsearch-sink` index.

   ```bash theme={null}
   curl 'http://localhost:9200/test-elasticsearch-sink/_search?q=content:health' | jq
   ```

   You should see the following output:

   ```json theme={null}
   {
     "took": 9,
     "timed_out": false,
     "_shards": {
       "total": 1,
       "successful": 1,
       "skipped": 0,
       "failed": 0
     },
     "hits": {
       "total": {
         "value": 1,
         "relation": "eq"
       },
       "max_score": 3.2154756,
       "hits": [
         {
           "_index": "test-elasticsearch-sink",
           "_type": "_doc",
           "_id": "test-elasticsearch-sink+0+11",
           "_score": 3.2154756,
           "_source": {
             "reportId": 4781,
             "reportDate": "2020-12-11T11:31:43",
             "reporterId": 3931,
             "content": "Rose Bowl community health, behavioral health, and the"
           }
         }
       ]
     }
   }
   ```

   You can see that all records have the word `health` listed in their `content` field.


# StreamNative Cloud Console Basics
Source: https://docs.streamnative.io/cloud/get-started/cloud-console



The StreamNative Cloud Console enables you to view and manage [instances](/cloud/clusters/manage-instances/instance), [clusters](/cloud/clusters/manage-clusters/cluster), [tenants](/cloud/manage-data-streams/tenant), [namespaces](/cloud/manage-data-streams/namespace), [topics](/cloud/manage-data-streams/topic), view your account billing information, and more. This page describes some common tasks you can complete in the Cloud Console.

## Use the Cloud Console

Access the Cloud Console at the following URL:

[https://console.streamnative.cloud](https://console.streamnative.cloud)

To access the console, you will be required to sign in to your StreamNative Cloud account. If you don't have an account, you can [sign up](/cloud/security/access/resource-hierarchy/organizations#create-an-organization) for one with \$200 in credits.

## Console Layout

There are 5 elements in the Cloud Console:

* **Breadcrumb**: At the top of the console, there is a Breadcrumb-based Navigation Bar. You can use the Breadcrumb to navigate the structure of StreamNative Cloud.
* **Navigation Pane**: On the left side of the console, there is a Navigation Pane that contains the resources available to you at the current level of navigation. The resources at each level are different based on the level of navigation.
* **Main Workspace**: To the right of the Navigation Pane is the Main Workspace. This area displays the content and functionality related to the resource you've selected in the Navigation Pane, allowing you to view, manage, and interact with your chosen resources.
* **User Menu**: In the top-right corner is the User Menu. This menu contains options for managing your account settings, viewing organization details and resources, accessing documentation, and logging out. It also displays your current user information.
* **Chat Dialog**: In the bottom-right corner of the Main Workspace, you can find a Chat Dialog. This dialog allows you to interact directly with the StreamNative support team.

## Breadcrumb

<span />

Use the Breadcrumb in the StreamNative Console to navigate the structure of StreamNative Cloud. You can switch between different instances, clusters, tenants, and namespaces by selecting your desired location from the dropdown menu.

<img alt="breadcrumb" />

### Breadcrumb Structure

The breadcrumb contains four values that correspond to resources within a StreamNative organization:

* [Instance](/cloud/clusters/manage-instances/instance)
* [Cluster](/cloud/clusters/manage-clusters/cluster)
* [Tenant](/cloud/manage-data-streams/tenant)
* [Namespace](/cloud/manage-data-streams/namespace)

Selecting an item in the breadcrumb will take you to the corresponding workspace to work on different levels of resources. The left Navigation Pane will show different resources based on the level of navigation.

## Organization Workspace

<span />

The Organization Workspace is the landing page when you log into the StreamNative Console. It offers a comprehensive overview of your instances and users, and provides a step-by-step guide to set up and manage your first StreamNative cluster.

To return to the Organization Workspace from any level of navigation, simply click the **Organization** button located to the right of the StreamNative logo.

In the Organization Workspace, the left Navigation Pane displays the following resources:

* **Organization Dashboard**: This central hub presents an overview of your instances and users. It also guides you through the entire process of setting up and managing your first Pulsar cluster.
* **Instances**: This section allows you to view and manage all instances associated with your organization.

### Organization Dashboard

The **Organization Dashboard** page provides an overview of the instances and users, and guides you through the complete process of setting up and managing your Pulsar cluster.

For users who don't have a cluster yet, the **Organization Dashboard** page serves as a prompt for moving forward in the provisioning process. For users who already have a cluster, the **Organization Dashboard** page encourages them to broaden their usage of StreamNative Cloud.

After logging in to the StreamNative Console, you can go to the **Organization Dashboard** page by clicking it on the left navigation pane. The **Organization Dashboard** page consists of three parts:

* The **Instance** card

  You can create, view, and manage your instances on this card. For details, see [work with instances](/cloud/clusters/manage-instances/instance).

* The **Users** card

  You can invite, view, and manage users on this card. For details, see [work with users](/cloud/security/authentication/user-accounts).

* The **Setup guide** area

  This section walks you through the basic setup and configuration process, such as adding payment methods, creating service accounts, creating your Pulsar instances and clusters, viewing your tenants and namespaces, creating your topics, setting up your client, and so on. When an operation is completed, the related checkbox is checked.

  <img alt="gif of dashboard" />

### Navigating to the Cluster Workspace

You can access the Cluster Workspace using either of these methods:

1. Utilize the Breadcrumb navigation:

   * First, select your desired instance.
   * Then, choose the specific cluster you want to access.

2. Direct navigation from the Instances page:
   * Navigate to the **Instances** page.
   * Click on the row of the instance that contains your desired cluster.

## Cluster Workspace

<span />

The Cluster Workspace is where you view and manage all the cluster-level resources, including Tenants and Secrets, and manage the cluster configuration. It also provides guides on how you can connect to the clusters using various clients, including Kafka, Pulsar, and MQTT. Additionally, it provides instructions on how to use our cloud metrics API to monitor the cluster.

### Switch a Cluster

You can use the Breadcrumb to switch to a different cluster within the same instance. If you want to switch to a different instance, you can go back to the **Organization Workspace** first and select a different instance from the dropdown menu.

### Left Navigation Pane

The left Navigation Pane displays the following resources:

* **Cluster Dashboard**: This page provides a dashboard of the cluster, including the number of topics, number of messages, number of subscriptions, storage & backlog size and throughput, and more. Additionally, it includes a details tab providing the cluster details and its access points.
* **Resources**:
  * **Tenants**: This page allows you to view and manage all the tenants available in this cluster.
  * **Secrets**: This page allows you to view and manage all secrets associated with your cluster.
* **Monitor**:
  * **Metrics API**: This page provides instructions on how to use the metrics API to monitor the cluster.
* **Clients**:
  * **Pulsar Clients**: This page provides instructions on how to connect to the cluster using Pulsar clients.
  * **Kafka Clients**: This page provides instructions on how to connect to the cluster using Kafka clients.
  * **MQTT Clients**: This page provides instructions on how to connect to the cluster using MQTT clients.
* **Admin**:
  * **Configuration**: This page allows you to view and manage the cluster configuration.

### Navigating to the Tenant Workspace

You can access the Tenant Workspace using either of these methods:

1. Utilize the Breadcrumb navigation:

   * First, select your desired instance.
   * Then, choose the specific cluster that contains your desired tenant.
   * Finally, click on the name of the tenant you want to access.

2. Direct navigation from the Tenants page:
   * Navigate to the **Tenants** page.
   * Click on the row of the tenant you want to access.

## Tenant Workspace

<span />

The Tenant Workspace is where you view and manage all the tenant-level resources, including namespaces, topics, and more. It also provides shortcuts to connect to the clusters using various clients, including Kafka, Pulsar, and MQTT.

### Switch a Tenant

You can use the **Tenant selector** in the Breadcrumb to switch to a different tenant within the same cluster. If you want to switch to a different cluster, you can use the **Cluster selector** in the Breadcrumb to navigate to a different Cluster's workspace.

### Left Navigation Pane

The left Navigation Pane of a Tenant Workspace displays the following resources:

* **Tenant Dashboard**: This page provides a dashboard of the tenant, including the number of topics, number of messages, number of subscriptions, storage & backlog size, throughput, and more.
* **Resources**:
  * **Namespaces**: This page allows you to view and manage all the namespaces in this tenant.
* **Clients**:
  * **Pulsar Clients**: This page provides instructions on how to connect to the cluster using Pulsar clients.
  * **Kafka Clients**: This page provides instructions on how to connect to the cluster using Kafka clients.
  * **MQTT Clients**: This page provides instructions on how to connect to the cluster using MQTT clients.
* **Admin**:
  * **Configuration**: This page allows you to view and manage the tenant configuration.

### Navigating to the Namespace Workspace

You can access the Namespace Workspace using either of these methods:

1. Utilize the Breadcrumb navigation:

   * First, select your desired instance.
   * Then, choose the specific cluster that contains your desired tenant.
   * Next, click on the name of the tenant you want to access.
   * Finally, click on the name of the namespace you want to access.

2. Direct navigation from the Namespaces page:
   * Navigate to the **Namespaces** page.
   * Click on the row of the namespace you want to access.

## Namespace Workspace

<span />

The Namespace Workspace is where you view and manage all the namespace-level resources, including topics, functions, connectors, and more. It also provides shortcuts to connect to the clusters using various clients, including Kafka, Pulsar, and MQTT.

### Switch a Namespace

You can use the **Namespace selector** in the Breadcrumb to switch to a different namespace within the same tenant. If you want to switch to a different tenant, you can use the **Tenant selector** in the Breadcrumb to navigate to a different Tenant's workspace.

### Left Navigation Pane

The left Navigation Pane of a Namespace Workspace displays the following resources:

* **Namespace Dashboard**: This page provides a dashboard of the namespace, including the number of topics, number of messages, number of subscriptions, storage & backlog size, throughput, and more.
* **Resources**:
  * **Topics**: This page allows you to view and manage all the topics in this namespace.
  * **Connectors**: This page allows you to view and manage all the connectors in this namespace.
  * **Functions**: This page allows you to view and manage all the functions in this namespace.
  * **pfSQL**: This page allows you to view and manage all the pfSQL queries in this namespace.
* **Clients**:
  * **Pulsar Clients**: This page provides instructions on how to connect to the cluster using Pulsar clients.
  * **Kafka Clients**: This page provides instructions on how to connect to the cluster using Kafka clients.
  * **MQTT Clients**: This page provides instructions on how to connect to the cluster using MQTT clients.
* **Admin**:
  * **Bundles**: This page allows you to view and manage all the bundles in this namespace.
  * **Configuration**: This page allows you to view and manage the namespace configuration.

## User Menu

<span />

The User Menu is located in the top-right corner of the StreamNative Console. It contains the following options:

* **Organization**: This option allows you to view and manage your organizations. You can use it to switch between organizations.
* **Organization Usage**: This page shows the billing usage of your organization, broken down by billing cycles across different billing dimensions.
* **Accounts & Accesses**: This page displays the accounts and accesses of your organization. You can access it for any account-related operations.
* **Cloud Environments**: This page is for managing the BYOC infrastructure. It is the central place for you to view and manage the [Cloud Connections](/cloud/clusters/byoc/create-cloud-connection) and [Cloud Environments](/cloud/clusters/byoc/create-cloud-environment) of your organization.
* **Billing & Payment**: This page shows the billing information of your organization, including your payment methods, invoices, and more.
* **Support**: This option allows you to submit a ticket to the StreamNative support portal.
* **Documentation**: This option provides access to the StreamNative documentation site.
* **Sign out**: This option allows you to log out of your StreamNative Cloud account.


# Quick Start for BYOC Cost Optimized Cluster
Source: https://docs.streamnative.io/cloud/get-started/quickstart-byoc-costprofile



StreamNative Cloud is a resilient, scalable, data streaming service , delivered as a fully managed Pulsar and Kafka service.

StreamNative Cloud provides multiple interfaces for management and interaction:

1. **StreamNative Cloud Console**: A user-friendly web-based interface for managing cluster resources, configuring settings, and handling billing.

2. **Command-Line Interface (CLI)**:

   * [StreamNative CLI (`snctl`)](/tools/cli/snctl/snctl-overview): The unified command-line interface for deploying and managing StreamNative Cloud infrastructure and interacting directly with your Pulsar clusters and Kafka-protocol endpoints.
   * [Pulsar CLI (`pulsarctl`)](/tools/cli/pulsarctl/pulsarctl-overview): For managing cluster-specific resources, such as tenants, namespaces, topics, functions, connectors, and more.

3. **REST APIs**: For programmatic access and integration with other systems.

   * [StreamNative Cloud API](/api-references/cloudapi/cloud-api)
   * [Pulsar Admin REST API](https://pulsar.apache.org/docs/en/admin-api-overview/)
   * [Kafka Admin API](https://kafka.apache.org/documentation/#adminapi)

4. **Terraform Providers**:
   * [StreamNative Terraform Provider](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs)
   * [Pulsar Terraform Provider](https://registry.terraform.io/providers/streamnative/pulsar/latest/docs)

These tools provide flexibility in how you interact with and manage your StreamNative Cloud environment, catering to different user preferences and use cases.

<Note title="Get started for Free">
  [Sign up for StreamNative Cloud](https://console.streamnative.cloud/?defaultMethod=signup) and get \$200 of free credits. No credit card required.
</Note>

This quick start guides you through getting started with StreamNative [BYOC](/cloud/clusters/cluster-types#byoc-clusters). It demonstrates how to use a StreamNative BYOC cluster to create topics, produce data to the cluster, and consume data from it.

<Note title="Note">
  This QuickStart assumes you are familiar with the [basics concepts of StreamNative Cloud Clusters](/cloud/clusters/streamnative-cluster-overview).
</Note>

## Prerequisites

* Access to [StreamNative Cloud](https://console.streamnative.cloud/?defaultMethod=signup).
* Internet connectivity.
* Access to Your AWS Account for provisioning the BYOC infrastructure.
* Install [Java 17](https://www.oracle.com/java/technologies/downloads/#java17). For details, see [overview of JDK installation](https://docs.oracle.com/en/java/javase/17/install/overview-jdk-installation.html).
* Install Maven. For details, see [installing Apache Maven](https://maven.apache.org/install.html).

## Step 1: Sign up

<Note title="Note">
  If you have an email account configured for using Single Sign-On (SSO) with StreamNative Cloud, use that email address and password when signing up.
</Note>

To sign up, navigate to the [StreamNative Cloud Console signup page](https://console.streamnative.cloud/?defaultMethod=signup). Follow the prompts to create an account.
After you click **Finish**, you might have to wait briefly for your first organization to be created. After your new organization is created, continue on to creating your first instance and cluster.

## Step 2: Grant StreamNative Vendor Access

Before deploying a StreamNative cluster within your cloud account, you must first grant StreamNative vendor access. Follow the instructions in [Account Access for BYOC on AWS](/cloud/clusters/byoc/grant-vendor-access/byoc-on-aws/byoc-aws-access#provision-aws-access) to provision AWS access for StreamNative Cloud. This step ensures StreamNative has the necessary permissions to manage resources in your AWS account.

Once completed, please note the account ID of the AWS account you have granted access to StreamNative Cloud. You will use this account ID to create a Cloud Connection.

## Step 3: Create a Cloud Connection

<Tabs>
  <Tab title="Cloud Console">
    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Cloud Environments**.

    2. Select **Cloud Connections** tab and click **New Cloud Connection**.

    3. In the **Create connection** dialog, enter the following information:

       * **Name**: Enter a name for the cloud connection. For example, `my-aws-connection`.
       * **Connection provider**: Select **AWS** as the connection provider.
       * **AWS Account ID**: Enter the account ID of the AWS account you noted in the previous step.
       * Check **Confirm if vendor access Terraform module is executed**.

    4. Click **Submit**.

    5. The cloud connection creation process will start. Once completed, you can see the status of the cloud connection turned to `Connected` in the **Cloud Connections** tab.
  </Tab>
</Tabs>

## Step 4: Create a Cloud Environment

Once you have created a cloud connection, you can create a cloud environment and provision a BYOC instance.

<Tabs>
  <Tab title="Cloud Console">
    1. Navigate to [Organization Dashboard](/cloud/get-started/cloud-console#organization-dashboard).

    2. Select **Instances** at the left navigation pane.

    3. On the **Instances** page, click **+ New Instance**.

    4. On the **Choose the deployment type for your instance** page, click **Deploy BYOC**.

    5. You will see a dialog "Cloud Environment required". Click **Create** button to create a cloud environment.

    6. On the **Cloud Connection** page, select the cloud connection you created in the previous step. In this case, it is `my-aws-connection`. Then click **Environment setup**.

    7. Then fill out the information for the cloud environment.

       * **Region**: Select the region where you want to deploy the BYOC cluster. In this example, it is `us-west-2`.
       * **Environment tag**: Enter a tag for the cloud environment. For example, `poc`.
       * **Configure StreamNative Managed VPC Network**:
         * **Network CIDR**: By default, it creates a StreamNative Managed VPC with a CIDR of `10.0.0.0/16`. If you need to specify a different CIDR, you can enter it here.
       * **Default Gateway**: You can configure how do you want to expose your BYOC cluster, whether it is **public** or **private**. By default, it is **public**.

    8. Click **Create**.
  </Tab>
</Tabs>

The provisioning process of a cloud environment usually takes about 40 minutes to complete. You can safely close the page and come back later. You will also receive an email notification when the cloud environment is ready.

For more information about provisioning BYOC infrastructure, see [Provision BYOC Infrastructure](/cloud/clusters/byoc/byoc-overview).

## Step 5: Create a StreamNative Instance & Cluster

Once the cloud environment is ready, you can create a Pulsar instance.

<Tabs>
  <Tab title="Cloud Console">
    1. Navigate to [Organization Dashboard](/cloud/get-started/cloud-console#organization-dashboard).

    2. Select **Instances** at the left navigation pane.

    3. On the **Instances** page, click **+ New Instance**.

    4. On the **Choose the deployment type for your instance** page, click **Deploy BYOC** again. You will not see the dialog "Cloud Environment required" this time.

    5. On the **Instance Configuration** page, fill out the information for the Pulsar instance. Then, click **Cluster Location** to start the cluster creation process.

       * **Instance Name**: Enter a name for the instance. For example, `my-instance`.
       * **Cloud Connection**: Select the cloud connection you created in the previous step. In this example, it is `my-aws-connection`.

    6. On the **Cluster Location** page, fill out the information for the cluster.

       * **Cluster Name**: Enter a name for the cluster. For example, `my-cluster`.
       * **Cloud Environment**: Select the cloud environment you created in the previous step. In this example, it is `aws-usw2-poc-<random-suffix>`.
       * **Cluster Profile**: Select the cluster profile. In this example, it is 'Cost Optimized'
       * **Availability Zone**: Select the Availability Zone. In this example, it is 'Multi AZ'

    7. Click **Cluster Size** to configure the cluster size.

       * You can use the slider to adjust the throughput accordingly.
       * You can also manually configure the number of brokers and its corresponding resources in the **Advanced** section.
       * There is no bookie in the cluster based on Cost Optimized profile.

       At the right navigation pane, you can see the estimated cost for the cluster.

    8. Click **Finish** to start the cluster creation process.

    The cluster page appears, displaying the cluster creation process. Depending on the chosen cloud provider and other settings, it may take several minutes to provision the cluster.
    Once the cluster has been provisioned, the page will show "Cluster Provisioned successfully" and you can click **Go To The Dashboard** to access the **Cluster Dashboard** page.
  </Tab>
</Tabs>

Now you can get started configuring apps and data on your new cluster.

## Step 6: Create a service account

To interact with your cluster by producing and consuming messages, you need to set up authentication and authorization. This is done by creating a [Service Account](/cloud/security/authentication/service-accounts/service-accounts), which serves as an identity for authenticating and authorizing access to the cluster. The service account will provide the necessary credentials for your applications to securely connect and perform operations on the Pulsar cluster.

<Tabs>
  <Tab title="Cloud Console">
    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Accounts & Accesses**.

    2. On the left navigation pane, click **Service Accounts**.

    3. On the **Service Account** page, click **+ New**.

    4. On the **Create Service Account** dialog, enter a name for the service account, and then click **Confirm**.

    5. On the **Service Account** page, in the row of the service account you just created, click the **...** icon, and select **Create API Key** in the dropdown menu.

    6. On the **New API Key** dialog:

       * Enter a name for the API key
       * Set the expiration date
       * Select the instance you created in previous step
       * Write a description for the API key
       * Click **Confirm**

    <Note title="Note">
      An API key and associated secret apply to the active StreamNative instance. If you add a new instance, you must create a new API key for producers and consumers on the new Pulsar instance. For more information, see [Use API Keys to Authenticate to StreamNative Cloud](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview).
    </Note>

    7. After the API key is created, you can see the API key shown in the **New API Key** dialog. Click the **Copy and close** button to copy the API key to your clipboard. **Please note that you cannot retrieve the API key later after closing the dialog.**
  </Tab>
</Tabs>

## Step 7: Create Tenant and Namespace, and Authorize the Service Account

After creating the service account and obtaining the API key, the next crucial step is to authorize the service account. This process grants the necessary permissions for the service account to interact with your StreamNative Cloud cluster.

Authorization involves setting up Access Control Lists (ACLs) that define what actions the service account can perform. Typically, you'll want to grant permissions for producing and consuming messages on specific topics or namespaces.

For more information about authorization, see [Authorization and ACLs](/cloud/security/access/access-control-lists/authorization-and-acls).

<Tabs>
  <Tab title="Cloud Console">
    1. Go back to the **Cluster Dashboard** page.

    2. On the left navigation pane, click **Instances**.

    3. On the **Instances** page, click the name of the instance you created in Step 2.

    4. On the **Cluster Dashboard** page, click **Tenants** on the left navigation pane.

    5. On the **Tenants** page, click **+ New Tenant**.

    6. On the **New Tenant** dialog:

       * Enter a name for the tenant
       * Select your user account as the **Admin role**
       * Select the cluster created in Step 2 as **Allowed clusters**
       * Click **Confirm**

    7. On the **Tenants** page, click the name of the tenant you just created. You will be directed to the **Tenant Dashboard** page.

    8. On the **Tenant Dashboard** page, click **Namespaces** on the left navigation pane.

    9. On the **Namespaces** page, click **New Namespace**.

    10. On the **New Namespace** dialog:

        * Enter a name for the namespace
        * Select the cluster created in Step 2 as **Allowed clusters** and **Replication clusters**
        * Click **Confirm**

    11. On the **Namespaces** page, click the name of the namespace you just created. You will be directed to the **Namespace Dashboard** page.

    12. On the **Namespace Dashboard** page, click **Configuration** on the left navigation pane.

    13. On the **Namespace configuration** page, click **ADD ROLE**. Select the name of the service account you just created, and choose the **consume** and **produce** permissions. This grants your service account the `produce` and `consume` permissions for this namespace.
  </Tab>
</Tabs>

## Step 8: Grant permission to access Kafka Schema Registry

This quick start uses AVRO to produce the message. You need to grant the service account to access the Kafka Schema registry.

You can do this by granting the service account the `produce` permission to the `public/__kafka_schemaregistry/__schema-registry` topic. This is required because the current implementation of Kafka Schema Registry uses this topic's ACL to authorize access to the schema registry.

<Tabs>
  <Tab title="Cloud Console">
    1. Navigate to the **Namespace Dashboard** page of `public / __kafka_schemaregistry` namepsace.

    2. On the left navigation pane, click **Topics**.

    3. On the **Topics** page, click the topic `__schema-registry`.

    4. On the topic details page, click **Policy** tab.

    5. Click **+ Add** button and, in the dropdown menu, select the name of the service account you just created and choose the **produce** permission.
  </Tab>
</Tabs>

Now you have created a tenant, namespace, and granted the service account the `produce` and `consume` permissions for this namespace. You also grant the service account permission to access the Kafka Schema Registry. You can now continue on to building a Python app, connecting to the cluster, and producing and consuming messages.

## Step 9: Produce and consume messages

<Note title="Note">
  This QuickStart provides you with an example Java app to get you up and running with consuming and producing messages. This is a simple example and is not intended for production environments.
</Note>

### Create a producer/consumer

<Tabs>
  <Tab title="Cloud Console">
    1. Return to the StreamNative Cloud Console and go to the "Cluster Dashboard" page.

    2. On the left navigation pane, click **Kafka Clients**.

    3. On the **Kafka client setup** page, click the **Code libraries** tab, follow the setup wizard to get the sample codes for your producer and consumer.

       a. Select **Java** as the client library and click **Next**.

       b. Select the service account you created and click **Next**.

       c. Select **API Key** as the authentication type and click **Next**. If you already have an API key, you can use the API key noted in Step 6. Otherwise, you can create a new API key.

       d. Check **Kafka Schema Registry** and click **Next**.

       d. Copy the required dependencies to your `pom.xml` file, and click **Next**.

       e. Select the target tenant, namespace, topic, and subscription.

       f. You are now ready to copy the auto-generated sample codes.

    4. Build your project.

       a. Create a new file named `pom.xml` and add the following content:

       ```xml theme={null}
       <?xml version="1.0" encoding="UTF-8"?>

       <project xmlns="http://maven.apache.org/POM/4.0.0"
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
       <modelVersion>4.0.0</modelVersion>

           <groupId>org.example</groupId>
           <artifactId>kafka-examples</artifactId>
           <version>1.0-SNAPSHOT</version>
           <packaging>jar</packaging>
           <properties>
               <log4j.version>2.17.1</log4j.version>
               <!-- Maven properties for compilation -->
               <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
               <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
           </properties>
           <dependencies>
               <!-- Add the Kafka dependency -->
               <dependency>
                   <groupId>org.apache.kafka</groupId>
                   <artifactId>kafka-clients</artifactId>
                   <version>3.4.0</version>
               </dependency>
               <dependency>
                   <groupId>io.confluent</groupId>
                   <artifactId>kafka-avro-serializer</artifactId>
                   <version>7.5.0</version>
               </dependency>
               <dependency>
                   <groupId>io.streamnative.pulsar.handlers</groupId>
                   <artifactId>oauth-client</artifactId>
                   <version>3.1.0.4</version>
               </dependency>
               <dependency>
                   <groupId>org.apache.kafka</groupId>
                   <artifactId>kafka-streams</artifactId>
                   <version>3.4.0</version>
               </dependency>
               <!-- Add the Log4j dependency -->
               <dependency>
                   <groupId>org.apache.logging.log4j</groupId>
                   <artifactId>log4j-slf4j-impl</artifactId>
                   <version>${log4j.version}</version>
               </dependency>
               <dependency>
                   <groupId>org.apache.logging.log4j</groupId>
                   <artifactId>log4j-core</artifactId>
                   <version>${log4j.version}</version>
               </dependency>
               <dependency>
                   <groupId>org.slf4j</groupId>
                   <artifactId>slf4j-log4j12</artifactId>
                   <version>1.7.30</version>
               </dependency>
           </dependencies>
           <repositories>
               <repository>
               <id>confluent</id>
               <url>https://packages.confluent.io/maven/</url>
               </repository>
           </repositories>

       </project>
       ```

       b. Create a folder `src/main/java/org/example`.

       c. Under `src/main/java/org/example`, create a file named `SNCloudTokenProducer.java`. Copy and paste the producer code to this file.

       d. Under `src/main/java/org/example`, create a file named `SNCloudTokenConsumer.java`. Copy and paste the consumer code to this file.

       e. In both files, replace **`<JWT Token>`** with the API key you copied from the Service Account page.

       c. Go to the root folder of your project and run the following command to build your project:

       ```bash theme={null}
       mvn clean install
       ```
  </Tab>
</Tabs>

### Run the clients to produce and consume your first message

1. Open a terminal window, navigate to the root folder of your project, and run the following command:

   ```bash theme={null}
   mvn exec:java -Dexec.mainClass="org.example.SNCloudTokenConsumer"
   ```

2. Open a second terminal window, navigate to the root folder of your project, and run the following command:

   ```bash theme={null}
   mvn exec:java -Dexec.mainClass="org.example.SNCloudTokenProducer"
   ```

   You will see a message like the following:

   ```bash theme={null}
   Send hello to <YOUR-TOPIC-NAME>-0@0
   ```

3. Return to the first terminal window. You should see the following:

   ```bash theme={null}
   Receive record: {"name": "jwt-sr", "age": 20} from <YOUR-TOPIC-NAME>-0@0
   ```

You can continue to produce and consume messages by repeating the above steps.

For example, you can run the producer in a loop to produce 100 messages:

```bash theme={null}
for i in {1..100}; do
  mvn exec:java -Dexec.mainClass="org.example.SNCloudTokenProducer"
done
```

## Step 10: Check the storage bucket

Since the Cost Optimized Profile currently uses S3 as the storage layer, you can check the storage bucket to verify the data is persisted.

1. Navigate to the **Cluster Dashboard** page.

2. Click **Details** tab.

3. On the **Details** page, you can find the S3 bucket name listed in the **Storage Bucket** field under the **Access Points** section.

4. Navigate to your AWS account and check the S3 bucket to verify the data is persisted.

5. You should be able to see the data persisted in the S3 bucket under folder `<org-id>-<cluster-name>-<randomized-string>-ursa`. There are two sub folders: `storage` and `compaction`. The `storage` folder contains the raw WAL files and the `compaction` folder contains the compacted lakehouse tables. Those lakehouse tables are organized by `<tenant>/<namespace>/<topic>`.

## Step 11: Query the compacted lakehouse tables using DuckDB

1. Install DuckDB. For details, see [DuckDB Installation](https://duckdb.org/docs/installation).

2. Run DuckDB.

   ```bash theme={null}
   duckdb
   ```

3. Load the lakehouse table into DuckDB.

   ```sql theme={null}
   CREATE SECRET (
      TYPE S3,
      PROVIDER CREDENTIAL_CHAIN
   );
   ```

   ```sql theme={null}
   SELECT COUNT(*) FROM delta_scan('s3://path/to/compacted/lakehouse/table')
   ```

   You should be able to see the output like the following:

   ```sql theme={null}
   ┌──────────────┐
   │ count_star() │
   │    int64     │
   ├──────────────┤
   │          101 │
   └──────────────┘
   ```

## Next steps

* After you have successfully provisioned a **BYOC** cluster and connected to the cluster, you can learn more about working with StreamNative Cloud by reading through [Cloud Console basics](/cloud/get-started/cloud-console).

* If you want to learn more about Kafka and StreamNative Cloud, take our developer courses at the [StreamNative Developer Portal](https://streamnative.io/dev-portal).


# Quick Start for BYOC Latency Optimized Cluster
Source: https://docs.streamnative.io/cloud/get-started/quickstart-byoc-latencyprofile



StreamNative Cloud is a resilient, scalable, data streaming service , delivered as a fully managed Pulsar and Kafka service.

StreamNative Cloud provides multiple interfaces for management and interaction:

1. **StreamNative Cloud Console**: A user-friendly web-based interface for managing cluster resources, configuring settings, and handling billing.

2. **Command-Line Interface (CLI)**:

   * [StreamNative CLI (`snctl`)](/tools/cli/snctl/snctl-overview): The unified command-line interface for deploying and managing StreamNative Cloud infrastructure and interacting directly with your Pulsar clusters and Kafka-protocol endpoints.
   * [Pulsar CLI (`pulsarctl`)](/tools/cli/pulsarctl/pulsarctl-overview): For managing cluster-specific resources, such as tenants, namespaces, topics, functions, connectors, and more.

3. **REST APIs**: For programmatic access and integration with other systems.

   * [StreamNative Cloud API](/api-references/cloudapi/cloud-api)
   * [Pulsar Admin REST API](https://pulsar.apache.org/docs/en/admin-api-overview/)
   * [Kafka Admin API](https://kafka.apache.org/documentation/#adminapi)

4. **Terraform Providers**:
   * [StreamNative Terraform Provider](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs)
   * [Pulsar Terraform Provider](https://registry.terraform.io/providers/streamnative/pulsar/latest/docs)

These tools provide flexibility in how you interact with and manage your StreamNative Cloud environment, catering to different user preferences and use cases.

<Note title="Get started for Free">
  [Sign up for StreamNative Cloud](https://console.streamnative.cloud/?defaultMethod=signup) and get \$200 of free credits. No credit card required.
</Note>

This quick start guides you through getting started with StreamNative [BYOC](/cloud/clusters/cluster-types#byoc-clusters). It demonstrates how to use a StreamNative BYOC cluster to create topics, produce data to the cluster, and consume data from it.

<Note title="Note">
  This QuickStart assumes you are familiar with the [basics concepts of StreamNative Cloud Clusters](/cloud/clusters/streamnative-cluster-overview).
</Note>

## Prerequisites

* Access to [StreamNative Cloud](https://console.streamnative.cloud/?defaultMethod=signup).
* Internet connectivity.
* Access to Your AWS Account for provisioning the BYOC infrastructure.
* Ensure you have installed Python 3.0 or higher versions and the Pulsar Python client.

  ```bash theme={null}
  python -m pip install pulsar-client
  ```

## Step 1: Sign up

<Note title="Note">
  If you have an email account configured for using Single Sign-On (SSO) with StreamNative Cloud, use that email address and password when signing up.
</Note>

To sign up, navigate to the [StreamNative Cloud Console signup page](https://console.streamnative.cloud/?defaultMethod=signup). Follow the prompts to create an account.
After you click **Finish**, you might have to wait briefly for your first organization to be created. After your new organization is created, continue on to creating your first instance and cluster.

## Step 2: Grant StreamNative Vendor Access

Before deploying a StreamNative cluster within your cloud account, you must first grant StreamNative vendor access. Follow the instructions in [Account Access for BYOC on AWS](/cloud/clusters/byoc/grant-vendor-access/byoc-on-aws/byoc-aws-access#provision-aws-access) to provision AWS access for StreamNative Cloud. This step ensures StreamNative has the necessary permissions to manage resources in your AWS account.

Once completed, please note the account ID of the AWS account you have granted access to StreamNative Cloud. You will use this account ID to create a Cloud Connection.

## Step 3: Create a Cloud Connection

<Tabs>
  <Tab title="Cloud Console">
    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Cloud Environments**.

    2. Select **Cloud Connections** tab and click **New Cloud Connection**.

    3. In the **Create connection** dialog, enter the following information:

       * **Name**: Enter a name for the cloud connection. For example, `my-aws-connection`.
       * **Connection provider**: Select **AWS** as the connection provider.
       * **AWS Account ID**: Enter the account ID of the AWS account you noted in the previous step.
       * Check **Confirm if vendor access Terraform module is executed**.

    4. Click **Submit**.

    5. The cloud connection creation process will start. Once completed, you can see the status of the cloud connection turned to `Connected` in the **Cloud Connections** tab.
  </Tab>
</Tabs>

## Step 4: Create a Cloud Environment

Once you have created a cloud connection, you can create a cloud environment and provision a BYOC instance.

<Tabs>
  <Tab title="Cloud Console">
    1. Navigate to [Organization Dashboard](/cloud/get-started/cloud-console#organization-dashboard).

    2. Select **Instances** at the left navigation pane.

    3. On the **Instances** page, click **+ New Instance**.

    4. On the **Choose the deployment type for your instance** page, click **Deploy BYOC**.

    5. You will see a dialog "Cloud Environment required". Click **Create** button to create a cloud environment.

    6. On the **Cloud Connection** page, select the cloud connection you created in the previous step. In this case, it is `my-aws-connection`. Then click **Environment setup**.

    7. Then fill out the information for the cloud environment.

       * **Region**: Select the region where you want to deploy the BYOC cluster. In this example, it is `us-west-2`.
       * **Environment tag**: Enter a tag for the cloud environment. For example, `poc`.
       * **Configure StreamNative Managed VPC Network**:
         * **Network CIDR**: By default, it creates a StreamNative Managed VPC with a CIDR of `10.0.0.0/16`. If you need to specify a different CIDR, you can enter it here.
       * **Default Gateway**: You can configure how do you want to expose your BYOC cluster, whether it is **public** or **private**. By default, it is **public**.

    8. Click **Create**.
  </Tab>
</Tabs>

The provisioning process of a cloud environment usually takes about 40 minutes to complete. You can safely close the page and come back later. You will also receive an email notification when the cloud environment is ready.

For more information about provisioning BYOC infrastructure, see [Provision BYOC Infrastructure](/cloud/clusters/byoc/byoc-overview).

## Step 5: Create a StreamNative Instance & Cluster

Once the cloud environment is ready, you can create a Pulsar instance.

<Tabs>
  <Tab title="Cloud Console">
    1. Navigate to [Organization Dashboard](/cloud/get-started/cloud-console#organization-dashboard).

    2. Select **Instances** at the left navigation pane.

    3. On the **Instances** page, click **+ New Instance**.

    4. On the **Choose the deployment type for your instance** page, click **Deploy BYOC** again. You will not see the dialog "Cloud Environment required" this time.

    5. On the **Instance Configuration** page, fill out the information for the Pulsar instance. Then, click **Cluster Location** to start the cluster creation process.

       * **Instance Name**: Enter a name for the Pulsar instance. For example, `my-pulsar-instance`.
       * **Cloud Connection**: Select the cloud connection you created in the previous step. In this example, it is `my-aws-connection`.

    6. On the **Cluster Location** page, fill out the information for the cluster.

       * **Cluster Name**: Enter a name for the cluster. For example, `my-byoc-cluster`.
       * **Cloud Environment**: Select the cloud environment you created in the previous step. In this example, it is `aws-usw2-poc-<random-suffix>`.
       * **Cluster Profile**: Select the cluster profile. In this example, it is 'Latency Optimized'
       * **Availability Zone**: Select the Availability Zone. In this example, it is 'Multi AZ'

    7. Click **Cluster Operations** to choose the release channel for the cluster and configure the cluster.

       * **Release Channel**: Select the release channel for the cluster. In this example, it is `LTS`.
       * **Features**: Select the features for the cluster. You can keep the default features or customize it.
       * **Maintenance Window**: If you have a Enterprise or Production support plan, you can customize the maintenance window for the cluster. Otherwise, you can skip this step.
       * **Custom Configuration**: If you need to customize the cluster configuration, you can expand the **Add optional custom configuration** section.

    8. Click **Cluster Size** to configure the cluster size.

       * You can use the slider to adjust the throughput accordingly.
       * You can also manually configure the number of brokers, bookies, and their corresponding resources in the **Advanced** section.

       At the right navigation pane, you can see the estimated cost for the cluster.

    9. Click **Finish** to start the cluster creation process.

    The cluster page appears, displaying the cluster creation process. Depending on the chosen cloud provider and other settings, it may take several minutes to provision the cluster.
    Once the cluster has been provisioned, the page will show "Cluster Provisioned successfully" and you can click **Go To The Dashboard** to access the **Cluster Dashboard** page.
  </Tab>
</Tabs>

Now you can get started configuring apps and data on your new cluster.

## Step 6: Create a service account

To interact with your cluster by producing and consuming messages, you need to set up authentication and authorization. This is done by creating a [Service Account](/cloud/security/authentication/service-accounts/service-accounts), which serves as an identity for authenticating and authorizing access to the cluster. The service account will provide the necessary credentials for your applications to securely connect and perform operations on the Pulsar cluster.

<Tabs>
  <Tab title="Cloud Console">
    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Accounts & Accesses**.

    2. On the left navigation pane, click **Service Accounts**.

    3. On the **Service Account** page, click **+ New**.

    4. On the **Create Service Account** dialog, enter a name for the service account, and then click **Confirm**.

    5. On the **Service Account** page, in the row of the service account you just created, click the **...** icon, and select **Create API Key** in the dropdown menu.

    6. On the **New API Key** dialog:

       * Enter a name for the API key
       * Set the expiration date
       * Select the instance you created in previous step
       * Write a description for the API key
       * Click **Confirm**

    <Note title="Note">
      An API key and associated secret apply to the active StreamNative instance. If you add a new instance, you must create a new API key for producers and consumers on the new Pulsar instance. For more information, see [Use API Keys to Authenticate to StreamNative Cloud](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview).
    </Note>

    7. After the API key is created, you can see the API key shown in the **New API Key** dialog. Click the **Copy and close** button to copy the API key to your clipboard. **Please note that you cannot retrieve the API key later after closing the dialog.**
  </Tab>
</Tabs>

## Step 7: Create Tenant and Namespace, and Authorize the Service Account

After creating the service account and obtaining the API key, the next crucial step is to authorize the service account. This process grants the necessary permissions for the service account to interact with your StreamNative Cloud cluster.

Authorization involves setting up Access Control Lists (ACLs) that define what actions the service account can perform. Typically, you'll want to grant permissions for producing and consuming messages on specific topics or namespaces.

For more information about authorization, see [Authorization and ACLs](/cloud/security/access/access-control-lists/authorization-and-acls).

<Tabs>
  <Tab title="Cloud Console">
    1. Go back to the **Cluster Dashboard** page.

    2. On the left navigation pane, click **Instances**.

    3. On the **Instances** page, click the name of the instance you created in Step 2.

    4. On the **Cluster Dashboard** page, click **Tenants** on the left navigation pane.

    5. On the **Tenants** page, click **+ New Tenant**.

    6. On the **New Tenant** dialog:

       * Enter a name for the tenant
       * Select your user account as the **Admin role**
       * Select the cluster created in Step 2 as **Allowed clusters**
       * Click **Confirm**

    7. On the **Tenants** page, click the name of the tenant you just created. You will be directed to the **Tenant Dashboard** page.

    8. On the **Tenant Dashboard** page, click **Namespaces** on the left navigation pane.

    9. On the **Namespaces** page, click **New Namespace**.

    10. On the **New Namespace** dialog:

        * Enter a name for the namespace
        * Select the cluster created in Step 2 as **Allowed clusters** and **Replication clusters**
        * Click **Confirm**

    11. On the **Namespaces** page, click the name of the namespace you just created. You will be directed to the **Namespace Dashboard** page.

    12. On the **Namespace Dashboard** page, click **Configuration** on the left navigation pane.

    13. On the **Namespace configuration** page, click **ADD ROLE**. Select the name of the service account you just created, and choose the **consume** and **produce** permissions. This grants your service account the `produce` and `consume` permissions for this namespace.
  </Tab>
</Tabs>

Now you have created a tenant, namespace, and granted the service account the `produce` and `consume` permissions for this namespace. You can now continue on to building a Python app, connecting to the cluster, and producing and consuming messages.

## Step 8: Produce and consume messages

<Note title="Note">
  This QuickStart provides you with an example Python app to get you up and running with consuming and producing messages. This is a simple example and is not intended for production environments.
</Note>

### Create a producer/consumer

<Tabs>
  <Tab title="Cloud Console">
    1. Return to the StreamNative Cloud Console and go to the "Cluster Dashboard" page.

    2. On the left navigation pane, click **Pulsar Clients**.

    3. On the **Pulsar client setup** page, click the **Code libraries** tab, follow the setup wizard to get the sample codes for your producer and consumer.

       a. Select **Python** as the client library and click **Next**.

       b. Select the service account you created and click **Next**.

       c. Select **API Key** as the authentication type and click **Next**.

       d. Install `pulsar-client` python client library.

       ```bash theme={null}
       pip3 install pulsar-client
       ```

       e. Select the target tenant, namespace, topic, and subscription.

       f. You are now ready to copy the auto-generated sample codes.

    4. Return to your text editor and create two new files: `producer.py` and `consumer.py`. Copy and paste the sample code for the producer into `producer.py` and the sample code for the consumer into `consumer.py`. In both files, replace **`<JWT Token>`** with the API key you copied from the Service Account page.
  </Tab>
</Tabs>

### Run the clients to produce and consume your first message

1. Open a terminal window, navigate to the folder containing the `consumer.py` file, and run the following command:

   ```bash theme={null}
   python3 consumer.py
   ```

2. Open a second terminal window, navigate to the folder containing the `producer.py` file, and run the following command:

   ```bash theme={null}
   python3 producer.py
   ```

3. Return to the first terminal window. You should see the following:

   ```bash theme={null}
   Received message 'Hello-0' id='<message-id>'
   Received message 'Hello-1' id='<message-id>'
   Received message 'Hello-2' id='<message-id>'
   Received message 'Hello-3' id='<message-id>'
   Received message 'Hello-4' id='<message-id>'
   Received message 'Hello-5' id='<message-id>'
   Received message 'Hello-6' id='<message-id>'
   Received message 'Hello-7' id='<message-id>'
   Received message 'Hello-8' id='<message-id>'
   Received message 'Hello-9' id='<message-id>'
   ```

You have now produced and consumed your first 10 messages.

## Next steps

* After you have successfully provisioned a **BYOC** cluster and connected to the cluster, you can learn more about working with StreamNative Cloud by reading through [Cloud Console basics](/cloud/get-started/cloud-console).

* If you want to learn more about Pulsar, Kafka, and StreamNative Cloud, take our developer courses at the [StreamNative Developer Portal](https://streamnative.io/dev-portal).


# Quick Start for StreamNative Cloud
Source: https://docs.streamnative.io/cloud/get-started/quickstart-console



StreamNative Cloud is a resilient, scalable, data streaming service , delivered as a fully managed Pulsar and Kafka service.

StreamNative Cloud provides multiple interfaces for management and interaction:

1. **StreamNative Cloud Console**: A user-friendly web-based interface for managing cluster resources, configuring settings, and handling billing.

2. **Command-Line Interface (CLI)**:

   * [StreamNative CLI (`snctl`)](/tools/cli/snctl/snctl-overview): The unified command-line interface for deploying and managing StreamNative Cloud infrastructure and interacting directly with your Pulsar clusters and Kafka-protocol endpoints.
   * [Pulsar CLI (`pulsarctl`)](/tools/cli/pulsarctl/pulsarctl-overview): For managing cluster-specific resources, such as tenants, namespaces, and topics.

3. **REST APIs**: For programmatic access and integration with other systems.
   * [StreamNative Cloud API](/api-references/cloudapi/cloud-api)
   * [Pulsar Admin REST API](https://pulsar.apache.org/docs/en/admin-api-overview/)
   * [Kafka Admin API](https://kafka.apache.org/documentation/#adminapi)

These tools provide flexibility in how you interact with and manage your StreamNative Cloud environment, catering to different user preferences and use cases.

<Note title="Get started for Free">
  [Sign up for StreamNative Cloud](https://console.streamnative.cloud/?defaultMethod=signup) and get \$200 of free credits. No credit card required.
</Note>

This quick start guides you through getting started with StreamNative Cloud using a [Serverless cluster](/cloud/clusters/cluster-types#serverless-clusters). It demonstrates how to use StreamNative Cloud to create topics, produce data to the cluster, and consume data from it.

<Note title="Note">
  This QuickStart assumes you are familiar with the [basics concepts of Apache Pulsar](/cloud/references/glossary).
</Note>

## Prerequisites

* Access to [StreamNative Cloud](https://console.streamnative.cloud/?defaultMethod=signup).
* Internet connectivity.
* Ensure you have installed Python 3.0 or higher versions and the Pulsar Python client.

  ```bash theme={null}
  python -m pip install pulsar-client
  ```

## Step 1: Sign up

<Note title="Note">
  If you have an email account configured for using Single Sign-On (SSO) with StreamNative Cloud, use that email address and password when signing up.
</Note>

To sign up, navigate to the [StreamNative Cloud Console login page](https://console.streamnative.cloud/?defaultMethod=signup). Follow the prompts to create an account.

After you click **Finish**, you might have to wait briefly for your first organization to be created.

After your new organization is created, continue on to creating your first instance and cluster.

## Step 2: Create a StreamNative instance and cluster

You first need to create an [instance](/cloud/references/glossary#instance) and a [cluster](/cloud/references/glossary#cluster).

<Tabs>
  <Tab title="Cloud Console">
    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Organizations** to list your created organizations.

    2. Click the name of your first organization. In the figure below, the organization name is `Demos`.

       <img alt="screenshot of organization section" />

    3. On the **Select an Instance** card of the **Organization Dashboard** page, click the `+` button.

    4. On the **Choose the deployment type for your instance**, click **Deploy Serverless** to start the instance creation process.

    5. On the **Instance Configuration** page, enter a name for your instance, and select a Cloud Provider. The instance name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

    6. Click **Cluster Location** to start the cluster creation process.

    7. On the **Cluster Location** page, enter a name for your cluster, select the cluster location, and then click **Finish**. The cluster name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

    The cluster page appears, displaying the cluster creation process. Depending on the chosen cloud provider and other settings, it may take a few seconds to several minutes to provision the cluster. Once the cluster has been provisioned, the **Cluster Dashboard** page will be displayed.

    Now you can get started configuring apps and data on your new cluster.
  </Tab>
</Tabs>

## Step 3: Create a service account

To interact with your cluster by producing and consuming messages, you need to set up authentication and authorization. This is done by creating a [Service Account](/cloud/security/authentication/service-accounts/service-accounts), which serves as an identity for authenticating and authorizing access to the cluster. The service account will provide the necessary credentials for your applications to securely connect and perform operations on the Pulsar cluster.

<Note title="Note">
  Currently, you can't edit a service account. If you need a service account to have Super Admin access, make sure to enable it when creating the service account. By default, service accounts do not have Super Admin enabled.
</Note>

<Tabs>
  <Tab title="Cloud Console">
    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Accounts & Accesses**.

    2. On the left navigation pane, click **Service Accounts**.

    3. On the **Service Account** page, click **+ New Service Account**.

    4. On the **Create Service Account** dialog, enter a name for the service account, and then click **Confirm**.

    5. On the **Service Account** page, in the row of the service account you just created, click the **...** icon, and select **Create API Key** in the dropdown menu.

    6. On the **New API Key** dialog:

       * Enter a name for the API key
       * Set the expiration date
       * Select the instance you created in Step 2
       * Write a description for the API key
       * Click **Confirm**

    <Note title="Note">
      An API key and associated secret apply to the active StreamNative instance. If you add a new instance, you must create a new API key for producers and consumers on the new Pulsar instance. For more information, see [Use API Keys to Authenticate to StreamNative Cloud](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview).
    </Note>

    7. After the API key is created, you can see the API key shown in the **New API Key** dialog. Click the **Copy and close** button to copy the API key to your clipboard. **Please note that you cannot retrieve the API key later after closing the dialog.**
  </Tab>
</Tabs>

## Step 4: Create Tenant and Namespace, and Authorize the Service Account

After creating the service account and obtaining the API key, the next crucial step is to authorize the service account. This process grants the necessary permissions for the service account to interact with your StreamNative Cloud cluster.

Authorization involves setting up Access Control Lists (ACLs) that define what actions the service account can perform. Typically, you'll want to grant permissions for producing and consuming messages on specific topics or namespaces.

For more information about authorization, see [Authorization and ACLs](/cloud/security/access/access-control-lists/authorization-and-acls).

<Tabs>
  <Tab title="Cloud Console">
    1. Go back to the **Cluster Dashboard** page.

    2. On the left navigation pane, click **Instances**.

    3. On the **Instances** page, click the name of the instance you created in Step 2.

    4. On the **Cluster Dashboard** page, click **Tenants** on the left navigation pane.

    5. On the **Tenants** page, click **+ New Tenant**.

    6. On the **New Tenant** dialog:

       * Enter a name for the tenant
       * Select your user account as the **Admin role**
       * Select the cluster created in Step 2 as **Allowed clusters**
       * Click **Confirm**

    7. On the **Tenants** page, click the name of the tenant you just created. You will be directed to the **Tenant Dashboard** page.

    8. On the **Tenant Dashboard** page, click **Namespaces** on the left navigation pane.

    9. On the **Namespaces** page, click **New Namespace**.

    10. On the **New Namespace** dialog:

        * Enter a name for the namespace
        * Select the cluster created in Step 2 as **Allowed clusters** and **Replication clusters**
        * Click **Confirm**

    11. On the **Namespaces** page, click the name of the namespace you just created. You will be directed to the **Namespace Dashboard** page.

    12. On the **Namespace Dashboard** page, click **Configuration** on the left navigation pane.

    13. On the **Namespace configuration** page, click **ADD ROLE**. Select the name of the service account you just created, and choose the **consume** and **produce** permissions. This grants your service account the `produce` and `consume` permissions for this namespace.
  </Tab>
</Tabs>

Now you have created a tenant, namespace, and granted the service account the `produce` and `consume` permissions for this namespace. You can now continue on to building a Python app, connecting to the cluster, and producing and consuming messages.

## Step 5: Produce and consume messages

<Note title="Note">
  This QuickStart provides you with an example Python app to get you up and running with consuming and producing messages. This is a simple example and is not intended for production environments.
</Note>

### Create a producer/consumer

<Tabs>
  <Tab title="Cloud Console">
    1. Return to the StreamNative Cloud Console and go to the "Cluster Dashboard" page.

    2. On the left navigation pane, click **Pulsar Clients**.

    3. On the **Pulsar client setup** page, click the **Code libraries** tab, follow the setup wizard to get the sample codes for your producer and consumer.

       a. Select **Python** as the client library and click **Next**.

       b. Select the service account you created and click **Next**.

       c. Select **API Key** as the authentication type and click **Next**.

       d. Install `pulsar-client` python client library.

       ```bash theme={null}
       pip3 install pulsar-client
       ```

       e. Select the target tenant, namespace, topic, and subscription.

       f. You are now ready to copy the auto-generated sample codes.

    4. Return to your text editor and create two new files: `producer.py` and `consumer.py`. Copy and paste the sample code for the producer into `producer.py` and the sample code for the consumer into `consumer.py`. In both files, replace **`<JWT Token>`** with the API key you copied from the Service Account page.
  </Tab>
</Tabs>

### Run the clients to produce and consume your first message

1. Open a terminal window, navigate to the folder containing the `consumer.py` file, and run the following command:

   ```bash theme={null}
   python3 consumer.py
   ```

2. Open a second terminal window, navigate to the folder containing the `producer.py` file, and run the following command:

   ```bash theme={null}
   python3 producer.py
   ```

3. Return to the first terminal window. You should see the following:

   ```bash theme={null}
   Received message 'Hello-0' id='<message-id>'
   Received message 'Hello-1' id='<message-id>'
   Received message 'Hello-2' id='<message-id>'
   Received message 'Hello-3' id='<message-id>'
   Received message 'Hello-4' id='<message-id>'
   Received message 'Hello-5' id='<message-id>'
   Received message 'Hello-6' id='<message-id>'
   Received message 'Hello-7' id='<message-id>'
   Received message 'Hello-8' id='<message-id>'
   Received message 'Hello-9' id='<message-id>'
   ```

You have now produced and consumed your first 10 messages.

## Next steps

* After you have successfully provisioned a **Serverless** cluster and connected to the cluster, you can learn more about working with StreamNative Cloud by reading through [Cloud Console basics](/cloud/get-started/cloud-console).

* If you want to learn more about Pulsar, Kafka, and StreamNative Cloud, take our developer courses at the [StreamNative Developer Portal](https://streamnative.io/dev-portal).

* If you want to create multiple Pulsar clusters replicating messages with geo-replication mechanism, you can learn more by reading through the [Geo Replication on StreamNative Cloud](/cloud/clusters/cloud-geo-replication).


# Overview of StreamNative Catalogs
Source: https://docs.streamnative.io/cloud/lakehouse/catalogs/catalog-overview



The StreamNative Catalog feature allows you to create connections to external catalog providers, such as Databricks Unity Catalog, Snowflake Open Catalog, Snowflake Horizon (Polaris) Catalog, Google BigLake metastore, and Amazon S3 Tables, to stream topic data from StreamNative Cloud as Iceberg tables.

You can register a catalog by selecting a provider, entering the required catalog details, and completing the registration process. StreamNative Cloud supports registering multiple catalogs per cluster, so a single Pulsar or Kafka cluster can land topics into different governance domains across BigQuery and Snowflake.

StreamNative Catalog is scoped at the organization level within StreamNative Cloud. Once a catalog is registered, it can be accessed and used across multiple StreamNative Clusters, enabling centralized catalog management and streamlined data access for streaming workloads.

<img alt="StreamNative Catalogs" />


# Manage StreamNative Catalogs
Source: https://docs.streamnative.io/cloud/lakehouse/catalogs/manage-catalogs



**View Catalogs**

On the **Catalogs** page, you can view a list of all catalogs that have been registered in StreamNative Cloud. The **Catalogs** page displays the following information for each registered catalog: Name, Catalog Type, Table Format, Status, and an Actions menu.

<img alt="StreamNative Catalogs" />

**Catalog Actions**

Each catalog listed on the **Catalogs** page includes an actions menu with the following options: **View Details** and **Delete**.

<img alt="StreamNative Catalogs" />

**Catalog Details**

Clicking on a catalog listed on the **Catalogs** page navigates you to the **Catalog Details** page, where you can view detailed information about the selected catalog.

<img alt="StreamNative Catalogs" />


# Private Networking for Catalog Integration
Source: https://docs.streamnative.io/cloud/lakehouse/catalogs/private-networking/overview



When integrating StreamNative Cloud with external catalog providers in production environments, you may need to configure private network connections to ensure that traffic between StreamNative Cloud and your catalog and storage services does not traverse the public internet.

Private networking for lakehouse catalog integration involves two components:

* **Storage connectivity** — Private network connections between StreamNative Cloud and your object storage (S3, GCS, or Azure Blob Storage).
* **Catalog connectivity** — Private network connections between StreamNative Cloud and your catalog provider.

The following diagram shows the network path between StreamNative Cloud and the catalog and storage services over private connectivity.

```mermaid theme={null}
flowchart TB
    subgraph BYOC["StreamNative BYOC VPC"]
        Cluster["BYOC Cluster"]
        subgraph Endpoints["Private Endpoints"]
            StorageEP["Storage Private Endpoint"]
            CatalogEP["Catalog Private Endpoint"]
        end
    end

    Storage["Object Storage"]
    Catalog["Catalog Provider"]

    Cluster -->|"storage traffic"| StorageEP --> Storage
    Cluster -->|"catalog traffic"| CatalogEP --> Catalog

    classDef byoc fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
    classDef edge fill:#fde68a,stroke:#b45309,color:#451a03
    classDef ext  fill:#ede9fe,stroke:#7c3aed,color:#4c1d95
    class Cluster byoc
    class StorageEP,CatalogEP edge
    class Storage,Catalog ext
    style Endpoints fill:#fffbeb,stroke:#f59e0b,color:#92400e
```

## Prerequisites

* A StreamNative BYOC cluster deployed on AWS, GCP, or Azure.
* A registered catalog in StreamNative Cloud. See [Register Catalog](/cloud/lakehouse/catalogs/register-catalog).
* An active integration with a supported catalog provider. See the [External Tables Integrations](/cloud/lakehouse/external-tables/external-tables-overview) for setup guides.

## Storage private connectivity

Private network connections between StreamNative Cloud and object storage are handled differently depending on the cloud provider.

<Tabs>
  <Tab title="AWS">
    On AWS, StreamNative configures private network connections to Amazon S3 endpoints by default in all StreamNative environments. The S3 VPC endpoint is configured per VPC.

    **No action is required on your side.** All traffic between your StreamNative BYOC cluster and S3 stays within the AWS private network automatically.
  </Tab>

  <Tab title="GCP">
    On GCP, StreamNative configures private network connections to Google Cloud Storage (GCS) endpoints by default in all StreamNative environments. The GCS private endpoint is configured per network.

    **No action is required on your side** in most cases. All traffic between your StreamNative BYOC cluster and GCS stays within the Google private network automatically.

    **Exception — Shared VPC:** If your BYOC cluster runs in a [Shared VPC](https://cloud.google.com/vpc/docs/shared-vpc), StreamNative cannot modify the network in the host project. You must configure private connectivity to GCS yourself by either:

    * Enabling [Private Google Access](https://cloud.google.com/vpc/docs/configure-private-google-access) on the subnets used by the cluster, or
    * Creating a [Private Service Connect endpoint for Google APIs](https://cloud.google.com/vpc/docs/configure-private-service-connect-apis) (see [Shared VPC deployment patterns](https://cloud.google.com/vpc/docs/private-service-connect-deployments) for host-project guidance).
  </Tab>

  <Tab title="Azure">
    On Azure, you must configure a private endpoint for your Azure Storage account to enable private connectivity between StreamNative Cloud and your storage.

    To set up a private endpoint for your storage account, follow the instructions in the Azure documentation:
    [Configure Azure Storage private endpoints](https://learn.microsoft.com/en-us/azure/storage/common/storage-private-endpoints).
  </Tab>
</Tabs>

## Catalog private connectivity

To establish private network connections between StreamNative Cloud and your catalog provider, follow the guide for your catalog provider:

<CardGroup>
  <Card title="Snowflake Open Catalog" icon="snowflake" href="/cloud/lakehouse/catalogs/private-networking/snowflake-open-catalog">
    Configure private connectivity to Snowflake Open Catalog using AWS PrivateLink, GCP Private Service Connect, or Azure Private Link.
  </Card>

  <Card title="Snowflake Horizon Catalog" icon="snowflake" href="/cloud/lakehouse/catalogs/private-networking/snowflake-horizon-catalog">
    Configure private connectivity to Snowflake Horizon Catalog using AWS PrivateLink, GCP Private Service Connect, or Azure Private Link.
  </Card>

  <Card title="Databricks Delta Lake" icon="database" href="/cloud/lakehouse/catalogs/private-networking/databricks-unity-catalog">
    Configure private connectivity to Databricks Unity Catalog for Delta Lake using AWS PrivateLink, GCP Private Service Connect, or Azure Private Link.
  </Card>

  <Card title="Databricks Iceberg" icon="database" href="/cloud/lakehouse/catalogs/private-networking/databricks-iceberg">
    Configure private connectivity to Databricks Unity Catalog for Iceberg using AWS PrivateLink, GCP Private Service Connect, or Azure Private Link.
  </Card>

  <Card title="Amazon S3 Tables" icon="aws" href="/cloud/lakehouse/catalogs/private-networking/s3-tables">
    Private connectivity for Amazon S3 Tables on AWS. Configured by default — no action required.
  </Card>

  <Card title="Google BigLake" icon="google" href="/cloud/lakehouse/catalogs/private-networking/google-biglake">
    Private connectivity for Google BigLake on GCP. Configured by default — no action required.
  </Card>
</CardGroup>


# Register Catalog
Source: https://docs.streamnative.io/cloud/lakehouse/catalogs/register-catalog



To register a catalog in StreamNative Cloud, navigate to the **Catalogs** page and click **Register Catalog**.

<img alt="StreamNative Catalogs" />

On the **Register Catalog** form, provide the required details based on the selected Catalog Provider.

### Databricks Unity Catalog (Delta Lake and Iceberg)

* **Name** – Enter a unique name for the catalog in StreamNative Cloud.
* **Catalog Provider** – Select the catalog provider where you want to ingest data.
* **Catalog Name** – Enter the name of the catalog that exists in Databricks Unity Catalog.
* **URI** – Provide the URI of the catalog.
* **Authentication Type** – Select the appropriate authentication method for connecting to the catalog provider.

### Snowflake Open Catalog

* **Name** – Enter a unique name for the catalog in StreamNative Cloud.
* **Catalog Provider** – Select the catalog provider where you want to ingest data.
* **Warehouse** – Enter the name of the warehouse that exists in Snowflake Open Catalog.
* **URI** – Provide the URI of the catalog.
* **Authentication Type** – Select the appropriate authentication method for connecting to the catalog provider.

### Amazon S3 Tables

* **Name** – Enter a unique name for the catalog in StreamNative Cloud.
* **Catalog Provider** – Select the catalog provider where you want to ingest data.
* **S3 Table Bucket** – Enter the ARN of the S3 Table bucket.

### Google BigLake metastore

The Google BigLake metastore is an Iceberg REST catalog for Google Cloud that integrates with BigQuery, so streamed tables are queryable from BigQuery without an extra import step.

* **Name** – Enter a unique name for the catalog in StreamNative Cloud.
* **Catalog Provider** – Select Google BigLake as the catalog provider.
* **Google Project** – Enter the name of the Google project.
* **Warehouse** – Enter the name of the Google Cloud Storage Warehouse.

### Snowflake Horizon (Polaris) Catalog

The Snowflake Horizon Catalog is an Iceberg REST catalog hosted on Snowflake. Access is governed by Snowflake roles, so existing Snowflake RBAC policies apply to streamed tables.

* **Name** – Enter a unique name for the catalog in StreamNative Cloud.
* **Catalog Provider** – Select Snowflake Horizon as the catalog provider.
* **Account** – Enter the Snowflake account identifier hosting the Horizon catalog.
* **Warehouse** – Enter the name of the warehouse that exists in Snowflake Horizon.
* **URI** – Provide the URI of the Polaris REST catalog endpoint.
* **Authentication Type** – Select the appropriate authentication method (for example, OAuth2 client credentials) for connecting to Snowflake Horizon.

## Register multiple catalogs per cluster

A single StreamNative cluster can be associated with multiple registered catalogs. This lets you route different topics to different catalogs — for example, landing some topics into BigLake for BigQuery analytics and others into Snowflake Horizon for governed Snowflake access — all from one Pulsar or Kafka cluster. Repeat the registration steps above for each catalog you want to make available to the cluster.

<img alt="StreamNative Catalogs" />


# Enable Lakehouse Table
Source: https://docs.streamnative.io/cloud/lakehouse/enable-lakehouse-integration



You can enable the Lakehouse Table for a cluster, a namespace, or a single topic from the StreamNative Cloud Console. Once enabled, topic data is delivered to the configured external lakehouse catalog automatically.

> Currently, only the **Deliver to External Table** mode is supported. The **Expose Internal Table** option is coming soon.

## Prerequisites

* A cluster (Pulsar or Kafka) running on a profile that supports Lakehouse Table delivery. See [Lakehouse Table Overview](/cloud/lakehouse/lakehouse-table-overview) for the supported cluster types and profiles.
* A registered catalog in your organization. If you have not registered a catalog yet, see [Register Catalog](/cloud/lakehouse/catalogs/register-catalog).

## Enable Lakehouse Table at the cluster level

You can enable the Lakehouse Table at the cluster level either when you create a new cluster or for an existing cluster.

### Enable when creating a new cluster

When you create a new cluster, the cluster wizard includes a **Lakehouse Table** configuration step.

<Frame>
  <img alt="Lakehouse Table step in the cluster creation wizard" />
</Frame>

1. Toggle on **Enable Lakehouse Table**.

2. Select a **Catalog Provider**. StreamNative Cloud supports the following providers:
   * Databricks Unity Catalog for Iceberg (Iceberg)
   * Databricks Unity Catalog for Delta Lake (Delta Lake)
   * Snowflake Open Catalog (Iceberg)
   * Snowflake Horizon Catalog (Iceberg)
   * Amazon S3Table (Iceberg)
   * Google BigLake (Iceberg) -- can only be configured **after** the cluster is created. Skip the Lakehouse Table step at cluster creation and enable it later from the existing cluster page.

3. Select the target **Catalog** that belongs to the catalog provider you selected. If the catalog you need is not registered yet, click **Register new catalog** to open the registration page. See [Register Catalog](/cloud/lakehouse/catalogs/register-catalog).

   <Frame>
     <img alt="Select a registered catalog or register a new one" />
   </Frame>

4. (Optional) Enable **Apply lakehouse setting to all topics** to automatically deliver every existing and new topic in the cluster to the lakehouse table. If this option is disabled, you can enable Lakehouse delivery later at the namespace or topic level.

### Enable for an existing cluster

For an existing Pulsar cluster (latency-optimized profile or cost-optimized profile), open the cluster page and click **Enable Lakehouse Table**.

<Frame>
  <img alt="Enable Lakehouse Table button on the cluster page" />
</Frame>

In the dialog, select a target catalog from the dropdown. If the catalog is not registered yet, click **Register new catalog** to register one.

<Frame>
  <img alt="Enable Lakehouse Table dialog with catalog dropdown" />
</Frame>

After you confirm, every topic in the cluster begins delivering data to the lakehouse table automatically.

## Enable Lakehouse Table at the namespace level

You can also enable the Lakehouse Table at the namespace level. Open the namespace and click **Enable Lakehouse Table**, then select a registered target catalog. If the catalog you need is not registered, click **Register new catalog** to register one.

> Namespace-level enablement is supported only on Pulsar clusters (latency-optimized and cost-optimized profiles). Kafka clusters do not support namespace-level enablement; for Kafka clusters, enable Lakehouse Table at the cluster level instead.

<Frame>
  <img alt="Enable Lakehouse Table at the namespace level" />
</Frame>

After you confirm, every topic in the namespace begins delivering data to the lakehouse table automatically.

## Enable Lakehouse Table at the topic level

You can enable the Lakehouse Table for a single topic. Open the topic and click **Enable Lakehouse Table**, then select a registered target catalog. If the catalog you need is not registered, click **Register new catalog** to register one.

> Topic-level enablement is supported only on Pulsar clusters (latency-optimized and cost-optimized profiles). Kafka clusters do not support topic-level enablement; for Kafka clusters, enable Lakehouse Table at the cluster level instead.

<Frame>
  <img alt="Enable Lakehouse Table at the topic level" />
</Frame>

After you confirm, the topic data is delivered to the lakehouse table automatically.

## Configuration override priority

When the same setting is configured at multiple levels, the most specific level wins:

```
Topic settings
    ↓ (override)
Namespace settings
    ↓ (override)
Cluster settings
```

For example, if Lakehouse delivery is enabled at the cluster level but disabled at a specific topic, that topic's data is not delivered.

### Per-namespace and per-topic catalog selection

Different namespaces in the same cluster can deliver data to **different catalogs**, and individual topics within a namespace can also override the namespace's catalog and deliver to a different one. For example, namespace `analytics/finance` can be configured to use a Snowflake Open Catalog, namespace `analytics/marketing` in the same cluster can use a Databricks Unity Catalog, and a single topic `analytics/marketing/realtime-events` inside `analytics/marketing` can be redirected to an Amazon S3Tables catalog. To assign different catalogs, enable the Lakehouse Table separately on each namespace or topic and select the desired catalog.

### Catalog override order

The catalog used for a topic is resolved in the same most-specific-wins order:

```
Topic catalog
    ↓ (overrides)
Namespace catalog
    ↓ (overrides)
Cluster catalog
```

If a catalog is set on the topic, that catalog is used regardless of the namespace or cluster catalog. If only the namespace catalog is set, every topic in that namespace uses the namespace's catalog. If neither is set, the cluster catalog is used.

## Next Steps

* Monitor data delivery progress for a topic from the Cloud Console: [Lakehouse Observability -- Monitor data delivery progress in the Cloud Console](/cloud/lakehouse/lakehouse-observability#monitor-data-delivery-progress-in-the-cloud-console).
* See how Pulsar topic names are mapped to lakehouse namespace and table names: [Lakehouse Table Overview -- Topic to lakehouse identifier mapping](/cloud/lakehouse/lakehouse-table-overview#topic-to-lakehouse-identifier-mapping).
* Explore features:
  * [Schema Evolution](/cloud/lakehouse/features/schema-evolution)
  * [Variant Type](/cloud/lakehouse/features/variant-type)
  * [Partition Key](/cloud/lakehouse/features/iceberg-partition-key)
  * [Upsert](/cloud/lakehouse/features/iceberg-upsert)
  * [Persist Key](/cloud/lakehouse/features/persist-key)
  * [Persist Extra Metadata](/cloud/lakehouse/features/persist-extra-metadata)


# Overview - External Tables
Source: https://docs.streamnative.io/cloud/lakehouse/external-tables/external-tables-overview



**External Tables**

External Tables are tables backed by StreamNative topics but represented in open table formats, such as Apache Iceberg or Delta Lake, with both data and metadata stored directly in user-managed object storage.

**Key Characteristics**

* **Open Table Formats** -

  Data is written as Parquet files and managed using Iceberg or Delta metadata logs.

* **Object Storage–Backed** -
  Tables reside in S3, GCS, Azure Blob, or other object storage systems.

Externally Managed
StreamNative Cloud materializes the data, but table lifecycle, optimization, schema governance, and cataloging are typically managed by a Lakehouse vendor, such as:

* Databricks Unity Catalog
* Snowflake Open Catalog
* Amazon S3 Tables

**High Interoperability**

External Tables are fully queryable by external compute platforms, BI tools, and SQL engines.

**When to Use External Tables**

* You want deep integration with an existing Lakehouse platform.
* You require open, interoperable formats for analytics, AI/ML, or downstream workloads.
* You prefer to manage optimization (compaction, vacuum, retention) in systems like Databricks, Snowflake, or Polaris.


# Partition Key
Source: https://docs.streamnative.io/cloud/lakehouse/features/iceberg-partition-key



Partition keys control how data is organized into partitions within the Iceberg table. Partitioning improves query performance by enabling partition pruning.

`partition.key` is a [dynamic configuration](../dynamic-configuration) key that takes effect **only at the topic level**. Setting it at the cluster or namespace level has no effect.

> **Cluster-name prefix:** All dynamic configuration keys must be prefixed with the cluster name (for example, `<cluster-name>.partition.key`). The cluster name is the value of `clusterName` in `conf/broker.conf` -- see [Finding the Cluster Name](../dynamic-configuration#finding-the-cluster-name). The examples below use `private-cloud` as the cluster name; replace it with the name of your cluster.

> **Cardinality limit:** Keep the **total number of partition values across all levels under 10** (the cardinality of `key1 × key2 × ... × keyN` should not exceed 10). If the partitioning would produce more than 10 distinct partition values, use the `bucket[N]` transform to bound it. Excessive partitions cause many small files, which degrade write throughput and query performance.

## Configuration Format

The partition key is specified as a JSON array. Each element has three fields:

| Field          | Required | Description                                                   |
| -------------- | -------- | ------------------------------------------------------------- |
| `sourceColumn` | Yes      | The field name from the topic schema                          |
| `transform`    | No       | Iceberg partition transform function. Defaults to `identity`. |
| `targetName`   | No       | Custom name for the transformed partition column              |

### Supported Iceberg Transforms

| Transform     | Description                      |
| ------------- | -------------------------------- |
| `identity`    | Use the field value as-is        |
| `bucket[N]`   | Hash into N buckets              |
| `truncate[N]` | Truncate strings to N characters |
| `year`        | Extract year from timestamp      |
| `month`       | Extract month from timestamp     |
| `day`         | Extract day from timestamp       |
| `hour`        | Extract hour from timestamp      |

For full semantics, see the [Iceberg partition transforms specification](https://iceberg.apache.org/spec/#partition-transforms).

## Apply at Topic Level

```bash theme={null}
bin/pulsar-admin topics update-properties \
  -p private-cloud.partition.key='[{\"sourceColumn\":\"<field>\",\"transform\":\"<transform>\",\"targetName\":\"<name>\"}]' \
  persistent://<tenant>/<namespace>/<topic>
```

> Setting `partition.key` at the cluster or namespace level has no effect. Apply it on the topic only.

## Example

Configure two partition keys on a topic:

* `timestamp` -- bucketed by hour, named `ts_hour`
* `address` -- truncated to 7 characters, named `t_address`

```bash theme={null}
bin/pulsar-admin topics update-properties \
  -p private-cloud.partition.key="[{\"sourceColumn\":\"timestamp\",\"transform\":\"hour\",\"targetName\":\"ts_hour\"},{\"sourceColumn\":\"address\",\"transform\":\"truncate[7]\",\"targetName\":\"t_address\"}]" \
  persistent://public/default/events
```

## Important Notes

1. The `sourceColumn` value must reference a field that exists in the topic schema.
2. The `targetName` is the name produced after applying the transform; it does not need to exist in the topic schema.
3. The JSON value must be a valid JSON array. When passing it on the shell, escape inner double quotes (`\"`).
4. If the JSON cannot be parsed, the system falls back to a non-partitioned table.
5. **Keep the total cardinality of partition values under 10.** If a column has high cardinality, wrap it with `bucket[N]` to bound the number of partitions (for example, `{"sourceColumn":"userId","transform":"bucket[8]"}`). High partition counts produce many small files and degrade performance.

## Related

* [Dynamic Configuration Guide](../dynamic-configuration) -- Cluster-name prefix, override priority, and apply procedure
* [Upsert](/cloud/lakehouse/features/iceberg-upsert) -- Combining partition keys with upsert has compatibility constraints


# Upsert
Source: https://docs.streamnative.io/cloud/lakehouse/features/iceberg-upsert



Upsert mode enables deduplication of records based on a primary key. When multiple records with the same key arrive, only the latest value is retained in the lakehouse table.

`upsert.mode.enabled` and `identifier.fields` are [dynamic configuration](../dynamic-configuration) keys.

> **Cluster-name prefix:** All dynamic configuration keys must be prefixed with the cluster name (for example, `<cluster-name>.upsert.mode.enabled`). The cluster name is the value of `clusterName` in `conf/broker.conf` -- see [Finding the Cluster Name](../dynamic-configuration#finding-the-cluster-name). The examples below use `private-cloud` as the cluster name; replace it with the name of your cluster.

## Configuration Keys

| Key                                  | Scope                       | Description                                                                                                           | Default |
| ------------------------------------ | --------------------------- | --------------------------------------------------------------------------------------------------------------------- | ------- |
| `<cluster-name>.upsert.mode.enabled` | Cluster / Namespace / Topic | Enable upsert mode (`true` / `false`). More-specific scopes override broader ones.                                    | `false` |
| `<cluster-name>.identifier.fields`   | Topic only                  | Comma-separated list of fields used as the primary key. Setting this at the cluster or namespace level has no effect. | --      |

## How Upsert Works

StreamNative Ursa implements upsert using **Iceberg equal-delete** files. For each upsert write, the compaction service emits an equal-delete entry that matches the previous record by the configured identifier fields, followed by the new record. Readers reconcile the delete and the new value at query time.

### Catalog Compatibility

Because upsert depends on equal-delete files, the underlying catalog and query engine must support reading Iceberg equal deletes. Catalogs differ in their delete-file support:

| Catalog                                      | Equal Delete              | Upsert Support     |
| -------------------------------------------- | ------------------------- | ------------------ |
| Snowflake Open Catalog (Polaris)             | No (position delete only) | Not supported      |
| Snowflake Horizon Catalog                    | No (position delete only) | Not supported      |
| Databricks Unity Catalog (Managed Iceberg)   | No (position delete only) | Not supported      |
| AWS S3Table                                  | Yes                       | Supported          |
| Google BigLake                               | Under confirmation        | Under confirmation |
| Iceberg Hadoop catalog (no external catalog) | Yes                       | Supported          |

If you need upsert with a catalog that only supports position deletes, the recommended approach is to use the Hadoop catalog or AWS S3Table for the affected tables.

## Apply at Cluster Level

Cluster-level properties are stored in the `sn/system` namespace. Cluster-level upsert applies as the default for every namespace and topic that does not override it.

```bash theme={null}
bin/pulsar-admin namespaces set-properties \
  -p private-cloud.cluster.upsert.mode.enabled=true \
  sn/system
```

## Apply at Namespace Level

```bash theme={null}
bin/pulsar-admin namespaces set-properties \
  -p private-cloud.upsert.mode.enabled=true \
  <tenant>/<namespace>
```

## Apply at Topic Level

`identifier.fields` must be set on the topic; setting `upsert.mode.enabled` on the topic is also valid and overrides any namespace or cluster default.

```bash theme={null}
bin/pulsar-admin topics update-properties \
  -p private-cloud.upsert.mode.enabled=true \
  -p private-cloud.identifier.fields=<field1>,<field2> \
  persistent://<tenant>/<namespace>/<topic>
```

## Example

Enable upsert on `persistent://public/default/users` with `userId` and `email` as the primary key:

```bash theme={null}
bin/pulsar-admin topics update-properties \
  -p private-cloud.upsert.mode.enabled=true \
  -p private-cloud.identifier.fields=userId,email \
  persistent://public/default/users
```

## Requirements

* Identifier fields **must exist** in the topic schema.
* Identifier fields **must be marked as `required`** in the schema (not nullable).
* The catalog and query engine must support reading Iceberg equal-delete files. See [Catalog Compatibility](#catalog-compatibility).

## Commit Behavior

* **Append-only writes** (default) are batch-committed: multiple Parquet files are grouped and committed to the catalog in a single commit.
* **Upsert writes** are committed one-by-one to ensure correct data ordering. This may slightly reduce throughput compared to append-only mode.

## Limitations

* When upsert is combined with a [partition key](/cloud/lakehouse/features/iceberg-partition-key), identifier fields only deduplicate within the same partition. For example, if the table has `partition.key=region` and `identifier.fields=userId`, two records with the same `userId` but different `region` values are both kept.
* Upsert is supported only for External Tables (SDT).

## Related

* [Dynamic Configuration Guide](../dynamic-configuration) -- Cluster-name prefix, override priority, and apply procedure
* [Partition Key](/cloud/lakehouse/features/iceberg-partition-key) -- Combining partition keys with upsert has compatibility constraints


# Persist Extra Metadata
Source: https://docs.streamnative.io/cloud/lakehouse/features/persist-extra-metadata



The **Persist Extra Metadata** feature injects a `__meta` column into the lakehouse table that contains additional message metadata captured at write time. Use it when downstream queries need access to information that is part of the message envelope but not part of the message body -- for example, the message offset, publish time, event time, or producer-side properties.

## What Gets Persisted

When the feature is enabled, every record written to the lakehouse table includes a `__meta` column populated with the following fields:

| Field             | Description                                                                        |
| ----------------- | ---------------------------------------------------------------------------------- |
| `__messageOffset` | The original Pulsar / Kafka message ID or offset                                   |
| `__publishTime`   | Pulsar publish timestamp (epoch ms)                                                |
| `__eventTime`     | Pulsar event time (epoch ms), if set by the producer                               |
| `__schemaVersion` | Numeric schema version of the source message                                       |
| `__properties`    | Producer-supplied key/value properties (Pulsar message properties / Kafka headers) |

The column is stored as an Iceberg / Delta **Variant** value. This keeps the schema stable when properties evolve (new keys, removed keys) and lets downstream engines extract individual fields with standard Variant accessors (for example, `__meta.__publishTime` in Spark SQL).

## Configuration

| Property               | Default | Description                                                                                           |
| ---------------------- | ------- | ----------------------------------------------------------------------------------------------------- |
| `persistExtraMetadata` | `false` | Master switch. When `true`, the `__meta` column is added to the table and populated for every record. |

### Required Companion Settings

Because `__meta` is a Variant column, the feature shares the [Variant type](/cloud/lakehouse/features/variant-type) prerequisites:

| Property                   | Default | Required When                                                               |
| -------------------------- | ------- | --------------------------------------------------------------------------- |
| `variantTypeEnabled`       | `false` | Always (master switch for Variant support)                                  |
| `tableEvolveSchemaEnabled` | `true`  | Always (the column is added by schema evolution)                            |
| `allowIcebergV3`           | `false` | Required for Iceberg (Variant is an Iceberg V3 feature). Ignored for Delta. |

> **Important:** Variant support is gated by a feature flag. Contact the StreamNative Support Team to enable it before turning on `persistExtraMetadata`.

### Iceberg

Add the following to the compaction service `custom` config:

```yaml theme={null}
persistExtraMetadata: "true"
variantTypeEnabled: "true"
tableEvolveSchemaEnabled: "true"   # default; only override if previously disabled
allowIcebergV3: "true"
```

> **Downstream query engine compatibility:** When `allowIcebergV3` is enabled, your readers (Spark, Trino, Athena, etc.) must support Iceberg V3 to read tables that contain Variant columns. See [Variant Type](/cloud/lakehouse/features/variant-type#iceberg) for details.

### Delta Lake

```yaml theme={null}
persistExtraMetadata: "true"
variantTypeEnabled: "true"
tableEvolveSchemaEnabled: "true"
```

Delta does not require `allowIcebergV3`.

## Querying the Metadata

Once enabled, the `__meta` column appears in the lakehouse table alongside the user-defined fields. Examples:

**Spark SQL (Iceberg):**

```sql theme={null}
SELECT __meta:__publishTime AS publish_time,
       __meta:__messageOffset AS offset,
       __meta:__properties:trace_id AS trace_id,
       *
FROM iceberg_catalog.namespace.events
WHERE __meta:__publishTime >= 1700000000000
LIMIT 10;
```

**Spark SQL (Delta):**

```sql theme={null}
SELECT variant_get(__meta, '$.__publishTime', 'long') AS publish_time,
       variant_get(__meta, '$.__messageOffset', 'string') AS offset,
       *
FROM delta.`s3://bucket/path/events`
LIMIT 10;
```

The exact Variant accessor syntax depends on your engine version; consult the engine's Variant documentation for the canonical form.

## Behavior Notes

* **Adding metadata to existing tables.** Because the column is added through schema evolution (`tableEvolveSchemaEnabled=true`), enabling the flag on a topic that already has a lakehouse table appends the `__meta` column on the next compaction. Records written before the flag was enabled will have a `null` value in the column.
* **Disabling the feature.** Setting `persistExtraMetadata` back to `false` stops new records from receiving metadata, but the column itself is not removed. Older rows retain their values.
* **Performance impact.** The `__meta` column is small per-row but increases storage and write throughput slightly. The Variant encoding is efficient and supports predicate pushdown when the engine extracts a specific field.

## Related

* [Variant Type](/cloud/lakehouse/features/variant-type) -- Prerequisite feature for `persistExtraMetadata`
* [Schema Evolution](/cloud/lakehouse/features/schema-evolution) -- The mechanism by which the `__meta` column is added to the table
* [Persist Key](/cloud/lakehouse/features/persist-key) -- Companion feature that persists the message key as a separate column


# Persist Key
Source: https://docs.streamnative.io/cloud/lakehouse/features/persist-key



The **Persist Key** feature injects a `__key` column into the lakehouse table that contains the original message key from the source topic. Use it when downstream queries need access to the partition key produced by the application -- for example, joining the lakehouse table with another dataset on the producer's key, computing per-key aggregations, or auditing message routing.

## What Gets Persisted

When the feature is enabled, every record written to the lakehouse table includes a `__key` column populated with the source message's key:

| Source          | Key value persisted                                                                           |
| --------------- | --------------------------------------------------------------------------------------------- |
| Pulsar producer | The Pulsar message key (`MessageBuilder.key(...)`), or `null` if the producer did not set one |
| Kafka producer  | The Kafka record key bytes, or `null` if the producer did not set one                         |

The column is added to the table by schema evolution and stored as a `binary` value, preserving the exact bytes the producer sent.

## Configuration

| Property     | Default | Description                                                                                          |
| ------------ | ------- | ---------------------------------------------------------------------------------------------------- |
| `persistKey` | `false` | Master switch. When `true`, the `__key` column is added to the table and populated for every record. |

### Required Companion Settings

The `__key` column is added through schema evolution, so schema evolution must remain enabled:

| Property                   | Default | Description                                                                               |
| -------------------------- | ------- | ----------------------------------------------------------------------------------------- |
| `tableEvolveSchemaEnabled` | `true`  | Required (the column is added by schema evolution). Only override if previously disabled. |

`persistKey` does **not** require Variant support, Iceberg V3, or any other feature flag.

### Configuration

Add the following to the compaction service `custom` config:

```yaml theme={null}
persistKey: "true"
tableEvolveSchemaEnabled: "true"   # default; only override if previously disabled
```

The same configuration applies to both Iceberg and Delta Lake.

## Querying the Key

Once enabled, the `__key` column appears in the lakehouse table alongside the user-defined fields. Examples:

**Spark SQL:**

```sql theme={null}
SELECT CAST(__key AS STRING) AS key,
       *
FROM iceberg_catalog.namespace.events
WHERE CAST(__key AS STRING) = 'user-42'
LIMIT 10;
```

```sql theme={null}
-- Per-key counts
SELECT CAST(__key AS STRING) AS key, COUNT(*) AS message_count
FROM iceberg_catalog.namespace.events
GROUP BY CAST(__key AS STRING)
ORDER BY message_count DESC;
```

If your producer keys are UTF-8 strings, cast the column to `STRING` for readability. For binary keys, query the column directly as `BINARY`.

## Behavior Notes

* **Adding the key column to existing tables.** Because the column is added through schema evolution (`tableEvolveSchemaEnabled=true`), enabling the flag on a topic that already has a lakehouse table appends the `__key` column on the next compaction. Records written before the flag was enabled will have a `null` value in the column.
* **Disabling the feature.** Setting `persistKey` back to `false` stops new records from receiving the key, but the column itself is not removed. Older rows retain their values.
* **Null keys.** Messages without a key are written with `null` in the `__key` column.
* **Combining with upsert.** The `__key` column is independent of [identifier fields](/cloud/lakehouse/features/iceberg-upsert) used for upsert. It records the producer-side key as-is and is not used for deduplication.

## Related

* [Persist Extra Metadata](/cloud/lakehouse/features/persist-extra-metadata) -- Companion feature that persists message envelope metadata (offset, publish time, properties) as a Variant column
* [Schema Evolution](/cloud/lakehouse/features/schema-evolution) -- The mechanism by which the `__key` column is added to the table
* [Upsert](/cloud/lakehouse/features/iceberg-upsert) -- For deduplication semantics on a primary key, which is a different concern from persisting the source message key


# Schema Evolution
Source: https://docs.streamnative.io/cloud/lakehouse/features/schema-evolution



Schema evolution is enabled by default (`tableEvolveSchemaEnabled=true`). When a producer changes the schema of a topic, the lakehouse table is automatically updated to match.

> **Recommendation:** For topics that have Lakehouse integration enabled, set the Pulsar schema compatibility strategy to **`BACKWARD_TRANSITIVE`**. This guarantees that every reader (including downstream Iceberg / Delta consumers) can read data written with all previous schema versions, which aligns with how the lakehouse table is evolved. See the [Pulsar schema compatibility documentation](https://pulsar.apache.org/docs/2.10.x/schema-evolution-compatibility/#backward-and-backward_transitive) for details on `BACKWARD` vs `BACKWARD_TRANSITIVE`.
>
> Apply with `pulsar-admin` at the namespace or topic level:
>
> ```bash theme={null}
> # Namespace level
> bin/pulsar-admin namespaces set-schema-compatibility-strategy \
>   --compatibility BACKWARD_TRANSITIVE \
>   <tenant>/<namespace>
>
> # Topic level
> bin/pulsar-admin topicPolicies set-schema-compatibility-strategy \
>   --strategy BACKWARD_TRANSITIVE \
>   <tenant>/<namespace>/<topic-name>
> ```

## How It Works

Schema evolution is triggered automatically during the compaction process when the compaction service encounters messages with a schema version newer than what the lakehouse table currently has.

### Step 1: Detection

When processing messages, the compaction service extracts the schema version from each message. If the version is newer than the latest version recorded in the table's schema mapping, evolution is triggered.

### Step 2: Retrieve All Schema Versions

The compaction service retrieves all schema versions from the schema registry (Pulsar or Confluent) up to the current version. This ensures that intermediate schema versions are not skipped.

### Step 3: Convert Schemas

Each source schema (Avro, JSON, ProtobufNative(Pulsar) or Protobuf(Kafka)) is converted to the target table format:

* **Avro** is converted directly to Iceberg Schema or Delta StructType
* **JSON** is first converted to Avro, then to the target format (Pulsar internally represents JSON schemas using Avro)
* **ProtobufNative(Pulsar)** is converted via the Protobuf descriptor to Avro, then to the target format
* **Protobuf(Kafka)** is converted via the Protobuf descriptor to Avro, then to the target format

### Step 4: Apply Changes

Schema changes are applied to the table iteratively, version by version in ascending order. Each version is processed as a transaction.

For **Iceberg** tables, the following operations are supported:

* **Add columns** -- new fields are always added as optional for backward compatibility
* **Delete columns** -- field is removed from the schema (or made optional in soft-delete mode)
* **Type promotion** -- `int` to `long`, `float` to `double`, and other compatible promotions
* **Nullability changes** -- required fields can be made optional (reverse is not supported)
* **Nested struct evolution** -- the same rules apply recursively to nested fields

For **Delta** tables:

* Column Mapping mode (`NAME`) is automatically enabled to support safe renames and deletes
* Each column receives a unique physical ID for safe schema evolution
* Schema changes are committed as Delta transactions

### Step 5: Record Schema Mapping

After each successful evolution, a mapping from the source schema version to the table's internal schema ID is recorded as a table property (`streamnative.schema.mapping`). This prevents re-processing the same version.

## Supported Operations

| Operation                                                            | Iceberg       | Delta         |
| -------------------------------------------------------------------- | ------------- | ------------- |
| Add optional columns                                                 | Supported     | Supported     |
| Delete columns                                                       | Supported     | Supported     |
| Type promotion (`int` -> `long`, `float` -> `double`)                | Supported     | Supported     |
| Required -> Optional                                                 | Supported     | Supported     |
| Optional -> Required                                                 | Not supported | Not supported |
| Nested struct evolution                                              | Supported     | Supported     |
| Type category changes (`struct` \<-> `list`, `primitive` \<-> `map`) | Not supported | Not supported |

## Limitations

* **Adding required fields:** New fields added to existing tables are automatically converted to optional. Required fields can only be defined at table creation time.
* **Type category changes:** Changing a field from one type category to another (e.g., struct to list, primitive to map) is not supported.
* **Backward incompatibility:** If the table schema has been evolved beyond the message's schema version, the message cannot be processed. Update the producer to use the latest schema.
* **Soft-delete mode:** Enable `schema.evolution.soft-delete.enabled=true` to make deleted fields optional instead of removing them entirely. This preserves backward compatibility for readers.

## Configuration

| Property                               | Description                                      | Default |
| -------------------------------------- | ------------------------------------------------ | ------- |
| `tableEvolveSchemaEnabled`             | Enable automatic schema evolution                | `true`  |
| `schema.evolution.soft-delete.enabled` | Make deleted fields optional instead of removing | `false` |
| `dlt.suffix`                           | Suffix appended to Dead Letter Table name        | `_dlt`  |

## Dead Letter Table (DLT)

When a message cannot be successfully processed into the lakehouse table, it is routed to a **Dead Letter Table (DLT)** instead of failing the entire compaction job. This allows the pipeline to continue processing valid messages while preserving failed records for investigation.

### When Are Messages Routed to DLT?

Messages are routed to DLT in the following scenarios:

| Scenario                       | Example                                                                                                                                                    |
| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Schema incompatibility**     | 1) Schema evolution is disabled and the record schema does not match the table schema; 2) New topic schema is incompatible with the Lakehouse table schema |
| **Unsupported schema changes** | Evolving a non-Variant field to Variant, or a Variant field to another type                                                                                |
| **Type conversion errors**     | Record cannot be serialized to the target lakehouse format (Iceberg/Delta/Parquet)                                                                         |
| **Null values**                | Null payload/record values from Kafka entries                                                                                                              |
| **Deserialization errors**     | Failures deserializing Pulsar or Kafka source data                                                                                                         |
| **Parsing failures**           | Schema parsing errors for Pulsar message format                                                                                                            |

> **Note:** Fatal errors (such as catalog connectivity issues, permission errors, or infrastructure failures) are **not** routed to DLT -- they cause the compaction task to fail and retry.

### DLT Table Naming

The DLT table name is derived from the original topic name by appending the configurable DLT suffix (default: `_dlt`).

**Format:**

```
<original-topic-local-name><dlt-suffix>
```

**Example:**

| Original Topic                                    | DLT Table                                                 |
| ------------------------------------------------- | --------------------------------------------------------- |
| `persistent://my-tenant/my-namespace/user-events` | `user-events_dlt` (in namespace `my-tenant/my-namespace`) |
| `persistent://public/default/orders`              | `orders_dlt` (in namespace `public/default`)              |

For Iceberg tables, the DLT table is created in the same namespace as the original table. For Delta tables, the DLT topic is `<tenant>/<namespace>/<topic-name>_dlt`.

### DLT Schema

Both Iceberg and Delta DLT tables use a consistent 3-field schema:

| Field           | Type   | Nullable | Description                                        |
| --------------- | ------ | -------- | -------------------------------------------------- |
| `messageId`     | String | No       | The original Pulsar or Kafka message ID            |
| `payload`       | String | Yes      | Base64-encoded original message payload            |
| `failureReason` | String | Yes      | The error message explaining why the record failed |

This allows you to:

* Identify which messages failed and why
* Replay or reprocess the failed messages after fixing the underlying issue
* Debug schema or serialization problems

### Configuring the DLT Suffix

The default DLT suffix is `_dlt`. To customize it, set the `dlt.suffix` property in the compaction service configuration:

```yaml theme={null}
compactionScheduler:
  config:
    custom:
      dlt.suffix: "_deadletter"
```

With this configuration, the DLT table for topic `user-events` would be `user-events-deadletter`.

### Monitoring DLT Activity

When the compaction writer closes, it logs a summary of DLT activity including:

* Total number of records routed to DLT
* Breakdown by failure reason (up to 100 distinct reasons tracked)
* Up to 10 sample message IDs per failure reason

Example log output:

```
WARN Sent 42 record(s) to DLT for topic: persistent://public/default/events, partition: 0,
     failure reasons with messageIds: {"Schema evolution is disabled...": ["msgId1", "msgId2", ...]}
```

Use this information to diagnose schema mismatches, invalid data, or other data quality issues upstream.


# Variant Type
Source: https://docs.streamnative.io/cloud/lakehouse/features/variant-type



The Variant type allows a single column to hold values of different data types, enabling flexible handling of semi-structured data without defining a rigid schema upfront.

StreamNative Ursa supports the Variant type for both **Apache Iceberg (V3)** and **Delta Lake** tables.

> **Important:** Variant type support is disabled by default. Contact the StreamNative Support Team to enable the feature flag before using Variant types.

## Enabling Variant Support

Variant support is gated by a small set of broker properties. The required combination depends on the target table format.

| Property                   | Default | Description                                                                                                                     |
| -------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `variantTypeEnabled`       | `false` | Master switch for Variant support. Required for both Iceberg and Delta.                                                         |
| `tableEvolveSchemaEnabled` | `true`  | Schema evolution must remain enabled so Variant fields can be added/removed during writes. Required for both Iceberg and Delta. |
| `allowIcebergV3`           | `false` | Enables Iceberg V3 features (including Variant). **Required for Iceberg**, ignored for Delta.                                   |

### Iceberg

Set the following properties on the compaction service `custom` config:

```yaml theme={null}
variantTypeEnabled: "true"
tableEvolveSchemaEnabled: "true"   # default; only override if previously disabled
allowIcebergV3: "true"
```

> **Downstream query engine compatibility:** When `allowIcebergV3` is enabled, the downstream query engine reading the table must also support Iceberg V3. Older Spark / Trino / Athena versions that only support Iceberg V2 will fail to read tables that use Variant or other V3-only features. Verify your engine's Iceberg support level before enabling.

### Delta Lake

Delta Lake's Variant type is not gated by an Iceberg version flag. Only the master switch and schema evolution are required:

```yaml theme={null}
variantTypeEnabled: "true"
tableEvolveSchemaEnabled: "true"   # default; only override if previously disabled
```

## Supported Data Types

* **Primitives:** `string`, `int`, `long`, `float`, `double`, `boolean`, `bytes`
* **Complex types:** `map`, `list / array`, `set`
* **Nested POJOs** and **entire POJOs**

## Configure Variant in Pulsar

### Avro Schema

Use the `@AvroSchema` annotation with `logicalType: "variant"`:

```java theme={null}
@Data
public class Event {
    private String name;

    // Variant for primitive type
    @AvroSchema("{\"type\": \"string\", \"logicalType\": \"variant\"}")
    private String flexibleField;

    // Variant for Map
    @AvroSchema("{\"type\": \"map\", \"values\": \"string\", \"logicalType\": \"variant\"}")
    private Map<String, String> metadata;

    // Variant for List
    @AvroSchema("{\"type\": \"array\", \"items\": \"string\", \"logicalType\": \"variant\"}")
    private List<String> tags;

    // Variant for nested POJO with metadata fields for query optimization
    @AvroSchema("{\"type\": \"record\", \"name\": \"Address\", \"fields\": ["
            + "  {\"name\": \"city\", \"type\": \"string\"},"
            + "  {\"name\": \"zip\", \"type\": \"int\"}"
            + "],"
            + "\"logicalType\": \"variant\","
            + "\"variant-metadata-fields\": \"[\\\"zip\\\", \\\"city\\\"]\" "
            + "}")
    private Address address;
}
```

### JSON Schema

Use `@JsonPropertyDescription` with the Variant annotation:

```java theme={null}
@Data
public class Event {
    private String name;

    @JsonPropertyDescription("logicalType: variant")
    private String flexibleField;

    @JsonPropertyDescription("logicalType: variant")
    private Map<String, String> metadata;

    @JsonPropertyDescription("logicalType: variant")
    private List<String> tags;

    @JsonPropertyDescription("logicalType: variant")
    private Address address;
}
```

### ProtobufNative Schema

Define a custom field option named `logical_type`:

```protobuf theme={null}
syntax = "proto3";

import "google/protobuf/descriptor.proto";

extend google.protobuf.FieldOptions {
  string logical_type = 1001;
}

message Event {
  string name = 1;
  string flexible_field = 2 [(logical_type) = "variant"];
  map<string, string> metadata = 3 [(logical_type) = "variant"];
  repeated string tags = 4 [(logical_type) = "variant"];
  Address address = 5 [(logical_type) = "variant"];
}

message Address {
  string city = 1;
  int32 zip = 2;
}
```

## Configure Variant in Ursa (Kafka Protocol)

### Avro Schema -- POJO Annotation

Same `@AvroSchema` annotations as Pulsar. Produce data using `ReflectionAvroSerializer` (do not use `KafkaAvroSerializer`).

### Avro Schema -- Inline Definition

Define `logicalType: "variant"` directly in the Avro schema:

```json theme={null}
{
  "type": "record",
  "name": "Event",
  "fields": [
    {
      "name": "id",
      "type": ["null", "string"],
      "default": null
    },
    {
      "name": "score",
      "type": {
        "type": "double",
        "logicalType": "variant"
      }
    },
    {
      "name": "tags",
      "type": {
        "type": "array",
        "items": "string",
        "logicalType": "variant"
      },
      "default": []
    },
    {
      "name": "attributes",
      "type": {
        "type": "map",
        "values": "string",
        "logicalType": "variant"
      },
      "default": {}
    },
    {
      "name": "address",
      "type": {
        "type": "record",
        "name": "Address",
        "fields": [
          {"name": "street", "type": "string"},
          {"name": "city", "type": "string"}
        ],
        "logicalType": "variant",
        "variant-metadata-fields": "[\"street\", \"city\"]"
      }
    }
  ]
}
```

### JSON Schema

Same `@JsonPropertyDescription("logicalType: variant")` annotations as Pulsar.

### Protobuf Schema

Protobuf is supported via the same `logical_type` custom field option as Pulsar's ProtobufNative schema. Define the option once in your `.proto` file and tag each Variant field:

```protobuf theme={null}
syntax = "proto3";

import "google/protobuf/descriptor.proto";

extend google.protobuf.FieldOptions {
  string logical_type = 1001;
}

message Event {
  string name = 1;
  string flexible_field = 2 [(logical_type) = "variant"];
  map<string, string> metadata = 3 [(logical_type) = "variant"];
  repeated string tags = 4 [(logical_type) = "variant"];
  Address address = 5 [(logical_type) = "variant"];
}

message Address {
  string city = 1;
  int32 zip = 2;
}
```

## Performance Optimization

Use `variant-metadata-fields` to specify fields that should be extracted as top-level columns. This accelerates query performance by enabling predicate pushdown on those fields:

```json theme={null}
"variant-metadata-fields": "[\"zip\", \"city\"]"
```

## Schema Evolution Rules for Variant

| Operation                                | Supported                               |
| ---------------------------------------- | --------------------------------------- |
| Adding new Variant fields                | Yes                                     |
| Removing existing Variant fields         | Yes                                     |
| Converting non-Variant field to Variant  | No (messages sent to Dead Letter Table) |
| Converting Variant field to another type | No (messages sent to Dead Letter Table) |

For more information about the Dead Letter Table (DLT), see [Schema Evolution -- Dead Letter Table](/cloud/lakehouse/features/schema-evolution#dead-letter-table-dlt).


# Overview - Internal Tables
Source: https://docs.streamnative.io/cloud/lakehouse/internal-tables/internal-tables-overview



**Internal Tables (Coming Soon)**

Internal Tables are topics inside StreamNative Cloud that store data in Ursa Format, the native, high-performance storage layer used by StreamNative’s Cost Optimized Clusters. This capability is currently not available and will be released in an upcoming version of StreamNative Cloud.

**Key Characteristics**

* **Stored in Ursa Format** -
  Data is written using a compact, high-throughput format optimized for streaming and low-latency access.

* **Fully Managed by StreamNative** -
  StreamNative will handle all storage lifecycle operations — retention, compaction, indexing, and metadata.

* **StreamNative-Native Performance** -
  Designed for fast ingestion, low latency, and operational efficiency within the StreamNative cloud environment.

* **Not Externally Queryable via Open Table Formats** -
  Internal Tables are optimized for streaming workloads, not external analytics engines.

**When to Use Internal Tables**

* You need low-latency ingestion and high-throughput streaming.
* You prefer StreamNative to manage all storage operations end-to-end.
* You don’t require external interoperability with Iceberg/Delta or an external Lakehouse system.

Note: Internal Tables are under development and will be introduced soon. The feature description above outlines planned capabilities.


# Lakehouse Features Overview
Source: https://docs.streamnative.io/cloud/lakehouse/lakehouse-features



## Feature Matrix

| Feature          | Iceberg (SDT)  | Delta (SDT)  | Iceberg (SBT) | Delta (SBT) |
| ---------------- | -------------- | ------------ | ------------- | ----------- |
| Schema Evolution | Supported      | Supported    | Coming Soon   | Coming Soon |
| Partition Key    | Configurable   | Configurable | Coming Soon   | Coming Soon |
| Upsert           | Supported      | Supported    | Coming Soon   | Coming Soon |
| Variant Type     | Supported (V3) | Supported    | Coming Soon   | Coming Soon |
| Streaming Read   | --             | --           | Coming Soon   | Coming Soon |
| Analytical Query | Supported      | Supported    | Coming Soon   | Coming Soon |

## Supported Cloud Vendors

| Provider             | WAL Storage | Lakehouse Table |
| -------------------- | ----------- | --------------- |
| AWS S3               | Supported   | Supported       |
| Google Cloud Storage | Supported   | Supported       |
| Azure Blob Storage   | Supported   | Supported       |

## Supported Schemas

| Schema Format           | Supported |
| ----------------------- | --------- |
| Avro                    | Yes       |
| JSON                    | Yes       |
| ProtobufNative (Pulsar) | Yes       |
| Protobuf (Kafka)        | Yes       |

Supported schema registries:

* StreamNative Kafka Schema Registry
* Confluent Schema Registry

## Feature Guides

Each feature has its own detailed guide:

* [Schema Evolution](/cloud/lakehouse/features/schema-evolution) -- How the lakehouse table evolves when the topic schema changes, including Dead Letter Table (DLT) behavior
* [Variant Type](/cloud/lakehouse/features/variant-type) -- Handle semi-structured data with Iceberg / Delta Variant type
* [Partition Key](/cloud/lakehouse/features/iceberg-partition-key) -- Configure table partitioning with Iceberg transforms
* [Upsert](/cloud/lakehouse/features/iceberg-upsert) -- Deduplication and primary key-based updates
* [Persist Extra Metadata](/cloud/lakehouse/features/persist-extra-metadata) -- Persist message-envelope metadata (offset, publish time, properties) as a `__meta` Variant column
* [Persist Key](/cloud/lakehouse/features/persist-key) -- Persist the source message key as a `__key` column

## Next Steps

* [Observability](/cloud/lakehouse/lakehouse-observability) -- Monitor lakehouse tables with metrics and Grafana dashboards


# Lakehouse Table Overview
Source: https://docs.streamnative.io/cloud/lakehouse/lakehouse-table-overview



Lakehouse Table is a zero-ETL integration that automatically converts streaming data from Apache Pulsar topics into open table formats -- Apache Iceberg and Delta Lake -- stored directly on object storage (AWS S3, GCS, Azure Blob Storage). This enables unified streaming and analytics access to the same data without building or maintaining separate data pipelines.

## Architecture

```
                          ┌──────────────────┐
                          │   Pulsar Broker  │
                          └────────┬─────────┘
                                   │
                   ┌───────────────┴────────────────┐
                   │  WAL Storage                   │
                   │  ┌──────────────────────────┐  │
                   │  │ Latency-Optimized:       │  │
                   │  │   Apache BookKeeper      │  │
                   │  ├──────────────────────────┤  │
                   │  │ Cost-Optimized:          │  │
                   │  │   Object Storage         │  │
                   │  │   (S3 / GCS / Azure)     │  │
                   │  └──────────────────────────┘  │
                   └───────────────┬────────────────┘
                                   │ Reads from both
                                   ▼
                    ┌─────────────────────────────┐
                    │    Compaction Service       │
                    │  (WAL → Parquet conversion) │
                    └────────────┬────────────────┘
                                 │ Commit
                                 ▼
                    ┌─────────────────────────────┐
                    │    Lakehouse Table          │
                    │  (Iceberg / Delta Lake)     │
                    └─────────────────────────────┘
                                 │
                    ┌────────────┴────────────┐
                    ▼                         ▼
             External Catalog          Query Engines
          (Unity Catalog, S3Table,   (Spark, Trino,
           BigLake, Snowflake)        DuckDB, Athena)
```

### WAL Storage Options

Lakehouse Table supports two WAL storage tiers:

* **Latency-optimized (Apache BookKeeper):** Low-latency writes for performance-sensitive workloads
* **Cost-optimized (Object Storage):** Direct writes to AWS S3, GCS, or Azure Blob Storage for cost efficiency

The **Compaction Service reads from both** BookKeeper and Object Storage, converts the data to Parquet format, and commits snapshots to the lakehouse catalog.

### Coordination

**Oxia** serves as the metadata store for coordination, leader election, schema storage, and offset index management.

The Compaction Service operates with a leader-worker architecture: the leader publishes compaction tasks and commits results to the lakehouse catalog, while workers perform the WAL-to-Parquet conversion.

## Table Modes

### External Table (SDT -- Stream Delivered Table)

An External Table delivers data from Pulsar topics into an external lakehouse catalog (such as Databricks Unity Catalog, Snowflake, AWS S3Table, or Google BigLake). A **separate copy** of the topic data is written to the Lakehouse table — the Pulsar topic and the Lakehouse table hold independent copies of the same records. In this mode:

* Data is written to Iceberg or Delta Lake tables managed by the external catalog as a separate copy from the Pulsar topic
* Analytical access via standard table APIs (Spark, Trino, DuckDB, Athena, etc.)
* Supports **upsert**, **partition key**, and **schema evolution**
* The external catalog governs data lifecycle (retention, deletion) for the Lakehouse copy independently of the Pulsar topic
* Streaming reads with offset semantics are **not** supported on the delivered data

**Use External Tables when:** you want to deliver streaming data into curated lakehouse tables for analytics, integrate with existing data platforms, or need upsert/deduplication capabilities.

### Internal Table (SBT -- Stream Backed Table)

> **Coming Soon** -- Internal Table support is under active development.

An Internal Table is managed entirely by Ursa Storage. The Pulsar topic and the Lakehouse table **share the same single copy of data** — there is no separate write to the Lakehouse table. The same physical data supports both streaming reads (with offset tracking and replay) and analytical queries -- true stream-table duality with zero data duplication.

## Supported Cluster Profiles and Protocols

StreamNative Private Cloud offers two cluster types (Pulsar and Kafka), each with two performance profiles (latency-optimized and cost-optimized). Lakehouse delivery support depends on the combination of cluster type, profile, and producer protocol.

| Cluster Type | Profile           | Producer Protocol | Lakehouse Delivery |
| ------------ | ----------------- | ----------------- | ------------------ |
| Pulsar       | Latency-optimized | Pulsar            | Supported          |
| Pulsar       | Latency-optimized | Kafka             | Coming Soon        |
| Pulsar       | Cost-optimized    | Kafka             | Supported          |
| Pulsar       | Cost-optimized    | Pulsar            | Not yet supported  |
| Kafka        | Cost-optimized    | Kafka             | Supported          |
| Kafka        | Latency-optimized | Kafka             | Coming Soon        |

Notes:

* A **Pulsar latency-optimized** cluster uses Apache BookKeeper as the WAL tier. Topic data produced via the Pulsar protocol can be delivered to Lakehouse today; Kafka-protocol delivery is on the roadmap.
* A **Pulsar cost-optimized** cluster uses object storage as the WAL tier. Topic data produced via the Kafka protocol is delivered to Lakehouse; the Pulsar protocol is not yet supported on this profile.
* A **Kafka cost-optimized** cluster delivers Kafka topic data to Lakehouse today.
* A **Kafka latency-optimized** cluster will support Lakehouse delivery in a future release.

## Supported Formats

| Format         | Status    |
| -------------- | --------- |
| Apache Iceberg | Supported |
| Delta Lake     | Supported |

## Supported Cloud Storage

| Provider             | WAL Storage | Lakehouse Table |
| -------------------- | ----------- | --------------- |
| AWS S3               | Supported   | Supported       |
| Google Cloud Storage | Supported   | Supported       |
| Azure Blob Storage   | Supported   | Supported       |

## Supported Catalogs

Catalog support varies by cloud provider:

| Catalog                                    | Table Format | AWS       | GCP       | Azure     |
| ------------------------------------------ | ------------ | --------- | --------- | --------- |
| Databricks Unity Catalog (Managed Iceberg) | Iceberg      | Supported | Supported | Supported |
| Databricks Unity Catalog (Delta Lake)      | Delta Lake   | Supported | Supported | Supported |
| Snowflake Horizon Catalog                  | Iceberg      | Supported | Supported | Supported |
| Snowflake Open Catalog (Polaris)           | Iceberg      | Supported | Supported | Supported |
| AWS S3Table                                | Iceberg      | Supported | --        | --        |
| Google BigLake                             | Iceberg      | --        | Supported | --        |

## Topic to lakehouse identifier mapping

Each Pulsar topic maps to exactly **one** Lakehouse table. The mapping is 1:1 regardless of how many partitions the topic has — data from all partitions of a partitioned topic is consolidated into a single Lakehouse table.

When data is delivered from a Pulsar topic to a lakehouse table, the topic's tenant, namespace, and topic name are mapped to a catalog **namespace** and a **table name**. The mapping rules differ by catalog type because Pulsar allows characters (`/`, `.`, `-`, `:`) that are not valid in many catalog identifiers.

The compaction service applies the following rules.

### Iceberg with hierarchical catalogs (Snowflake Open Catalog, Snowflake Horizon, Iceberg REST, Iceberg Hadoop)

The original Pulsar identifiers are used unchanged:

* Catalog namespace: `<tenant>.<namespace>` (two-level)
* Table name: the topic local name (the part after the namespace)

For example, the topic `persistent://my-tenant/my-namespace/orders` is mapped to namespace `my-tenant.my-namespace` and table `orders`.

### Iceberg with flat-namespace catalogs (AWS S3Tables, Google BigLake, Hive)

These catalogs only accept a single-level namespace, so the tenant and namespace are flattened into one identifier with a cluster-name prefix. Each component is escaped to remove invalid characters:

| Source character | Replacement               |
| ---------------- | ------------------------- |
| `/`              | `___` (three underscores) |
| `.`              | `_` (one underscore)      |
| `-`              | `__` (two underscores)    |
| `:`              | `____` (four underscores) |

* Catalog namespace: `<cluster>_<formatted-tenant>_<formatted-namespace>` (default cluster prefix is `pulsar`)
* Table name (S3Tables): the topic local name with the same character escapes applied
* Table name (BigLake, Hive): the topic local name as-is

For example, with the default cluster prefix `pulsar`, topic `persistent://public-v1/default.v2/test-table-v1`:

* On **AWS S3Tables**: namespace `pulsar_public__v1_default_v2`, table `test__table__v1`
* On **Google BigLake**: namespace `pulsar_public__v1_default_v2`, table `test-table-v1`

### Databricks Unity Catalog (Iceberg or Delta Lake)

Unity Catalog uses a three-level identifier (`catalog.schema.table`). The compaction service writes all topics into a single schema and encodes the full Pulsar topic path into the **table name**, so each catalog table maps 1:1 to a Pulsar topic. The full topic path `<tenant>/<namespace>/<topic>` is flattened with these escapes:

| Source character | Replacement               |
| ---------------- | ------------------------- |
| `/`              | `__` (two underscores)    |
| `.`              | `____` (four underscores) |
| `-`              | `___` (three underscores) |

For example, topic `persistent://public/default/test-topic` is mapped to table name `public__default__test___topic`. Topic `persistent://public/default/v1.events` is mapped to `public__default__v1____events`.

## Limitations

### Schema limitations

The following topic schema constructs are not supported when delivering data to a Lakehouse Table:

* **Recursive schemas:** Schemas that reference themselves -- directly or indirectly -- are not supported. For example, a record with a field whose type is the same record (such as a tree node that holds a list of child nodes of the same type). Iceberg and Delta Lake require a fixed, finite column structure, which cannot represent a self-referential schema. Topics that use a recursive schema cannot be delivered to a Lakehouse Table.

## Next Steps

* [Deploy Lakehouse Table](/cloud/lakehouse/deploy-lakehouse-table) -- Set up the infrastructure with `private-cloud.yaml`
* [Prepare Lakehouse Catalogs](/cloud/lakehouse/prepare-lakehouse-catalogs) -- Set up your external catalog service
* [Dynamic Configuration Guide](/cloud/lakehouse/dynamic-configuration) -- Cluster-name prefix, override priority, and the full set of dynamic configuration keys
* [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog) -- Connect catalogs to the compaction service
* [Enable Lakehouse Integration](/cloud/lakehouse/enable-lakehouse-integration) -- Enable at cluster, namespace, or topic level
* Features:
  * [Schema Evolution](/cloud/lakehouse/features/schema-evolution)
  * [Variant Type](/cloud/lakehouse/features/variant-type)
  * [Partition Key](/cloud/lakehouse/features/iceberg-partition-key)
  * [Upsert](/cloud/lakehouse/features/iceberg-upsert)
  * [Persist Key](/cloud/lakehouse/features/persist-key)
  * [Persist Extra Metadata](/cloud/lakehouse/features/persist-extra-metadata)
* [Observability](/cloud/lakehouse/lakehouse-observability) -- Metrics, alerts, and Grafana dashboard


# Overview - Lakehouse Tables
Source: https://docs.streamnative.io/cloud/lakehouse/lakehouse-tables-overview



**StreamNative Lakehouse Tables**

StreamNative Lakehouse Tables provide a unified way to expose streaming topics as open table format objects—such as Apache Iceberg and Delta Lake—directly within StreamNative Cloud. With Lakehouse Tables, data produced to Pulsar or Kafka-compatible topics can be automatically stored in object storage in a transactional, analytics-ready table format. This enables seamless integration between real-time data streams and downstream analytics, AI/ML, and governance systems.

**Overview**

StreamNative Lakehouse Tables bridge the gap between streaming and batch systems by converting message data from topics into table-backed datasets. Each Lakehouse Table maintains:

* Metadata (schema, manifest lists, snapshots)
* Data files (Parquet/columnar format)
* Transaction logs (for table evolution)

These components reside in user-controlled object storage, ensuring low cost, high durability, and interoperability with a broad ecosystem of tools.

**Key Capabilities**

**1. Open Table Format Support**

StreamNative Lakehouse Tables support industry-standard formats:

* Apache Iceberg
* Delta Lake (Delta 2.0 and above)

This ensures compatibility with engines such as Databricks, Snowflake, Spark, Trino, Flink, BigQuery, StarTree, and more.

**2. Native Topic-to-Table Mapping**

Each table is backed by one or more StreamNative topics. Data is automatically:

* Ingested from the streaming topic
* Serialized into Parquet files
* Committed into the table as immutable snapshots
* Made available for SQL queries and analytical engines

This provides a streaming-first lakehouse architecture without external ingestion pipelines.

**3. Object Storage as the Source of Truth**

All table artifacts are stored in object storage such as:

* Amazon S3
* Google Cloud Storage
* Azure Blob Storage

This enables low-cost storage, independent scaling, and easy interoperability.

**4. Schema-Aware and Schema-Safe**

Lakehouse Tables use StreamNative’s schema registry to ensure:

* Schema inference from topics
* Backward/forward compatible evolution
* Safe writes with schema enforcement
* Automatic mapping to Iceberg/Delta schemas
* Users retain full control over table evolution policies.

**5. Transactional Guarantees**

Using open table format guarantees, Lakehouse Tables support:

* ACID transactions
* Snapshot isolation
* Time travel (via historical snapshots)
* Incremental reads

This brings reliability and consistency to streaming data workflows.

6. Full Interoperability with Data and AI Platforms

Once a table is materialized in Iceberg or Delta Lake format, it is fully queryable by:

* Databricks
* Snowflake
* BigQuery Managed Tables
* Apache Spark, Flink, and Trino
* StarTree and Pinot
* DuckDB
* pandas & PyArrow

No connectors or intermediate ETL is required.


# BigLake for Iceberg
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/biglake/iceberg



This guide describes how to prepare a Google BigLake metastore for use with StreamNative Ursa as an Iceberg REST catalog.

For background, see the Google Cloud documentation: [Use the BigLake metastore Iceberg REST catalog](https://docs.cloud.google.com/biglake/docs/blms-rest-catalog).

## Prerequisites

* A GCP project with permissions to create BigLake catalogs and modify IAM roles
* A StreamNative Ursa cluster running on GCP

## 1. Create a BigLake Catalog

In the Google Cloud Console, search for **BigLake**.

<img alt="Search BigLake" />

Create a new catalog by selecting a Cloud Storage bucket.

> **Important:**
>
> * The bucket must be in the **same region** as your StreamNative Ursa cluster, otherwise cross-region traffic and latency are introduced.
> * BigLake does not support sub-directories within a bucket. Each BigLake catalog maps to exactly one bucket (1:1 mapping).

In the **Authentication** section, choose **Credential vending mode**.

<img alt="Credential vending mode" />

After the catalog is created, view the catalog details to obtain the **REST Catalog URI**, **GCS Warehouse**, and **Project**.

<img alt="Catalog information" />

Click **Set bucket permissions** to grant the BigLake service account access to the bucket.

<img alt="Set bucket permissions" />

## 2. Grant IAM Roles to the Ursa Broker Service Account

Locate the StreamNative Ursa broker service account (the format is typically `iamaccount-<id>@<gcp-project>.iam.gserviceaccount.com`).

<img alt="Broker service account" />

In the GCP IAM console, grant the broker service account the following roles:

* **BigLake Editor**
* **Storage Object User**
* **Service Usage Consumer**

<img alt="Grant IAM roles" />

<img alt="Grant IAM roles" />

## Catalog Information Summary

When the steps above are complete, collect the following values for the StreamNative Ursa compaction service. Several values are fixed for any BigLake catalog -- copy them as-is.

| Value                                        | Description                                                                                              |
| -------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| `iceberg.catalog-backend`                    | `BIGLAKE`                                                                                                |
| `iceberg.type`                               | `rest`                                                                                                   |
| `iceberg.uri`                                | The **REST Catalog URI** from step 1 (typically `https://biglake.googleapis.com/iceberg/v1/restcatalog`) |
| `iceberg.warehouse`                          | The **GCS Warehouse** from step 1 (e.g., `gs://<bucket-name>`)                                           |
| `iceberg.header.x-goog-user-project`         | The **Project** from step 1                                                                              |
| `iceberg.rest.auth.type`                     | `org.apache.iceberg.gcp.auth.GoogleAuthManager` (fixed)                                                  |
| `iceberg.io-impl`                            | `org.apache.iceberg.gcp.gcs.GCSFileIO` (fixed)                                                           |
| `iceberg.rest-metrics-reporting-enabled`     | `false` (fixed -- BigLake does not yet support REST metrics reporting)                                   |
| `iceberg.header.X-Iceberg-Access-Delegation` | `vended-credentials` (fixed)                                                                             |

For the next steps, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog).


# Horizon Catalog for Iceberg on AWS
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/horizon-catalog/iceberg/aws



This guide describes how to prepare a Snowflake Horizon Catalog for use with StreamNative Ursa as an Iceberg catalog on AWS.

For background, see the Snowflake documentation: [Write to Apache Iceberg tables using an external query engine through Snowflake Horizon Catalog](https://docs.snowflake.com/en/LIMITEDACCESS/iceberg/tables-iceberg-write-using-external-write-engine-snowflake-horizon).

## Prerequisites

* A Snowflake account with Horizon Catalog enabled
* An AWS account with permissions to create S3 buckets and IAM roles
* The `ACCOUNTADMIN` role in Snowflake (required for several SQL operations below)

## 1. Create an External Volume

The Horizon Catalog uses a Snowflake **External Volume** to access object storage. Reference: [Tutorial: Create your first Apache Iceberg table](https://docs.snowflake.com/en/user-guide/tutorials/create-your-first-iceberg-table#create-a-table).

### 1.1 Identify the Snowflake Account Region

Find the region of your Snowflake account so the S3 bucket can be created in the same region.

<img alt="Snowflake region" />

### 1.2 Create an S3 Bucket

Create an S3 bucket in the AWS console, in the same region as your Snowflake account.

<img alt="Create S3 bucket" />

### 1.3 Create an IAM Policy

Create an IAM policy granting access to the bucket. Replace `<your-bucket>` with the bucket name from step 1.2.

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:DeleteObject",
        "s3:DeleteObjectVersion"
      ],
      "Resource": "arn:aws:s3:::<your-bucket>/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": "arn:aws:s3:::<your-bucket>",
      "Condition": {
        "StringLike": {
          "s3:prefix": ["*"]
        }
      }
    }
  ]
}
```

<img alt="IAM policy" />

### 1.4 Create an IAM Role

Create an IAM role with an External ID (for example, `iceberg_table_external_id`). The trust policy will be updated in step 1.6.

<img alt="Create IAM role" />

Attach the policy from step 1.3 to the role.

<img alt="Bind policy to role" />

### 1.5 Create the External Volume in Snowflake

Switch to the `ACCOUNTADMIN` role and run the following SQL, substituting your values:

```sql theme={null}
CREATE OR REPLACE EXTERNAL VOLUME <volume_name>
   STORAGE_LOCATIONS = (
      (
         NAME = '<location-name>'
         STORAGE_PROVIDER = 'S3'
         STORAGE_BASE_URL = 's3://<your-bucket>/'
         STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::<your-account-id>:role/<your-role-name>'
         STORAGE_AWS_EXTERNAL_ID = '<your-external-id>'
      )
   )
   ALLOW_WRITES = TRUE;
```

<img alt="External volume" />

If the command fails with a permission error, ensure you are using the `ACCOUNTADMIN` role:

<img alt="Permission issue" />

<img alt="Switch to ACCOUNTADMIN" />

### 1.6 Configure the Trust Relationship

After the external volume is created, retrieve the Snowflake-generated IAM user ARN:

```sql theme={null}
DESC EXTERNAL VOLUME <volume_name>;
```

The `STORAGE_AWS_IAM_USER_ARN` field contains the value (for example, `arn:aws:iam::<snowflake-account>:user/<id>`).

<img alt="Volume info" />

Return to the AWS IAM console and update the trust policy of the role created in step 1.4 to allow `STORAGE_AWS_IAM_USER_ARN` to assume the role.

<img alt="Update trust policy" />

<img alt="Update trust policy" />

## 2. Configure Access Control

> **Note:** If you already have roles configured with access to the Iceberg tables you want to use, you can skip this section.

For details, see [Configuring access control](https://docs.snowflake.com/en/user-guide/security-access-control-configure).

```sql theme={null}
GRANT ROLE ACCOUNTADMIN, SYSADMIN TO USER <username>;
```

<img alt="Grant roles" />

## 3. Create the Catalog Database

Switch to `ACCOUNTADMIN` and create the catalog database in the Snowflake UI. Then bind the database to the external volume:

```sql theme={null}
ALTER DATABASE <db_name> SET CATALOG = 'SNOWFLAKE';
ALTER DATABASE <db_name> SET EXTERNAL_VOLUME = '<volume_name>';
```

<img alt="Create catalog" />

## 4. Obtain an Access Token

Snowflake Horizon supports three authentication methods: External OAuth, Key-pair authentication, and Programmatic Access Token (PAT). This guide uses **PAT**.

### 4.1 Create an Authentication Policy

Switch to `ACCOUNTADMIN` and run:

```sql theme={null}
USE <db_name>;

CREATE AUTHENTICATION POLICY <policy_name>
  PAT_POLICY=(NETWORK_POLICY_EVALUATION = ENFORCED_NOT_REQUIRED);

ALTER ACCOUNT SET AUTHENTICATION POLICY <policy_name>;

ALTER AUTHENTICATION POLICY <policy_name>
  SET AUTHENTICATION_METHODS = ('OAUTH', 'PASSWORD', 'PROGRAMMATIC_ACCESS_TOKEN');

ALTER AUTHENTICATION POLICY <policy_name>
  SET PAT_POLICY = (MAX_EXPIRY_IN_DAYS=365, DEFAULT_EXPIRY_IN_DAYS=365);

ALTER USER IF EXISTS <username> ADD PROGRAMMATIC ACCESS TOKEN <token_name>
  DAYS_TO_EXPIRY = 365
  COMMENT = 'PAT for StreamNative Ursa';
```

Record the generated PAT.

<img alt="Generate PAT" />

### 4.2 Verify the Endpoint

Generate an OAuth access token using the PAT and verify the Horizon REST endpoint:

```bash theme={null}
curl -i --fail -X POST \
  "https://<account_identifier>.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens" \
  --header 'Content-Type: application/x-www-form-urlencoded' \
  --data-urlencode 'grant_type=client_credentials' \
  --data-urlencode 'scope=session:role:<role>' \
  --data-urlencode 'client_secret=<PAT>'

curl -i --fail -X GET \
  "https://<account_identifier>.snowflakecomputing.com/polaris/api/catalog/v1/config?warehouse=<database_name>" \
  -H "Authorization: Bearer <access_token>" \
  -H "Content-Type: application/json"
```

### 4.3 Grant Permissions

```sql theme={null}
GRANT ROLE PUBLIC TO USER <username>;
GRANT USAGE ON DATABASE <db_name> TO ROLE PUBLIC;
GRANT USAGE ON EXTERNAL VOLUME <volume_name> TO ROLE PUBLIC;
```

## Catalog Information Summary

When the steps above are complete, collect the following values for the StreamNative Ursa compaction service:

```sql theme={null}
-- Determine the URI prefix
SELECT CURRENT_ORGANIZATION_NAME() || '-' || CURRENT_ACCOUNT_NAME();
```

| Value                | Description                                                                   |
| -------------------- | ----------------------------------------------------------------------------- |
| `iceberg.uri`        | `https://<organization>-<account>.snowflakecomputing.com/polaris/api/catalog` |
| `iceberg.warehouse`  | The catalog database name created in step 3                                   |
| `iceberg.credential` | The PAT generated in step 4.1                                                 |
| `iceberg.scope`      | `session:role:<role>` (e.g., `session:role:PUBLIC`)                           |

For the next steps, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog).


# Open Catalog for Iceberg on AWS
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/open-catalog/iceberg/aws



This guide describes how to prepare a Snowflake Open Catalog (Polaris) for use with StreamNative Ursa as an Iceberg catalog on AWS.

> **Important:** Polaris does not support reading buckets from a different region. The StreamNative Ursa cluster, the storage bucket, and the Polaris catalog must all reside in the **same AWS region**.

## Prerequisites

* A Snowflake standard account
* An AWS account with permissions to create S3 buckets and IAM roles
* Access to the Snowflake Open Catalog feature (request via your Snowflake account team if not yet enabled)

## 1. Create a Snowflake Open Catalog Account

The Snowflake Open Catalog console requires a dedicated **Open Catalog** account. From the standard Snowflake console, navigate to **Admin -> Accounts** and use the toggle to **Create Snowflake Open Catalog Account**.

<img alt="Snowflake console" />

<img alt="Create Open Catalog account" />

Configure the account with:

* **Cloud:** AWS
* **Region:** the region in which your S3 bucket resides (for example, `US East (Ohio)`)
* **Edition:** any

<img alt="Account configuration" />

Provide an admin username and password.

<img alt="Account credentials" />

After creation, click the **Account URL** to sign in to the Open Catalog console.

<img alt="Account created" />

<img alt="Open Catalog console" />

## 2. Create an S3 Bucket

Create an S3 bucket in the same region as the Open Catalog account.

<img alt="Create bucket" />

## 3. Create an IAM Policy

Navigate to **AWS IAM -> Policies -> Create policy**.

<img alt="Create policy" />

Paste the following policy, replacing the bucket name and subpath with your values:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:DeleteObject",
        "s3:DeleteObjectVersion"
      ],
      "Resource": "arn:aws:s3:::<your-bucket>/<your-subpath>/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": "arn:aws:s3:::<your-bucket>/<your-subpath>",
      "Condition": {
        "StringLike": {
          "s3:prefix": ["*"]
        }
      }
    }
  ]
}
```

<img alt="Policy JSON" />

<img alt="Policy next step" />

## 4. Create an IAM Role

Navigate to **AWS IAM -> Roles -> Create role** and configure:

* **Trusted entity type:** AWS account
* **An AWS account:** This account
* **Enable External ID** with a unique value (you will reference this when creating the Polaris catalog)

<img alt="Create role" />

<img alt="Trust settings" />

Attach the policy created in step 4.

<img alt="Attach policy" />

Provide a role name and create the role.

<img alt="Save role" />

Record the role ARN (for example, `arn:aws:iam::<account-id>:role/<role-name>`).

<img alt="Role ARN" />

## 5. Create the Polaris Catalog

In the Snowflake Open Catalog console, create a new catalog.

<img alt="Create catalog" />

Configure the catalog with:

* **External:** disabled
* **Storage provider:** S3
* **Default base location:** `s3://<your-bucket>/<your-subpath>` (the path from step 3)
* **S3 role ARN:** the role ARN recorded in step 5
* **External ID:** the External ID configured in step 5

<img alt="Catalog configuration" />

<img alt="Catalog created" />

Open the catalog details and record the **IAM user ARN** that Polaris uses to access AWS. You will use this in step 7 to update the trust policy of the IAM role.

<img alt="Catalog IAM user ARN" />

## 6. Update the IAM Role Trust Policy

Return to the AWS IAM console, open the role created in step 5, and edit the trust relationship.

<img alt="Find role" />

<img alt="Edit trust policy" />

Update `Principal.AWS` to the Polaris IAM user ARN recorded in step 6.

<img alt="Update trust policy" />

Click **Update policy**.

## 7. Create a Connection (Service Principal)

In the Open Catalog console, create a new connection that StreamNative Ursa will use to authenticate.

<img alt="Create connection" />

Configure with:

* **Name:** any name
* **Create new principal role:** enabled
* **Principal Role Name:** any name

<img alt="Connection configuration" />

After creation, record the **Client ID** and **Client Secret** -- the secret cannot be retrieved later.

<img alt="Connection credentials" />

## 8. Create a Catalog Role and Grant Privileges

Navigate to **Catalogs -> \[your catalog] -> Roles -> + Catalog Role** and create a role with the following privileges:

* `NAMESPACE_CREATE`
* `NAMESPACE_LIST`
* `NAMESPACE_READ_PROPERTIES`
* `NAMESPACE_WRITE_PROPERTIES`
* `TABLE_LIST`
* `TABLE_CREATE`
* `TABLE_WRITE_DATA`
* `TABLE_READ_DATA`
* `TABLE_READ_PROPERTIES`
* `TABLE_WRITE_PROPERTIES`

<img alt="Create catalog role" />

Click **Grant to Principals Role** and grant the catalog role to the principal role created in step 8.

<img alt="Grant to principal role" />

<img alt="Grant configuration" />

<img alt="Role bindings" />

For background on the relationship between catalogs, catalog roles, principal roles, and principals, see the [Polaris Quick Start](https://polaris.io/#section/Quick-Start/Defining-a-Catalog).

## Catalog Information Summary

When the steps above are complete, collect the following values for the StreamNative Ursa compaction service:

| Value                | Description                                                                                                                                                    |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `iceberg.uri`        | Polaris REST endpoint (e.g., `https://<account>.<region>.aws.snowflakecomputing.com/polaris/api/catalog`). The format follows the URL of your Polaris console. |
| `iceberg.warehouse`  | The Polaris catalog name created in step 6                                                                                                                     |
| `iceberg.credential` | `<client-id>:<client-secret>` from step 8                                                                                                                      |
| `iceberg.scope`      | `PRINCIPAL_ROLE:ALL`                                                                                                                                           |

## Table Maintenance

Snowflake Open Catalog (Polaris) and the Hadoop catalog do **not** run table maintenance on your behalf. Streaming writes from the StreamNative Ursa compaction service produce many small Parquet files and accumulate snapshot history over time, which degrades query performance and inflates storage costs. You are responsible for scheduling and running maintenance against every Iceberg table written by Ursa.

Run the maintenance operations below on a regular schedule. They are provided as [Apache Iceberg Spark stored procedures](https://iceberg.apache.org/docs/latest/spark-procedures/) and can be triggered from any Spark cluster (Databricks, AWS EMR, AWS Glue, GCP Dataproc, or self-managed Spark) that has the Iceberg Spark runtime, catalog credentials, and IAM access to the warehouse bucket.

**Maintenance operations**

| Operation             | Purpose                                                                                                                               | Suggested cadence                                                                         |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| `rewrite_data_files`  | Compact small Parquet files into fewer, larger files. Reduces file-listing overhead and improves scan performance.                    | Hourly to daily, depending on ingestion rate                                              |
| `expire_snapshots`    | Drop snapshots older than the retention window so their data and manifest files can be cleaned up.                                    | Daily; retain at least 1–7 days so in-flight readers and time-travel queries keep working |
| `remove_orphan_files` | Delete files in the table location that are no longer referenced by any snapshot (typically left behind by failed or partial writes). | Weekly                                                                                    |
| `rewrite_manifests`   | Rewrite manifest files so they align with the current partition layout. Improves query planning time.                                 | Weekly, or after large schema or partition changes                                        |

**Example: run maintenance from Spark**

The following examples assume the catalog has been registered in Spark as `<catalog>`. Replace `<catalog>`, `<namespace>`, and `<table>` with your values.

```sql theme={null}
-- Compact small files. Iceberg targets files smaller than the default 512 MB.
CALL <catalog>.system.rewrite_data_files(table => '<namespace>.<table>');

-- Expire snapshots older than 3 days; keep the 5 most recent snapshots.
CALL <catalog>.system.expire_snapshots(
  table       => '<namespace>.<table>',
  older_than  => TIMESTAMP '2026-05-20 00:00:00',
  retain_last => 5
);

-- Remove orphan files older than 7 days.
CALL <catalog>.system.remove_orphan_files(
  table      => '<namespace>.<table>',
  older_than => TIMESTAMP '2026-05-20 00:00:00'
);

-- Rewrite manifests to match the current partition layout.
CALL <catalog>.system.rewrite_manifests(table => '<namespace>.<table>');
```

**Operational guidance**

* **Credentials.** The principal that runs maintenance must have catalog privileges to read and write the target table (for example, the same `TABLE_READ_DATA`, `TABLE_WRITE_DATA`, `TABLE_READ_PROPERTIES`, and `TABLE_WRITE_PROPERTIES` privileges configured for the Ursa compaction service) and IAM access to the warehouse bucket so it can read and rewrite the underlying data files. With the Hadoop catalog there is no catalog service to authenticate against — only the bucket IAM access is required.
* **Concurrency.** Iceberg uses optimistic concurrency control. If maintenance commits race with the Ursa compaction writer, one of them retries. Schedule heavy operations (`rewrite_data_files`, `rewrite_manifests`) during low-write windows when possible.
* **Retention vs. time travel.** `expire_snapshots` and `remove_orphan_files` permanently delete files. Choose a retention window that exceeds the longest expected read query and your time-travel SLA.
* **Schedule the workload.** Most teams orchestrate these procedures from Databricks Jobs, AWS EMR steps, Airflow, Dagster, or a Kubernetes `CronJob`. Pick a scheduler that fits your existing operational stack.
* **Reference.** See the [Iceberg Spark procedures](https://iceberg.apache.org/docs/latest/spark-procedures/#metadata-management) documentation for the full parameter list, including options for partial rewrites (`where`), file-size targets, and merge-on-read delete file compaction.

For the next steps, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog).


# Open Catalog for Iceberg on Azure
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/open-catalog/iceberg/azure



This guide describes how to prepare a Snowflake Open Catalog (Polaris) for use with StreamNative Ursa as an Iceberg catalog on Microsoft Azure.

## Prerequisites

* A Snowflake standard account
* An Azure subscription with permissions to create storage accounts and configure trusted apps
* Access to the Snowflake Open Catalog feature

## 1. Create a Snowflake Open Catalog Account

The Snowflake Open Catalog console requires a dedicated **Open Catalog** account. From the standard Snowflake console, navigate to **Admin -> Accounts** and use the toggle to **Create Snowflake Open Catalog Account**.

<img alt="Snowflake console" />

<img alt="Create Open Catalog account" />

Configure the account with:

* **Cloud:** AWS or Azure (per Polaris availability)
* **Region:** the region in which your storage container resides
* **Edition:** any

<img alt="Account configuration" />

Provide an admin username and password.

<img alt="Account credentials" />

After creation, click the **Account URL** to sign in to the Open Catalog console.

<img alt="Account created" />

<img alt="Open Catalog console" />

## 2. Collect Azure Account Information

### 2.1 Azure Tenant ID

In the Azure portal, search for **Tenant properties** and record the **Tenant ID**.

<img alt="Search Tenant properties" />

<img alt="Tenant ID" />

### 2.2 Storage Service Endpoint

In the Azure portal, navigate to **Storage accounts**, open the target storage account, and click **Settings -> Endpoints**. Record the **Blob service** primary endpoint (for example, `https://<account>.blob.core.windows.net/`).

<img alt="Search Storage accounts" />

<img alt="Storage endpoint" />

### 2.3 Create a Container

In the storage account, navigate to **Data storage -> Containers -> + Container** and create a new container.

<img alt="Create container" />

## 3. Create the Polaris Catalog

In the Snowflake Open Catalog console, create a new catalog.

<img alt="Create catalog" />

> **Important:** The StreamNative compaction service writes data using the AzureDFS protocol, so the Polaris catalog must use the same protocol. Use `abfss://<container>@<account>.dfs.core.windows.net` (note `dfs`, not `blob`).

Configure the catalog with:

* **External:** disabled
* **Storage provider:** AZURE
* **Default base location:** `abfss://<container>@<storage-account>.dfs.core.windows.net`
* **Azure tenant ID:** the value from step 2.1

<img alt="Catalog configuration" />

## 4. Create a Trusted App in Azure

Open the catalog details and record the values of `AZURE_CONSENT_URL` and `AZURE_MULTI_TENANT_APP_NAME`.

<img alt="Catalog Azure values" />

Open the `AZURE_CONSENT_URL` in a browser and click **Accept**. This redirects to Snowflake and creates a trusted app in Azure. The trusted app name is the portion of `AZURE_MULTI_TENANT_APP_NAME` before the underscore.

> **Note:** Provisioning the trusted app in Azure can take several minutes.

## 5. Grant Container Permissions to the Trusted App

In the Azure portal, navigate to the storage account's **Access Control (IAM) -> + Add -> Add role assignment**.

<img alt="Add role assignment" />

Search for **Storage Blob Data** and select **Storage Blob Data Contributor**.

<img alt="Select role" />

Click **Select members**, search for the trusted app name from step 4, select it, and click **Review + assign**.

<img alt="Select members" />

<img alt="Role assigned" />

## 6. Create a Connection (Service Principal)

In the Open Catalog console, create a new connection that StreamNative Ursa will use to authenticate.

<img alt="Create connection" />

Configure with:

* **Name:** any name
* **Create new principal role:** enabled
* **Principal Role Name:** any name

<img alt="Connection configuration" />

After creation, record the **Client ID** and **Client Secret** -- the secret cannot be retrieved later.

<img alt="Connection credentials" />

## 7. Create a Catalog Role and Grant Privileges

Navigate to **Catalogs -> \[your catalog] -> Roles -> + Catalog Role** and create a role with the following privileges:

* `NAMESPACE_CREATE`
* `NAMESPACE_READ_PROPERTIES`
* `TABLE_CREATE`
* `TABLE_WRITE_DATA`
* `TABLE_READ_DATA`

<img alt="Create catalog role" />

Click **Grant to Principals Role** and grant the catalog role to the principal role created in step 6.

<img alt="Grant to principal role" />

<img alt="Grant configuration" />

## Catalog Information Summary

When the steps above are complete, collect the following values for the StreamNative Ursa compaction service:

| Value                | Description                                                                                                                                                |
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `iceberg.uri`        | Polaris REST endpoint (e.g., `https://<account>.<region>.snowflakecomputing.com/polaris/api/catalog`). The format follows the URL of your Polaris console. |
| `iceberg.warehouse`  | The Polaris catalog name created in step 3                                                                                                                 |
| `iceberg.credential` | `<client-id>:<client-secret>` from step 6                                                                                                                  |
| `iceberg.scope`      | `PRINCIPAL_ROLE:ALL`                                                                                                                                       |

## Table Maintenance

Snowflake Open Catalog (Polaris) and the Hadoop catalog do **not** run table maintenance on your behalf. Streaming writes from the StreamNative Ursa compaction service produce many small Parquet files and accumulate snapshot history over time, which degrades query performance and inflates storage costs. You are responsible for scheduling and running maintenance against every Iceberg table written by Ursa.

Run the maintenance operations below on a regular schedule. They are provided as [Apache Iceberg Spark stored procedures](https://iceberg.apache.org/docs/latest/spark-procedures/) and can be triggered from any Spark cluster (Databricks, AWS EMR, AWS Glue, GCP Dataproc, or self-managed Spark) that has the Iceberg Spark runtime, catalog credentials, and IAM access to the warehouse bucket.

**Maintenance operations**

| Operation             | Purpose                                                                                                                               | Suggested cadence                                                                         |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| `rewrite_data_files`  | Compact small Parquet files into fewer, larger files. Reduces file-listing overhead and improves scan performance.                    | Hourly to daily, depending on ingestion rate                                              |
| `expire_snapshots`    | Drop snapshots older than the retention window so their data and manifest files can be cleaned up.                                    | Daily; retain at least 1–7 days so in-flight readers and time-travel queries keep working |
| `remove_orphan_files` | Delete files in the table location that are no longer referenced by any snapshot (typically left behind by failed or partial writes). | Weekly                                                                                    |
| `rewrite_manifests`   | Rewrite manifest files so they align with the current partition layout. Improves query planning time.                                 | Weekly, or after large schema or partition changes                                        |

**Example: run maintenance from Spark**

The following examples assume the catalog has been registered in Spark as `<catalog>`. Replace `<catalog>`, `<namespace>`, and `<table>` with your values.

```sql theme={null}
-- Compact small files. Iceberg targets files smaller than the default 512 MB.
CALL <catalog>.system.rewrite_data_files(table => '<namespace>.<table>');

-- Expire snapshots older than 3 days; keep the 5 most recent snapshots.
CALL <catalog>.system.expire_snapshots(
  table       => '<namespace>.<table>',
  older_than  => TIMESTAMP '2026-05-20 00:00:00',
  retain_last => 5
);

-- Remove orphan files older than 7 days.
CALL <catalog>.system.remove_orphan_files(
  table      => '<namespace>.<table>',
  older_than => TIMESTAMP '2026-05-20 00:00:00'
);

-- Rewrite manifests to match the current partition layout.
CALL <catalog>.system.rewrite_manifests(table => '<namespace>.<table>');
```

**Operational guidance**

* **Credentials.** The principal that runs maintenance must have catalog privileges to read and write the target table (for example, the same `TABLE_READ_DATA`, `TABLE_WRITE_DATA`, `TABLE_READ_PROPERTIES`, and `TABLE_WRITE_PROPERTIES` privileges configured for the Ursa compaction service) and IAM access to the warehouse bucket so it can read and rewrite the underlying data files. With the Hadoop catalog there is no catalog service to authenticate against — only the bucket IAM access is required.
* **Concurrency.** Iceberg uses optimistic concurrency control. If maintenance commits race with the Ursa compaction writer, one of them retries. Schedule heavy operations (`rewrite_data_files`, `rewrite_manifests`) during low-write windows when possible.
* **Retention vs. time travel.** `expire_snapshots` and `remove_orphan_files` permanently delete files. Choose a retention window that exceeds the longest expected read query and your time-travel SLA.
* **Schedule the workload.** Most teams orchestrate these procedures from Databricks Jobs, AWS EMR steps, Airflow, Dagster, or a Kubernetes `CronJob`. Pick a scheduler that fits your existing operational stack.
* **Reference.** See the [Iceberg Spark procedures](https://iceberg.apache.org/docs/latest/spark-procedures/#metadata-management) documentation for the full parameter list, including options for partial rewrites (`where`), file-size targets, and merge-on-read delete file compaction.

For the next steps, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog).


# Open Catalog for Iceberg on GCP
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/open-catalog/iceberg/gcp



This guide describes how to prepare a Snowflake Open Catalog (Polaris) for use with StreamNative Ursa as an Iceberg catalog on Google Cloud Platform (GCP).

> **Important:** Polaris does not support reading buckets from a different region. The StreamNative Ursa cluster, the GCS bucket, and the Polaris catalog must all reside in the **same region**.

## Prerequisites

* A Snowflake standard account
* A GCP project with permissions to create GCS buckets and IAM roles
* Access to the Snowflake Open Catalog feature

## 1. Create a Snowflake Open Catalog Account

The Snowflake Open Catalog console requires a dedicated **Open Catalog** account. From the standard Snowflake console, navigate to **Admin -> Accounts** and use the toggle to **Create Snowflake Open Catalog Account**.

<img alt="Snowflake console" />

<img alt="Create Open Catalog account" />

Configure the account with:

* **Cloud:** GCP
* **Region:** the region in which your GCS bucket resides
* **Edition:** any

<img alt="Account configuration" />

Provide an admin username and password.

<img alt="Account credentials" />

After creation, click the **Account URL** to sign in to the Open Catalog console.

<img alt="Account created" />

<img alt="Open Catalog console" />

## 2. Create the Polaris Catalog

In the Snowflake Open Catalog console, create a new catalog.

<img alt="Create catalog" />

Configure the catalog with:

* **External:** disabled
* **Storage provider:** GCS
* **Default base location:** the GCS path used by the Ursa cluster (`gs://<bucket>/<subpath>`)

<img alt="Catalog configuration" />

<img alt="Catalog created" />

Open the catalog details and record the **GCP\_SERVICE\_ACCOUNT** value. Polaris uses this service account to access GCS, so it must be granted permission on the bucket.

<img alt="Catalog GCP service account" />

## 3. Grant Bucket Permissions to the Polaris Service Account

### 3.1 Create a Custom IAM Role

In the GCP console, navigate to **IAM & Admin -> Roles -> Create role** and add the following permissions:

* `storage.buckets.get`
* `storage.objects.create`
* `storage.objects.delete`
* `storage.objects.get`
* `storage.objects.list`

<img alt="Create role" />

<img alt="Role setup" />

<img alt="Permissions" />

<img alt="Permissions selected" />

### 3.2 Assign the Role to the Polaris Service Account

Open the bucket, navigate to **PERMISSIONS -> View BY PRINCIPALS -> GRANT ACCESS**.

<img alt="Grant bucket access" />

Add the **GCP\_SERVICE\_ACCOUNT** from step 3, choose the role created in step 4.1, and click **SAVE**.

<img alt="Save access" />

## 4. Create a Connection (Service Principal)

In the Open Catalog console, create a new connection that StreamNative Ursa will use to authenticate.

<img alt="Create connection" />

Configure with:

* **Name:** any name
* **Create new principal role:** enabled
* **Principal Role Name:** any name

<img alt="Connection configuration" />

After creation, record the **Client ID** and **Client Secret** -- the secret cannot be retrieved later.

<img alt="Connection credentials" />

## 5. Create a Catalog Role and Grant Privileges

Navigate to **Catalogs -> \[your catalog] -> Roles -> + Catalog Role** and create a role with the following privileges:

* `NAMESPACE_CREATE`
* `NAMESPACE_LIST`
* `NAMESPACE_READ_PROPERTIES`
* `NAMESPACE_WRITE_PROPERTIES`
* `TABLE_LIST`
* `TABLE_CREATE`
* `TABLE_WRITE_DATA`
* `TABLE_READ_DATA`
* `TABLE_READ_PROPERTIES`
* `TABLE_WRITE_PROPERTIES`

<img alt="Create catalog role" />

Click **Grant to Principals Role** and grant the catalog role to the principal role created in step 5.

<img alt="Grant to principal role" />

<img alt="Grant configuration" />

<img alt="Role bindings" />

For background on the relationship between catalogs, catalog roles, principal roles, and principals, see the [Polaris Quick Start](https://polaris.io/#section/Quick-Start/Defining-a-Catalog).

## Catalog Information Summary

When the steps above are complete, collect the following values for the StreamNative Ursa compaction service:

| Value                | Description                                                                                                                                                    |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `iceberg.uri`        | Polaris REST endpoint (e.g., `https://<account>.<region>.gcp.snowflakecomputing.com/polaris/api/catalog`). The format follows the URL of your Polaris console. |
| `iceberg.warehouse`  | The Polaris catalog name created in step 3                                                                                                                     |
| `iceberg.credential` | `<client-id>:<client-secret>` from step 5                                                                                                                      |
| `iceberg.scope`      | `PRINCIPAL_ROLE:ALL`                                                                                                                                           |

## Table Maintenance

Snowflake Open Catalog (Polaris) and the Hadoop catalog do **not** run table maintenance on your behalf. Streaming writes from the StreamNative Ursa compaction service produce many small Parquet files and accumulate snapshot history over time, which degrades query performance and inflates storage costs. You are responsible for scheduling and running maintenance against every Iceberg table written by Ursa.

Run the maintenance operations below on a regular schedule. They are provided as [Apache Iceberg Spark stored procedures](https://iceberg.apache.org/docs/latest/spark-procedures/) and can be triggered from any Spark cluster (Databricks, AWS EMR, AWS Glue, GCP Dataproc, or self-managed Spark) that has the Iceberg Spark runtime, catalog credentials, and IAM access to the warehouse bucket.

**Maintenance operations**

| Operation             | Purpose                                                                                                                               | Suggested cadence                                                                         |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| `rewrite_data_files`  | Compact small Parquet files into fewer, larger files. Reduces file-listing overhead and improves scan performance.                    | Hourly to daily, depending on ingestion rate                                              |
| `expire_snapshots`    | Drop snapshots older than the retention window so their data and manifest files can be cleaned up.                                    | Daily; retain at least 1–7 days so in-flight readers and time-travel queries keep working |
| `remove_orphan_files` | Delete files in the table location that are no longer referenced by any snapshot (typically left behind by failed or partial writes). | Weekly                                                                                    |
| `rewrite_manifests`   | Rewrite manifest files so they align with the current partition layout. Improves query planning time.                                 | Weekly, or after large schema or partition changes                                        |

**Example: run maintenance from Spark**

The following examples assume the catalog has been registered in Spark as `<catalog>`. Replace `<catalog>`, `<namespace>`, and `<table>` with your values.

```sql theme={null}
-- Compact small files. Iceberg targets files smaller than the default 512 MB.
CALL <catalog>.system.rewrite_data_files(table => '<namespace>.<table>');

-- Expire snapshots older than 3 days; keep the 5 most recent snapshots.
CALL <catalog>.system.expire_snapshots(
  table       => '<namespace>.<table>',
  older_than  => TIMESTAMP '2026-05-20 00:00:00',
  retain_last => 5
);

-- Remove orphan files older than 7 days.
CALL <catalog>.system.remove_orphan_files(
  table      => '<namespace>.<table>',
  older_than => TIMESTAMP '2026-05-20 00:00:00'
);

-- Rewrite manifests to match the current partition layout.
CALL <catalog>.system.rewrite_manifests(table => '<namespace>.<table>');
```

**Operational guidance**

* **Credentials.** The principal that runs maintenance must have catalog privileges to read and write the target table (for example, the same `TABLE_READ_DATA`, `TABLE_WRITE_DATA`, `TABLE_READ_PROPERTIES`, and `TABLE_WRITE_PROPERTIES` privileges configured for the Ursa compaction service) and IAM access to the warehouse bucket so it can read and rewrite the underlying data files. With the Hadoop catalog there is no catalog service to authenticate against — only the bucket IAM access is required.
* **Concurrency.** Iceberg uses optimistic concurrency control. If maintenance commits race with the Ursa compaction writer, one of them retries. Schedule heavy operations (`rewrite_data_files`, `rewrite_manifests`) during low-write windows when possible.
* **Retention vs. time travel.** `expire_snapshots` and `remove_orphan_files` permanently delete files. Choose a retention window that exceeds the longest expected read query and your time-travel SLA.
* **Schedule the workload.** Most teams orchestrate these procedures from Databricks Jobs, AWS EMR steps, Airflow, Dagster, or a Kubernetes `CronJob`. Pick a scheduler that fits your existing operational stack.
* **Reference.** See the [Iceberg Spark procedures](https://iceberg.apache.org/docs/latest/spark-procedures/#metadata-management) documentation for the full parameter list, including options for partial rewrites (`where`), file-size targets, and merge-on-read delete file compaction.

For the next steps, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog).


# S3Table for Iceberg
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/s3table/iceberg



This guide describes how to prepare an AWS S3Table catalog for use with StreamNative Ursa as an Iceberg catalog.

> **Important:** The StreamNative Ursa cluster must run in the **same region** as the S3Table bucket. Cross-region access is not supported and will fail with an AWS client error.

## Prerequisites

* An AWS account with permissions to create S3Table buckets and modify IAM roles
* A StreamNative Ursa cluster (the cluster's IAM role will be granted access to the S3Table bucket)

## 1. Create an S3Table Bucket

In the AWS S3 console, create an **S3Table bucket** in the same region as the Ursa cluster. Record the bucket ARN, which has the form:

```
arn:aws:s3tables:<region>:<account-id>:bucket/<bucket-name>
```

## 2. Grant S3Table Permissions via a Table Bucket Policy

Apply a [table bucket policy](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-bucket-policy.html) on the S3Table bucket so that the StreamNative Ursa cluster's IAM role can read and write tables in the bucket.

### 2.1 Find the StreamNative cluster role ARN

The role ARN to grant access to is the IAM role bound to the broker pods of the StreamNative cluster. When you enable Lakehouse Table for **Amazon S3 Tables** at the cluster, namespace, or topic level in the StreamNative Cloud Console, the dialog displays the exact IAM role ARN that needs access to your S3Table bucket. Copy that ARN.

For details on enabling S3 Tables, see [Enable Lakehouse Table](/cloud/lakehouse/enable-lakehouse-integration).

### 2.2 Apply the table bucket policy

In the AWS S3 console, open your S3Table bucket and go to the **Permissions** tab. Choose **Edit** under **Table bucket policy** and paste the following JSON, replacing `<streamnative-cluster-role-arn>` with the ARN you copied in the previous step and `<your-region>`, `<your-aws-account>`, and `<your-bucket-name>` with the values for your bucket.

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3ListTableBucket",
      "Effect": "Allow",
      "Principal": {
        "AWS": "<streamnative-cluster-role-arn>"
      },
      "Action": [
        "s3tables:ListTableBuckets"
      ],
      "Resource": ["*"]
    },
    {
      "Sid": "DataAccessPermissionsForS3TableBucket",
      "Effect": "Allow",
      "Principal": {
        "AWS": "<streamnative-cluster-role-arn>"
      },
      "Action": [
        "s3tables:GetTableBucket",
        "s3tables:CreateNamespace",
        "s3tables:GetNamespace",
        "s3tables:ListNamespaces",
        "s3tables:CreateTable",
        "s3tables:GetTable",
        "s3tables:ListTables",
        "s3tables:UpdateTableMetadataLocation",
        "s3tables:GetTableMetadataLocation",
        "s3tables:GetTableData",
        "s3tables:PutTableData"
      ],
      "Resource": [
        "arn:aws:s3tables:<your-region>:<your-aws-account>:bucket/<your-bucket-name>",
        "arn:aws:s3tables:<your-region>:<your-aws-account>:bucket/<your-bucket-name>/*"
      ]
    }
  ]
}
```

The policy panel is on the **Permissions** tab of the table bucket, as shown below:

<img alt="Set the S3 Table bucket policy" />

<img alt="Configuration example" />

<img alt="Configuration example" />

<img alt="IAM permissions example" />

## 3. Adjust S3Table Maintenance Settings (Recommended)

By default, S3 Tables retains snapshots for 120 hours (5 days). For production workloads, StreamNative recommends shortening this window to **6 hours**.

| Setting              | Default            | Recommended |
| -------------------- | ------------------ | ----------- |
| `MaximumSnapshotAge` | 120 hours (5 days) | 6 hours     |

**Why this matters.** S3 Tables enforces a fixed 5 MB cap on the Iceberg metadata file size. Long retention windows cause many unexpired snapshots to accumulate in the metadata file. When the metadata file grows beyond 5 MB, **subsequent snapshot commits will fail**, which stops new data from being written into the lakehouse table. A shorter `MaximumSnapshotAge` keeps the metadata file under the cap.

You can update the snapshot management settings from the S3 Table bucket console or via the AWS CLI after the bucket is created. See [S3 Tables maintenance -- Snapshot management](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html#s3-tables-maintenance-snapshot) for the full list of maintenance settings.

## 4. (Optional) Configure AWS Athena Access

If you intend to query the S3Table data via AWS Athena, you must integrate it with AWS Lake Formation.

### 4.1 Prerequisites

Refer to [S3 Tables integration prerequisites](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-aws.html#table-integration-prerequisites). In summary:

* Attach `AWSLakeFormationDataAdmin` to your IAM principal
* Add `glue:PassConnection` and `lakeformation:RegisterResource` permissions
* Use the latest version of the AWS CLI

When the S3Table bucket is created with AWS analytics services enabled, AWS automatically provisions a role (for example, `S3TablesRoleForLakeFormation_1`).

<img alt="S3Tables role" />

Add the following inline policy to your IAM principal to grant Lake Formation operations:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AWSLakeFormationDataAdminAllow",
      "Effect": "Allow",
      "Action": [
        "lakeformation:*",
        "cloudtrail:DescribeTrails",
        "cloudtrail:LookupEvents",
        "glue:CreateCatalog", "glue:UpdateCatalog", "glue:DeleteCatalog",
        "glue:GetCatalog", "glue:GetCatalogs",
        "glue:GetDatabase", "glue:GetDatabases",
        "glue:CreateDatabase", "glue:UpdateDatabase", "glue:DeleteDatabase",
        "glue:GetConnections", "glue:SearchTables",
        "glue:GetTable", "glue:CreateTable", "glue:UpdateTable", "glue:DeleteTable",
        "glue:GetTableVersions", "glue:GetPartitions", "glue:GetTables",
        "glue:ListWorkflows", "glue:BatchGetWorkflows", "glue:DeleteWorkflow",
        "glue:GetWorkflowRuns", "glue:StartWorkflowRun", "glue:GetWorkflow",
        "glue:PassConnection",
        "s3:ListBucket", "s3:GetBucketLocation", "s3:ListAllMyBuckets", "s3:GetBucketAcl",
        "iam:ListUsers", "iam:ListRoles", "iam:GetRole", "iam:GetRolePolicy"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AWSLakeFormationDataAdminDeny",
      "Effect": "Deny",
      "Action": ["lakeformation:PutDataLakeSettings"],
      "Resource": "*"
    }
  ]
}
```

### 4.2 Create a Resource Link

A resource link provides a Glue Data Catalog reference to your S3Table namespace. See [Creating a resource link](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-aws.html#database-link-tables).

```bash theme={null}
aws glue create-database --region <region> --catalog-id "<account-id>" --database-input '{
  "Name": "<resource-link-name>",
  "TargetDatabase": {
    "CatalogId": "<account-id>:s3tablescatalog/<table-bucket-name>",
    "DatabaseName": "<namespace>"
  },
  "CreateTableDefaultPermissions": []
}'
```

### 4.3 Grant Lake Formation Permissions

#### On the Table

In the AWS Lake Formation console, navigate to **Data permissions -> Grant** and configure:

1. **Principals**: the IAM user, role, or SAML group that will run queries.
2. **LF-Tags or catalog resources**: choose **Named Data Catalog resources**.
3. **Catalogs**: the Glue Data Catalog created when the table bucket was integrated (`<account-id>:s3tablescatalog/<table-bucket-name>`).
4. **Databases**: the S3Table namespace.
5. **Tables**: the S3Table table.
6. **Table permissions**: **Super**.

<img alt="Grant table permission" />

<img alt="Grant table permission" />

#### On the Resource Link

Resource links require their own grants in addition to the underlying namespace/table grants:

1. **Principals**: the IAM user/role/group.
2. **Catalogs**: your account's default catalog.
3. **Databases**: the resource link from step 4.2.
4. **Resource link permissions**: **Describe**.

<img alt="Grant resource link permission" />

<img alt="Athena query" />

## Catalog Information Summary

When the steps above are complete, collect the following values for the StreamNative Ursa compaction service:

| Value                         | Description                                                                                                                                                                                                             |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `iceberg.catalog-backend`     | `s3table`                                                                                                                                                                                                               |
| `iceberg.uri`                 | `https://s3tables.<region>.amazonaws.com/iceberg` (region-specific endpoint -- see the [S3 Tables regions list](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-regions-quotas.html#s3-tables-regions)) |
| `iceberg.warehouse`           | The S3Table bucket ARN: `arn:aws:s3tables:<region>:<account>:bucket/<bucket-name>`                                                                                                                                      |
| `iceberg.rest.signing-region` | The region of the S3Table bucket                                                                                                                                                                                        |

For the next steps, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog).


# Unity Catalog for Delta Lake on AWS
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/unity-catalog/delta-lake/aws



This guide describes how to prepare a Databricks Unity Catalog for use with StreamNative Ursa as a Delta Lake catalog on AWS.

## Prerequisites

* An AWS account with permissions to create S3 buckets and IAM roles
* A Databricks account with permissions to create workspaces

## 1. Create a Databricks Workspace

> Skip this step if you already have the Databricks Workspace

In the Databricks account console, create a new workspace. The workspace creation flow uses an AWS CloudFormation stack, so you must be logged into AWS in the same browser session.

<img alt="Workspace list" />

Click **Create workspace**.

<img alt="Create workspace" />

Choose **Quickstart**.

<img alt="Quickstart option" />

Enter a workspace name and select the AWS region in which your S3 bucket resides (for example, `us-east-2`). Click **Start Quickstart**.

<img alt="Workspace settings" />

In the AWS console, acknowledge the IAM resource creation and click **Create Stack**.

<img alt="Create CloudFormation stack" />

<img alt="Stack creating" />

When the stack reaches `CREATE_COMPLETE`, return to the Databricks console and open the workspace.

<img alt="Stack complete" />

<img alt="Workspace ready" />

<img alt="Unity Catalog console" />

## 2. (Recommend) Generate an OAuth2 Service Principal

If you prefer OAuth2 over a personal access token, create a service principal:

Navigate to **Developer -> Identity and access -> Service principals -> Manage**.

<img alt="Service principals menu" />

Click **Add service principal -> Add new**, give it a name, and click **Add**.

<img alt="Add service principal" />

Open the service principal, click **Secrets -> Generate secret**, choose an expiration period, and **Generate**.

<img alt="Generate secret" />

Record the **Client ID** and **Client Secret** -- the secret cannot be retrieved later.

<img alt="Generated credentials" />

## 3. (Alternative) Generate a User Token

A Databricks user token can be used by StreamNative Ursa to authenticate against Unity Catalog.

Open **User Settings**.

<img alt="User settings" />

Navigate to **Developer -> Access tokens -> Manage** and generate a new token. Record the token value -- it cannot be retrieved later.

<img alt="Developer settings" />

<img alt="Access tokens management" />

<img alt="Create token" />

## 4. Configure Unity Catalog Access

Navigate to **Catalog -> Settings -> Metastore**.

<img alt="Catalog settings" />

Enable **External data access** on the metastore.

<img alt="Enable external data access" />

Grant privileges on the catalog with the following settings:

* **Principal:** All accounts (or the specific user/service principal)
* **Privilege presets:** Data Editor (selects related privileges automatically)
* **EXTERNAL USE SCHEMA:** Enabled

<img alt="Grant privileges" />

<img alt="Privilege settings" />

If you use OAuth2 authentication, set the **Principal** to the service principal name created in step 3.

<img alt="OAuth2 privileges" />

## 5. Create an S3 Bucket

In your AWS account, create an S3 bucket for the Unity Catalog managed location (for example, `delta-unity-catalog-bucket`).

<img alt="S3 bucket" />

## 6. Create an IAM Policy

Navigate to **AWS IAM -> Policies -> Create policy**, choose JSON, and paste the following (replace `<your-bucket>`):

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:DeleteObject",
        "s3:DeleteObjectVersion"
      ],
      "Resource": "arn:aws:s3:::<your-bucket>/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": "arn:aws:s3:::<your-bucket>",
      "Condition": {
        "StringLike": {
          "s3:prefix": ["*"]
        }
      }
    }
  ]
}
```

<img alt="Create policy" />

<img alt="Policy JSON" />

<img alt="Save policy" />

## 7. Create an IAM Role

Navigate to **AWS IAM -> Roles -> Create role** and configure:

* **Trusted entity type:** AWS account
* **An AWS account:** This account
* **Enable External ID** with placeholder value `0000` (will be updated in step 9)

<img alt="Create role" />

<img alt="Trust settings" />

Attach the policy from step 6.

<img alt="Attach policy" />

<img alt="Save role" />

Record the role ARN (for example, `arn:aws:iam::<account-id>:role/<role-name>`).

<img alt="Role ARN" />

## 8. Create a Storage Credential in Unity Catalog

Navigate to **Catalog -> Settings -> Credentials**.

<img alt="Credentials menu" />

<img alt="Create credential" />

Configure with:

* **Credential:** Storage Credential
* **Type:** AWS IAM Role
* **Name:** any name
* **Role ARN:** the ARN recorded in step 7

<img alt="Credential form" />

Databricks generates a trust relationship policy. Copy it.

<img alt="Trust policy generated" />

## 9. Update the IAM Role Trust Policy

Return to the AWS IAM console, open the role created in step 7, and replace the trust policy with the one generated by Databricks.

<img alt="Update trust policy" />

Click **Validate** in the Unity Catalog console to verify the credential.

<img alt="Validate credential" />

## 10. Create an External Location

Navigate to **Catalog -> Settings -> External Locations**.

<img alt="External locations" />

<img alt="Create external location" />

Choose **Manual** (the AWS Quickstart creates a new bucket).

<img alt="Manual external location" />

Configure:

* **External location name:** any name
* **URL:** `s3://<your-bucket>`
* **Storage credential:** the credential from step 8

<img alt="External location form" />

After creation, click **Test connection** to verify access.

<img alt="Test external location" />

If you use OAuth2, grant **ALL PRIVILEGES** on the external location to the service principal:

<img alt="External location details" />

<img alt="Grant OAuth2 permissions" />

## 11. Create the Catalog

In Databricks, create a new catalog and bind it to the external location created in step 10.

<img alt="Create catalog" />

## Catalog Information Summary

When the steps above are complete, collect the following values for the StreamNative Ursa compaction service:

| Value                                               | Description                                                              |
| --------------------------------------------------- | ------------------------------------------------------------------------ |
| `unityCatalogUri`                                   | Databricks workspace URL (e.g., `https://dbc-xxxx.cloud.databricks.com`) |
| `unityCatalogName`                                  | The Unity Catalog name created in step 11                                |
| `unityCatalogToken`                                 | Personal access token from step 2, **or**                                |
| `unityCatalogClientId` / `unityCatalogClientSecret` | OAuth2 credentials from step 3                                           |

For the next steps, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog).


# Unity Catalog for Delta Lake on Azure
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/unity-catalog/delta-lake/azure



This guide describes how to prepare a Databricks Unity Catalog for use with StreamNative Ursa as a Delta Lake catalog on Microsoft Azure.

## Prerequisites

* An Azure subscription with permissions to create storage accounts and Access Connectors
* A Databricks workspace on Azure

## 1. Create an Access Connector for Azure Databricks

In the Azure Marketplace, search for **Access Connector for Azure Databricks** and click **Create**.

<img alt="Search Access Connector" />

Choose the resource group, provide a connector name (for example, `unity-catalog-access-connector`), and click **Next**.

<img alt="Connector configuration" />

In the **Managed Identity** panel, enable **System assigned identity**, then click **Next** -> **Create**.

<img alt="Enable managed identity" />

<img alt="Connector created" />

Record the connector **Resource ID**:

<img alt="Connector Resource ID" />

## 2. Grant `Storage Blob Data Contributor` to the Connector

Open the storage account that will hold the Delta tables, navigate to **Access Control (IAM) -> Add -> Add role assignment**.

<img alt="Access control" />

Search for and select **Storage Blob Data Contributor**, then click **Next**.

<img alt="Select Blob Data Contributor" />

Choose **Managed identity** and select the Access Connector created in step 1.

<img alt="Select members" />

Click **Next -> Review + assign**.

<img alt="Role assigned" />

## 3. Grant `Storage Queue Data Contributor` to the Connector

Repeat the process from step 2 with the **Storage Queue Data Contributor** role.

<img alt="Queue Data Contributor" />

Both roles are now assigned to the Access Connector.

<img alt="Both roles assigned" />

## 4. Create a Storage Credential in Unity Catalog

In the Databricks Catalog console, navigate to **Catalog -> Settings -> Credentials**.

<img alt="Credentials menu" />

Click **Create Credential**, provide a name, and paste the Access Connector **Resource ID** from step 1.

<img alt="Create credential" />

<img alt="Credential created" />

## 5. Create an External Location

In the Databricks Catalog console, create a new external location.

<img alt="External locations" />

Configure with:

* **Storage type:** Azure Data Lake Storage
* **URL:** `abfss://<container>@<storage-account>.dfs.core.windows.net`
* **Storage credential:** the credential created in step 4

<img alt="External location form" />

<img alt="External location created" />

Click **Test Connection** to verify the credential.

<img alt="Test connection" />

> **Troubleshooting:** If the test fails with a `Hierarchical Namespace Enabled` error, ensure that **Hierarchical namespace** is enabled on the storage account.

<img alt="Hierarchical namespace" />

<img alt="Hierarchical namespace" />

## 6. Create a Service Principal

Navigate to **User -> Settings -> Identity and access -> Service principals -> Manage**.

<img alt="Service principals" />

Click **Add service principal -> Add new**.

<img alt="Add service principal" />

Choose **Databricks managed** and provide a name.

<img alt="Name service principal" />

Open the service principal, click **Secrets**, choose an expiration period, and **Generate**.

<img alt="Generate secret" />

Record both the **Client ID** and **Client Secret** -- the secret cannot be retrieved later.

<img alt="Secret and Client ID" />

## 7. Create the Catalog

Create a new Catalog with **Type: Standard** and select the **storage location** created in step 5.

<img alt="Create catalog" />

<img alt="Catalog form" />

## 8. Grant Permissions to the Service Principal

### 8.1 Catalog Permissions

Navigate to the new catalog and click **Permissions -> Grant**.

<img alt="Catalog permissions" />

Configure:

* **Principals:** the service principal from step 6
* **Privilege presets:** Data Editor
* **EXTERNAL USE SCHEMA:** Enabled

<img alt="Grant catalog permissions" />

<img alt="Permissions granted" />

### 8.2 External Location Permissions

Open the external location from step 5.

<img alt="External location details" />

<img alt="External location details" />

Click **Grant**, choose the service principal, select **ALL PRIVILEGES**, and click **Confirm**.

<img alt="Grant external location permission" />

<img alt="Permission granted" />

## Catalog Information Summary

When the steps above are complete, collect the following values for the StreamNative Ursa compaction service:

| Value                                               | Description                                                                       |
| --------------------------------------------------- | --------------------------------------------------------------------------------- |
| `unityCatalogUri`                                   | Databricks workspace URL (e.g., `https://adb-<workspace-id>.azuredatabricks.net`) |
| `unityCatalogName`                                  | The Unity Catalog name created in step 7                                          |
| `unityCatalogClientId` / `unityCatalogClientSecret` | OAuth2 credentials from step 6                                                    |

For the next steps, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog).


# Unity Catalog for Delta Lake on GCP
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/unity-catalog/delta-lake/gcp



This guide describes how to prepare a Databricks Unity Catalog for use with StreamNative Ursa as a Delta Lake catalog on Google Cloud Platform (GCP).

## Prerequisites

* A GCP project with permissions to create GCS buckets and IAM roles
* A Databricks account with permissions to create workspaces

## 1. Create a Databricks Workspace

> Skip this step if you already have the Databricks Workspace in GCP

In the GCP Databricks account console, click **Create workspace**.

<img alt="Create workspace" />

Enter the workspace name, choose the region, and provide your GCP project ID.

<img alt="Workspace configuration" />

Click **Save**. The workspace status shows **Provisioning** while initialization is in progress.

<img alt="Workspace provisioning" />

When the status changes to **Running**, the workspace is ready.

<img alt="Workspace running" />

Open the workspace to enter the Unity Catalog console.

<img alt="Unity Catalog console" />

## 2. (Recommend) Generate an OAuth2 Service Principal

For OAuth2 authentication, navigate to **Identity and access -> Service principals -> Manage**.

<img alt="Service principals" />

Click **Add service principal -> Add new** and provide a name.

<img alt="Add service principal" />

<img alt="Create service principal" />

<img alt="Service principal created" />

Open the service principal, click **Secrets -> Generate secret**, choose an expiration period, and **Generate**.

<img alt="Generate secret" />

Record both the **Client ID** and **Client Secret** -- the secret cannot be retrieved later.

<img alt="Client ID and Secret" />

## 3. (Alternative) Generate a User Token

A Databricks user token can be used by StreamNative Ursa to authenticate against Unity Catalog.

Open **User Settings**.

<img alt="User settings" />

Navigate to **Developer -> Access tokens -> Manage** and generate a new token. Record the token value -- it cannot be retrieved later.

<img alt="Developer menu" />

<img alt="Access tokens" />

<img alt="Generate token" />

## 4. Configure Unity Catalog Access

Navigate to **Catalog -> Settings -> Metastore**.

<img alt="Catalog settings" />

Enable **External data access** on the metastore.

<img alt="External data access" />

Grant catalog privileges with the following settings:

* **Principal:** All accounts (or the specific user/service principal)
* **Privilege presets:** Data Editor (selects related privileges automatically)
* **EXTERNAL USE SCHEMA:** Enabled

<img alt="Grant privileges" />

<img alt="Privilege configuration" />

## 5. Grant Bucket Permissions to the Databricks Service Account

When the Databricks workspace is initialized, a service account is created for Unity Catalog.

Navigate to **Catalog -> Settings -> Credentials** to find the service account.

<img alt="Credentials menu" />

Example service account name:

```
db-uc-credential-<id>@uc-uswest1.iam.gserviceaccount.com
```

<img alt="Databricks service account" />

### 5.1 Create a Custom IAM Role

In the GCP console, navigate to **IAM & Admin -> Roles -> Create role** and add the following permissions:

* `storage.buckets.get`
* `storage.objects.create`
* `storage.objects.delete`
* `storage.objects.get`
* `storage.objects.list`

<img alt="Create role" />

<img alt="Role setup" />

<img alt="Permissions" />

<img alt="Permissions selected" />

### 5.2 Assign the Role to the Databricks Service Account

Open your bucket, click **PERMISSIONS -> View BY PRINCIPALS -> GRANT ACCESS**.

<img alt="Grant bucket access" />

Add the Databricks service account, select the role created in step 6.1, and click **SAVE**.

<img alt="Save access" />

## 6. Create an External Location in Unity Catalog

Navigate to **Catalog -> Settings -> External Locations** and create a new external location.

<img alt="External locations" />

<img alt="Create external location" />

Configure with:

* **External location name:** any name
* **URL:** the GCS bucket path
* **Storage credential:** the Unity Catalog credential

<img alt="External location form" />

Click **Test connection** to verify access.

<img alt="External location created" />

Grant **ALL PRIVILEGES** on the external location to the service principal.

<img alt="Grant OAuth2 permissions" />

## Catalog Information Summary

When the steps above are complete, collect the following values for the StreamNative Ursa compaction service:

| Value                                               | Description                                                               |
| --------------------------------------------------- | ------------------------------------------------------------------------- |
| `unityCatalogUri`                                   | Databricks workspace URL (e.g., `https://<workspace>.gcp.databricks.com`) |
| `unityCatalogName`                                  | The Unity Catalog name                                                    |
| `unityCatalogToken`                                 | Personal access token from step 2, **or**                                 |
| `unityCatalogClientId` / `unityCatalogClientSecret` | OAuth2 credentials from step 3                                            |

For the next steps, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog).


# Unity Catalog Managed Iceberg Table on AWS
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/unity-catalog/iceberg/aws



This guide describes how to prepare a Databricks Unity Catalog for use with StreamNative Ursa as a managed Iceberg table catalog on AWS.

## Prerequisites

* A Databricks account with Unity Catalog and Iceberg Managed Table enabled
* An AWS account with permissions to create S3 buckets and IAM roles

## 1. Create an S3 Bucket

In your AWS account, create an S3 bucket to use as the Unity Catalog storage location (for example, `aws-unitycatalog-iceberg-bucket`).

<img alt="Create S3 bucket" />

## 2. Create the IAM Role

### 2.1 Create the Role with a Placeholder Trust Policy

Create an IAM role that allows the Unity Catalog master role to assume it. Use the following trust policy with a placeholder `External ID` of `0000` (you will replace it with the value generated by Databricks in step 3):

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": ["arn:aws:iam::414351767826:role/unity-catalog-prod-UCMasterRole-14S5ZJVKOTYTL"]
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "0000"
        }
      }
    }
  ]
}
```

Skip the permissions policy on this screen -- it will be added in the next steps.

<img alt="Create IAM role" />

<img alt="Save IAM role" />

### 2.2 Attach the S3 Access Policy

Create the following policy and attach it to the role. Replace `<your-bucket>` and `<your-account-id>` with your values.

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads",
        "s3:ListMultipartUploadParts",
        "s3:AbortMultipartUpload"
      ],
      "Resource": [
        "arn:aws:s3:::<your-bucket>/*",
        "arn:aws:s3:::<your-bucket>"
      ]
    },
    {
      "Effect": "Allow",
      "Action": ["sts:AssumeRole"],
      "Resource": ["arn:aws:iam::<your-account-id>:role/<your-role-name>"]
    }
  ]
}
```

<img alt="S3 access policy" />

### 2.3 Attach the File Events Policy

Create a second policy for managed file events (S3 notifications, SNS, SQS) and attach it to the same role:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ManagedFileEventsSetupStatement",
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketNotification",
        "s3:PutBucketNotification",
        "sns:ListSubscriptionsByTopic",
        "sns:GetTopicAttributes",
        "sns:SetTopicAttributes",
        "sns:CreateTopic",
        "sns:TagResource",
        "sns:Publish",
        "sns:Subscribe",
        "sqs:CreateQueue",
        "sqs:DeleteMessage",
        "sqs:ReceiveMessage",
        "sqs:SendMessage",
        "sqs:GetQueueUrl",
        "sqs:GetQueueAttributes",
        "sqs:SetQueueAttributes",
        "sqs:TagQueue",
        "sqs:ChangeMessageVisibility",
        "sqs:PurgeQueue"
      ],
      "Resource": [
        "arn:aws:s3:::<your-bucket>",
        "arn:aws:sqs:*:*:csms-*",
        "arn:aws:sns:*:*:csms-*"
      ]
    },
    {
      "Sid": "ManagedFileEventsListStatement",
      "Effect": "Allow",
      "Action": [
        "sqs:ListQueues",
        "sqs:ListQueueTags",
        "sns:ListTopics"
      ],
      "Resource": [
        "arn:aws:sqs:*:*:csms-*",
        "arn:aws:sns:*:*:csms-*"
      ]
    },
    {
      "Sid": "ManagedFileEventsTeardownStatement",
      "Effect": "Allow",
      "Action": [
        "sns:Unsubscribe",
        "sns:DeleteTopic",
        "sqs:DeleteQueue"
      ],
      "Resource": [
        "arn:aws:sqs:*:*:csms-*",
        "arn:aws:sns:*:*:csms-*"
      ]
    }
  ]
}
```

<img alt="File events policy" />

Verify that both policies are attached to the role.

<img alt="Attach policies to role" />

## 3. Create an External Location in Unity Catalog

In the Databricks Catalog console, create a new external location pointing to the S3 bucket and the IAM role created above.

<img alt="Create external location" />

<img alt="External location settings" />

<img alt="External location summary" />

When you submit the form, Databricks generates an **External ID** and a trust policy. Copy these values.

<img alt="Generated External ID" />

## 4. Update the IAM Role Trust Policy

Return to the AWS IAM console and replace the role's trust policy with the one generated by Databricks in step 3, using the new External ID.

<img alt="Update trust policy" />

<img alt="Trust policy applied" />

After saving the trust policy, click **IAM role configured** in the Databricks catalog console and then **Test connection** to verify the credential.

<img alt="Test connection" />

## 5. Create the Unity Catalog

Create a new catalog in Databricks bound to the external location created in step 3:

* **Type:** Standard
* **Storage location:** the external location created above

<img alt="Create catalog" />

<img alt="Select Standard type" />

<img alt="Select external location" />

## 6. Grant Catalog Permissions

Grant permissions on the catalog. The `EXTERNAL_USE_SCHEMA` permission is **required** for Iceberg Managed Tables in Unity Catalog.

<img alt="Grant permissions" />

<img alt="EXTERNAL_USE_SCHEMA permission" />

## 7. Create OAuth2 Credentials

Create an OAuth2 service principal that StreamNative Ursa will use to authenticate against Unity Catalog.

<img alt="OAuth2 setup" />

<img alt="OAuth2 setup" />

<img alt="OAuth2 setup" />

Generate a secret for the principal and record both the **Client ID** and **Client Secret**.

<img alt="Generate secret" />

## 8. Enable External Data Access on the Metastore

This step is **required** for Unity Catalog Iceberg Managed Tables. Enable **External data access** on the metastore in Databricks.

<img alt="Enable external data access" />

<img alt="External data access enabled" />

## Catalog Information Summary

When the steps above are complete, collect the following values for the StreamNative Ursa compaction service:

| Value      | Description                                                                                                 |
| ---------- | ----------------------------------------------------------------------------------------------------------- |
| URI        | Databricks workspace URL (e.g., `https://dbc-xxxx.cloud.databricks.com/api/2.1/unity-catalog/iceberg-rest`) |
| Warehouse  | The Unity Catalog name created in step 5                                                                    |
| Credential | `<client-id>:<client-secret>` from step 7                                                                   |

For the next steps, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog).


# Unity Catalog Managed Iceberg Table on Azure
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/unity-catalog/iceberg/azure



This guide describes how to prepare a Databricks Unity Catalog for use with StreamNative Ursa as a managed Iceberg table catalog on Microsoft Azure.

## Prerequisites

* A Databricks workspace on Azure with Unity Catalog and Iceberg Managed Table enabled
* An Azure subscription with permissions to create storage accounts and Access Connectors

## 1. Create an Azure Storage Container

Create a storage container in your Azure Storage Account (for example, `unity-catalog-iceberg`). The container path will follow the format:

```
abfss://<container>@<storage-account>.dfs.core.windows.net
```

<img alt="Create storage container" />

<img alt="Storage container" />

## 2. Create an Access Connector for Azure Databricks

Refer to the [Azure Databricks Managed Identities documentation](https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/cloud-storage/azure-managed-identities) for the canonical procedure.

In the Azure Portal, create an **Access Connector for Azure Databricks**.

<img alt="Access Connector" />

<img alt="Access Connector settings" />

<img alt="Access Connector created" />

Record the connector **Resource ID**, which has the form:

```
/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Databricks/accessConnectors/<connector-name>
```

## 3. Grant Storage Permissions to the Access Connector

The Access Connector identity requires the following roles:

| Scope           | Role                                      |
| --------------- | ----------------------------------------- |
| Storage Account | `Storage Blob Data Contributor`           |
| Storage Account | `Storage Queue Data Contributor`          |
| Resource Group  | `EventGrid EventSubscription Contributor` |

### 3.1 Grant `Storage Blob Data Contributor`

<img alt="Grant Blob Data Contributor" />

<img alt="Grant Blob Data Contributor" />

<img alt="Grant Blob Data Contributor" />

<img alt="Grant Blob Data Contributor" />

### 3.2 Grant `Storage Queue Data Contributor`

<img alt="Grant Queue Data Contributor" />

<img alt="Grant Queue Data Contributor" />

<img alt="Grant Queue Data Contributor" />

### 3.3 Grant `EventGrid EventSubscription Contributor`

<img alt="Grant EventGrid Contributor" />

<img alt="Grant EventGrid Contributor" />

<img alt="Grant EventGrid Contributor" />

<img alt="Grant EventGrid Contributor" />

## 4. Create the Unity Catalog Metastore

Create the Unity Catalog metastore in Databricks.

<img alt="Create metastore" />

<img alt="Metastore configuration" />

<img alt="Metastore created" />

## 5. Create a Storage Credential

In the Databricks Catalog console, create a storage credential linked to the Access Connector created in step 2.

<img alt="Create credential" />

<img alt="Credential form" />

## 6. Create the External Location

Create an external location pointing to the Azure storage container:

* **URL:** `abfss://<container>@<storage-account>.dfs.core.windows.net`
* **Storage credential:** the credential created in step 5

<img alt="Create external location" />

<img alt="External location settings" />

## 7. Create the Unity Catalog

Create a new Catalog and bind it to the external location created in step 6.

<img alt="Create catalog" />

<img alt="Catalog form" />

## 8. Grant Catalog Permissions

Grant permissions on the catalog. The `EXTERNAL_USE_SCHEMA` permission is **required** for Iceberg Managed Tables in Unity Catalog.

<img alt="Grant permissions" />

<img alt="EXTERNAL_USE_SCHEMA permission" />

## 9. Enable External Data Access on the Metastore

> **Note:** This action requires **Azure Account Admin** privileges; without them, the **Metastore** entry is not visible.

Enable **External data access** on the metastore. This step is **required** for Unity Catalog Iceberg Managed Tables.

<img alt="Enable external data access" />

<img alt="External data access enabled" />

## 10. Create OAuth2 Credentials

Create an OAuth2 service principal that StreamNative Ursa will use to authenticate.

<img alt="OAuth2 setup" />

<img alt="OAuth2 setup" />

<img alt="OAuth2 setup" />

Generate a secret for the principal and record both the **Client ID** and **Client Secret**.

<img alt="Generate secret" />

## Catalog Information Summary

When the steps above are complete, collect the following values for the StreamNative Ursa compaction service:

| Value      | Description                                                                                                          |
| ---------- | -------------------------------------------------------------------------------------------------------------------- |
| URI        | Databricks workspace URL (e.g., `https://adb-<workspace-id>.azuredatabricks.net/api/2.1/unity-catalog/iceberg-rest`) |
| Warehouse  | The Unity Catalog name created in step 7                                                                             |
| Credential | `<client-id>:<client-secret>` from step 10                                                                           |

For the next steps, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog).


# Unity Catalog Managed Iceberg Table on GCP
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/unity-catalog/iceberg/gcp



This guide describes how to prepare a Databricks Unity Catalog for use with StreamNative Ursa as a managed Iceberg table catalog on Google Cloud Platform (GCP).

## Prerequisites

* A Databricks workspace on GCP with Unity Catalog and Iceberg Managed Table enabled
* A GCP project with permissions to create GCS buckets

## 1. Create a GCS Bucket

Create a GCS bucket to use as the Unity Catalog storage location (for example, `unity-catalog-iceberg-bucket`).

> **Important:** The bucket must be located in the **same region** as your Databricks workspace and your StreamNative Ursa cluster. Cross-region access introduces additional network traffic and latency.

<img alt="Create GCS bucket" />

For additional details, see the [Databricks GCP Unity Catalog documentation](https://docs.databricks.com/gcp/en/data-governance/unity-catalog/create-metastore).

## 2. Create a Storage Credential in Unity Catalog

In the Databricks Catalog console, create a new storage credential. Databricks generates a service account that needs permissions on the bucket.

<img alt="Create credential" />

<img alt="Credential form" />

After creation, record the generated service account name. Example:

```
db-uc-credential-<identifier>@uc-uswest1.iam.gserviceaccount.com
```

<img alt="Generated service account" />

## 3. Grant GCS Permissions to the Service Account

In the GCP console, navigate to the bucket's **Permissions** tab and click **Grant access**.

<img alt="Grant access" />

Grant the following roles to the service account from step 2:

* **Storage Legacy Bucket Reader**
* **Storage Object Admin**

<img alt="Assign storage roles" />

## 4. Create the External Location

In the Databricks Catalog console, create an external location with the following settings:

* **External location name:** any name
* **URL:** the GCS bucket path created in step 1
* **Storage credential:** the credential created in step 2

<img alt="Create external location" />

<img alt="External location settings" />

Use **Test connection** to verify the credential has sufficient permissions.

<img alt="Test connection" />

## 5. Create the Unity Catalog

Create a new Catalog with:

* **Type:** Standard
* **Storage location:** the external location created in step 4 (a sub-path within this location may be specified)

<img alt="Create catalog" />

<img alt="Catalog form" />

<img alt="Catalog created" />

## 6. Grant Catalog Permissions

Grant permissions on the catalog. The `EXTERNAL_USE_SCHEMA` permission is **required** for Iceberg Managed Tables in Unity Catalog.

<img alt="Grant catalog permissions" />

<img alt="EXTERNAL_USE_SCHEMA permission" />

## 7. Enable External Data Access on the Metastore

> **Note:** This action requires **Databricks Account Admin** privileges; without them, the **Metastore** entry is not visible.

Enable **External data access** on the metastore. This step is **required** for Unity Catalog Iceberg Managed Tables.

<img alt="External data access" />

<img alt="External data access enabled" />

## 8. Create OAuth2 Credentials

Create an OAuth2 service principal that StreamNative Ursa will use to authenticate.

<img alt="OAuth2 setup" />

<img alt="OAuth2 setup" />

<img alt="OAuth2 setup" />

Generate a secret for the principal and record both the **Client ID** and **Client Secret**.

<img alt="Generate secret" />

## Catalog Information Summary

When the steps above are complete, collect the following values for the StreamNative Ursa compaction service:

| Value      | Description                                                                                                  |
| ---------- | ------------------------------------------------------------------------------------------------------------ |
| URI        | Databricks workspace URL (e.g., `https://<workspace>.gcp.databricks.com/api/2.1/unity-catalog/iceberg-rest`) |
| Warehouse  | The Unity Catalog name created in step 5                                                                     |
| Credential | `<client-id>:<client-secret>` from step 8                                                                    |

For the next steps, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog).


# Prepare Lakehouse Catalogs
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-lakehouse-catalogs



Preparing an external catalog is **optional**. StreamNative Ursa supports the following catalog modes:

* **Iceberg with the Hadoop catalog (no external catalog).** Iceberg metadata and data files are written directly to the configured object storage path. No external catalog service is required, and downstream engines can read the table by pointing at the storage location.
* **Delta Lake without a catalog.** Delta tables are written directly to the configured storage path; readers point at the path (`s3://...`, `gs://...`, or `abfss://...`) to query them.
* **External catalog.** Use a managed catalog service (Databricks Unity Catalog, Snowflake Open Catalog, Snowflake Horizon Catalog, AWS S3Table, or Google BigLake) to register the tables. This is required if you want governance, discoverability, or integration with managed query engines (Databricks SQL, Snowflake, Athena, BigQuery, etc.).

If you choose to use an external catalog, this page is the index for the catalog preparation guides. Each guide focuses solely on provisioning the catalog itself and its supporting cloud resources (storage bucket, IAM role or service principal, and credentials). Once the catalog is ready, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog) for how to wire it into the compaction service.

> **Skip this page if you are using the Hadoop catalog (Iceberg) or no catalog (Delta).** Go directly to [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog), which describes the no-catalog configuration.

## Catalog Support Matrix

| Catalog                                    | Table Format | AWS                                                                   | GCP                                                                   | Azure                                                                     |
| ------------------------------------------ | ------------ | --------------------------------------------------------------------- | --------------------------------------------------------------------- | ------------------------------------------------------------------------- |
| Databricks Unity Catalog (Managed Iceberg) | Iceberg      | [AWS](/cloud/lakehouse/prepare-catalogs/unity-catalog/iceberg/aws)    | [GCP](/cloud/lakehouse/prepare-catalogs/unity-catalog/iceberg/gcp)    | [Azure](/cloud/lakehouse/prepare-catalogs/unity-catalog/iceberg/azure)    |
| Databricks Unity Catalog (Delta Lake)      | Delta Lake   | [AWS](/cloud/lakehouse/prepare-catalogs/unity-catalog/delta-lake/aws) | [GCP](/cloud/lakehouse/prepare-catalogs/unity-catalog/delta-lake/gcp) | [Azure](/cloud/lakehouse/prepare-catalogs/unity-catalog/delta-lake/azure) |
| Snowflake Open Catalog (Polaris)           | Iceberg      | [AWS](/cloud/lakehouse/prepare-catalogs/open-catalog/iceberg/aws)     | [GCP](/cloud/lakehouse/prepare-catalogs/open-catalog/iceberg/gcp)     | [Azure](/cloud/lakehouse/prepare-catalogs/open-catalog/iceberg/azure)     |
| Snowflake Horizon Catalog                  | Iceberg      | [AWS](/cloud/lakehouse/prepare-catalogs/horizon-catalog/iceberg/aws)  | --                                                                    | --                                                                        |
| AWS S3Table                                | Iceberg      | [Iceberg](/cloud/lakehouse/prepare-catalogs/s3table/iceberg)          | --                                                                    | --                                                                        |
| Google BigLake                             | Iceberg      | --                                                                    | [Iceberg](/cloud/lakehouse/prepare-catalogs/biglake/iceberg)          | --                                                                        |

## Choosing a Catalog

| Use case                                                            | Recommended catalog                                                        |
| ------------------------------------------------------------------- | -------------------------------------------------------------------------- |
| Manage Iceberg tables alongside Databricks workloads                | [Unity Catalog (Iceberg)](#databricks-unity-catalog-managed-iceberg-table) |
| Manage Delta Lake tables alongside Databricks workloads             | [Unity Catalog (Delta Lake)](#databricks-unity-catalog-delta-lake)         |
| Cloud-portable Iceberg REST catalog from Snowflake                  | [Snowflake Open Catalog (Polaris)](#snowflake-open-catalog-polaris)        |
| Governed Iceberg tables managed by Snowflake Horizon                | [Snowflake Horizon Catalog](#snowflake-horizon-catalog)                    |
| AWS-native Iceberg tables with built-in Athena/Redshift integration | [AWS S3Table](#aws-s3table)                                                |
| GCP-native Iceberg with BigQuery/BigLake integration                | [Google BigLake](#google-biglake)                                          |

## Catalog Preparation Guides

### Databricks Unity Catalog (Managed Iceberg Table)

Use Unity Catalog to manage Iceberg tables governed by Databricks. The compaction service writes to Iceberg Managed Tables; downstream readers query them via the Unity Catalog Iceberg REST endpoint.

| Cloud | Guide                                                                                                         |
| ----- | ------------------------------------------------------------------------------------------------------------- |
| AWS   | [Unity Catalog Managed Iceberg Table on AWS](/cloud/lakehouse/prepare-catalogs/unity-catalog/iceberg/aws)     |
| GCP   | [Unity Catalog Managed Iceberg Table on GCP](/cloud/lakehouse/prepare-catalogs/unity-catalog/iceberg/gcp)     |
| Azure | [Unity Catalog Managed Iceberg Table on Azure](/cloud/lakehouse/prepare-catalogs/unity-catalog/iceberg/azure) |

### Databricks Unity Catalog (Delta Lake)

Use Unity Catalog to manage Delta Lake tables. The compaction service writes Delta Lake files governed by Unity Catalog and queryable from Databricks SQL or external Spark sessions.

| Cloud | Guide                                                                                                     |
| ----- | --------------------------------------------------------------------------------------------------------- |
| AWS   | [Unity Catalog for Delta Lake on AWS](/cloud/lakehouse/prepare-catalogs/unity-catalog/delta-lake/aws)     |
| GCP   | [Unity Catalog for Delta Lake on GCP](/cloud/lakehouse/prepare-catalogs/unity-catalog/delta-lake/gcp)     |
| Azure | [Unity Catalog for Delta Lake on Azure](/cloud/lakehouse/prepare-catalogs/unity-catalog/delta-lake/azure) |

### Snowflake Open Catalog (Polaris)

Snowflake Open Catalog (Polaris) is a cloud-agnostic Iceberg REST catalog operated by Snowflake. It can be used as the catalog for Iceberg tables hosted on AWS, GCP, or Azure object storage.

| Cloud | Guide                                                                                             |
| ----- | ------------------------------------------------------------------------------------------------- |
| AWS   | [Open Catalog for Iceberg on AWS](/cloud/lakehouse/prepare-catalogs/open-catalog/iceberg/aws)     |
| GCP   | [Open Catalog for Iceberg on GCP](/cloud/lakehouse/prepare-catalogs/open-catalog/iceberg/gcp)     |
| Azure | [Open Catalog for Iceberg on Azure](/cloud/lakehouse/prepare-catalogs/open-catalog/iceberg/azure) |

### Snowflake Horizon Catalog

Snowflake Horizon provides governed Iceberg tables with native Snowflake integration. The catalog uses a Snowflake **External Volume** backed by an S3 bucket and authenticates via a Programmatic Access Token (PAT).

| Cloud | Guide                                                                                               |
| ----- | --------------------------------------------------------------------------------------------------- |
| AWS   | [Horizon Catalog for Iceberg on AWS](/cloud/lakehouse/prepare-catalogs/horizon-catalog/iceberg/aws) |

### AWS S3Table

AWS S3Table is the AWS-native Iceberg catalog with first-class integration into AWS analytics services such as Athena, Redshift, and EMR.

| Cloud | Guide                                                                    |
| ----- | ------------------------------------------------------------------------ |
| AWS   | [S3Table for Iceberg](/cloud/lakehouse/prepare-catalogs/s3table/iceberg) |

> **Important:** The Ursa cluster must run in the **same region** as the S3Table bucket. Cross-region access is not supported.

### Google BigLake

Google BigLake provides an Iceberg REST catalog tightly integrated with BigQuery and Google Cloud Storage.

| Cloud | Guide                                                                    |
| ----- | ------------------------------------------------------------------------ |
| GCP   | [BigLake for Iceberg](/cloud/lakehouse/prepare-catalogs/biglake/iceberg) |

> **Important:** The Ursa cluster, GCS bucket, and BigLake catalog must all be in the same region. Each BigLake catalog maps to exactly one GCS bucket (1:1 mapping; sub-paths are not supported).

## Next Steps

After preparing your catalog, proceed to:

1. [Dynamic Configuration Guide](/cloud/lakehouse/dynamic-configuration) -- Reference for all dynamic configuration keys and the cluster-name prefix requirement.
2. [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog) -- Connect the prepared catalog to the StreamNative Ursa compaction service.
3. [Enable Lakehouse Integration](/cloud/lakehouse/enable-lakehouse-integration) -- Enable SDT (External Table) at the cluster, namespace, or topic level.


# Manage Data Streams in StreamNative Cloud
Source: https://docs.streamnative.io/cloud/manage-data-streams/data-streams-overview



StreamNative Cloud leverages Apache Pulsar, a multi-tenant messaging and data streaming platform. Within Apache Pulsar, data streams are organized into Topics, which are grouped into Tenants and Namespaces. Tenants can span multiple [clusters](/cloud/clusters/manage-clusters/cluster) within a [StreamNative Instance](/cloud/clusters/manage-instances/instance), and each tenant serves as an administrative unit where storage quotas, message TTL (Time to Live), isolation policies and etc. are managed.

The structure of topic URLs in Pulsar, which highlights its multi-tenant architecture, is as follows:

```
[persistent|non-persistent]://tenant/namespace/topic
```

In this structure, the tenant is the primary categorization unit for topics, more fundamental than the namespace or topic name itself.

You can create and manage tenants, namespaces, topics, and their configurations and policies through the [StreamNative Cloud console](https://console.streamnative.cloud), the StreamNative CLI[ (`snctl`)](/tools/cli/snctl/snctl-overview),the [Pulsar CLI (`pulsarctl`)](/tools/cli/pulsarctl/pulsarctl-overview), or [Pulsar Admin APIs](https://pulsar.apache.org/docs/admin-api-overview/).

## Tenants

A [Pulsar tenant](/cloud/manage-data-streams/tenant) is an administrative entity used for capacity allocation and implementing authentication or authorization schemes. Each tenant in a Pulsar instance can have:

* An assigned authorization scheme
* A defined set of clusters where the tenant's configurations are applicable

## Namespace

A [namespace](/cloud/manage-data-streams/namespace) in Pulsar is a logical grouping of topics. It serves as an administrative segment within a tenant, where configuration policies set at the namespace level apply to all topics within that namespace. A tenant can establish multiple namespaces to accommodate different applications, for example:

```bash theme={null}
persistent://tenant/app1/topic-1
persistent://tenant/app2/topic-2
persistent://tenant/app3/topic-3
```

## Topic

A [topic](/cloud/manage-data-streams/topic) in Pulsar is a storage unit that organizes messages into a stream. Similar to other publish-subscribe systems, topics in Pulsar act as named channels that facilitate message transmission from producers to consumers. The naming convention for topics follows a specific URL pattern:

```bash theme={null}
{persistent|non-persistent}://tenant/namespace/topic
```

The components of a topic name are defined as:

| Topic name component          | Description                                                                                                                                                                                                                                                                        |
| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `persistent / non-persistent` | Indicates the type of topic. Pulsar supports both persistent and non-persistent topics. Persistent topics save all messages on disk, ensuring durability (messages are stored on multiple disks unless the broker is standalone). Non-persistent topics do not save data on disks. |
| `tenant`                      | Represents the tenant within the instance, highlighting Pulsar's support for multi-tenancy.                                                                                                                                                                                        |
| `namespace`                   | Acts as a grouping mechanism for related topics, with most configuration done at this level. Each tenant may have one or more namespaces.                                                                                                                                          |
| `topic`                       | The specific name of the topic within a namespace. Topic names in Pulsar do not carry intrinsic meanings beyond their hierarchical organization.                                                                                                                                   |

## Relate topics

This section could include links or references to further reading on related topics or advanced configurations in Pulsar.


# Work with Namespaces in StreamNative Cloud
Source: https://docs.streamnative.io/cloud/manage-data-streams/namespace



<Note title="Note">
  This document introduces the instructions for working with namespaces on StreamNative Console. The details may vary depending on the specific product and version number that you use.
</Note>

A **namespace** is a logical grouping of topics. After creating a tenant, you can create one or more namespaces for the tenant.

## Namespace Overview

A namespace represents an administrative unit within a tenant. The configuration policies set on a namespace apply to all the topics created in that namespace. You can create multiple namespaces for a tenant using the StreamNative Cloud Console, REST API or the pulsar-admin CLI tool.

### Permissions

In Pulsar, permissions are managed at the namespace level (within tenants and clusters). You can grant permissions to specific users for lists of operations such as `produce` and `consume`. In addition, you can revoke permissions from specific users, which means that those users cannot access the specified namespace.

### Backlog quotas

Backlogs are sets of unacknowledged messages for a topic that have been stored by bookies. Pulsar stores all unacknowledged messages in backlogs until they are processed and acknowledged.
You can use the backlog quotas to control the allowable size of backlogs at the namespace level. You can set the following items for a backlog quota:

* an allowable size threshold for each topic in the namespace
* a retention policy that determines the action the broker takes if the threshold is exceeded.

The following table lists available retention policies.

| Policy                      | Action                                                            |
| --------------------------- | ----------------------------------------------------------------- |
| `producer_request_hold`     | The broker holds but does not persist producers' request payload. |
| `producer_exception`        | The broker disconnects from the client by throwing an exception.  |
| `consumer_backlog_eviction` | The broker begins discarding backlog messages.                    |

### Bundles

For assignment, a namespace is sharded into a list of bundles, with each bundle comprising a portion of the overall hash range of the namespace. A bundle is a virtual group of topics that belong to the same namespace. A namespace bundle is defined as a range between two 32-bit hashes, such as 0x00000000 and 0xffffffff. By default, four bundles are supported for each namespace.

Since the load for topics in a bundle might change over time, one bundle can be split into two bundles by brokers. Then, the new smaller bundle is reassigned to different brokers. By default, the newly split bundles are immediately offloaded to other brokers to facilitate the traffic distribution.

### Dispatch rate

Dispatch rate refers to the number of messages dispatched per second by topics for a namespace. Dispatch rate can be restricted by the number of messages per second (`msg-dispatch-rate`) or by the number of bytes of messages per second (`byte-dispatch-rate`). Dispatch rate is in seconds and it can be configured with `dispatch-rate-period`. By default, `msg-dispatch-rate` and `byte-dispatch-rate` are both set to -1, which indicates that throttling is disabled.

## Create a namespace

<Tabs>
  <Tab title="StreamNative Console">
    To create a namespace, follow these steps.

    1. On the left navigation pane, under **Tenants/Namespaces**, select the name of the tenant you want to associate with the new namespace, and click **New Namespace**.

    2. On the Namespace page, click **New Namespace**.

    3. Enter a name for the namespace and then click **Confirm**. A namespace name can contain any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-).
  </Tab>
</Tabs>

## Manage a namespace

<Tabs>
  <Tab title="StreamNative Console">
    This section describes how to manage namespaces through the StreamNative Console.

    1. On the left navigation pane, under **Tenants/Namespaces**, select the name of the Tenant/Namespace you want to manage.

    2. Select the **OVERVIEW** tab to check statistics about the namespace, as well as unload and split bundles. The following table lists statistics about the namespace.

       | Item           | Description                              |
       | -------------- | ---------------------------------------- |
       | In Rate        | The ingress rate of the namespace.       |
       | Out Rate       | The egress rate of the namespace.        |
       | In Throughput  | The ingress throughput of the namespace. |
       | Out Throughput | The egress throughput of the namespace.  |

    3. Select the **TOPICS** tab to check the number of topics included in the namespace and statistics about topics. In addition, you can update specific topics. For details about how to update topics, see [edit topics](/cloud/manage-data-streams/topic#create-a-topic).

    4. Select the **POLICIES** tab to configure related policies for the namespace. For details, see [configure policies for a namespace](#configure-policies-for-a-namespace).
  </Tab>
</Tabs>

### Unload bundles for a namespace

<Tabs>
  <Tab title="StreamNative Console">
    For an assignment, a namespace is sharded into a list of bundles, with each bundle comprising a portion of the overall hash range of the namespace. By default, four bundles are supported for each namespace.

    To unload bundles for a namespace, follow these steps.

    1. On the left navigation pane, under **Tenants/Namespaces**, select the name of the Tenant/Namespace you need to edit, and click **New Namespace**.

    2. Click **Edit** in the **Actions** column.

    3. Select the **OVERVIEW** tab.

       <img alt="image of namespace overview" />

    4. In the **Bundle** section, select the target cluster from the **Cluster** drop-down list, and then do one of the following:

       * Click **Unload All** to unload all bundles for the namespaces in the cluster.

       * Click **Unload** in the Operation column to unload a specific bundle for the namespace.
  </Tab>
</Tabs>

### Split bundles for a namespace

<Tabs>
  <Tab title="StreamNative Console">
    Because the load for topics in a bundle might change over time, one bundle can be split in two bundles by brokers. Then, the new smaller bundles are reassigned to different brokers. By default, the newly split bundles are immediately offloaded to other brokers to facilitate the traffic distribution.

    To split bundles for a namespace, follow these steps.

    1. From the left navigation pane, click **Namespaces**.

    2. Click **Edit** icon (it looks like a pencil.) in the **Action** column or click the link of the namespace name.

    3. Select **OVERVIEW** tab.

       <img alt="screenshot for splitting bundles" />

    4. Click **Split**.
  </Tab>
</Tabs>

### Clear namespace backlog

<Tabs>
  <Tab title="StreamNative Console">
    To clear backlog for bundles of a namespace, follow these steps.

    1. On the left navigation pane, under **ADMIN**, click **Tenants/Namespaces**.

    2. Select the name of the tenant that is associated with the target namespace.

    3. Click the name of the namespace that you want to clear backlog for.

    4. In the **Bundle** section of the **OVERVIEW** tab, click **Clean All Backlog** to clear the backlog for all the topics of the namespace.
  </Tab>
</Tabs>

### Configure policies for a namespace

<Tabs>
  <Tab title="StreamNative Console">
    To configure policies for a namespace, follow these steps.

    1. From the left navigation pane, click **Namespaces**.

    2. Click **Edit** in the **Action** column or click the link of the namespace name.

    3. Select the **Policies** tab.

       <img alt="screenshot of namespace policy" />

    4. Configure policies for the namespace, as listed in the following table.

       | Item     | Description                                                                                                                                                                                                                                                            |
       | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
       | Clusters | Select a replication cluster. Messages of the topics in this namespace are asynchronically replicated between the configured replication clusters. Currently, this item is set to the cluster you created because there is only one cluster available for an instance. |

    <br />

    <br />

    <br />

    <br />

    <br />

    <br />

    <br />

    <br />
  </Tab>
</Tabs>

## Delete a namespace

<Note title="Note">
  You cannot delete a namespace if there are resources associated with the namespace.
</Note>

<Tabs>
  <Tab title="StreamNative Console">
    To delete a namespace, follow these steps.

    1. On the left navigation pane, in the **Admin** section, click **Namespaces**.

    2. In the **Warning** section at the bottom of the Namespace Policy page, click **Delete Namespace**. A dialog box displays asking, *Are you sure you want to delete this?*

    3. Enter the namespace name and then click **Confirm**.
  </Tab>
</Tabs>

## Next step

* [Work with topics](/cloud/manage-data-streams/topic)

## Related content

* Learn more information about managing namespaces using Pulsar Admin API, see [Managing Namespaces](https://pulsar.apache.org/docs/admin-api-namespaces/).


# Work with Tenants in StreamNative Cloud
Source: https://docs.streamnative.io/cloud/manage-data-streams/tenant



<Note title="Note">
  This document introduces the instructions for working with tenants on StreamNative Cloud. The details may vary depending on the specific product and version number that you use.
</Note>

A **tenant** is an administrative unit for allocating capacity and enforcing an authentication or authorization scheme. After creating a cluster, you can create one or more tenants for the organization.

## Create a tenant

<Tabs>
  <Tab title="StreamNative Console">
    To create a tenant, follow these steps.

    1. On the left navigation pane, under **Tenants/Namespaces**, select the current default tenant/namespace, and then click **New Tenant**.

    2. On the Tenants page, click **New Tenant**.

    3. Configure the tenant, as outlined in the following table.

       | Item             | Description                                                                                                                       |
       | ---------------- | --------------------------------------------------------------------------------------------------------------------------------- |
       | Tenants          | Enter a name for the tenant. A tenant name can contain any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-) |
       | Allowed Clusters | Select an allowed cluster for the tenant.                                                                                         |
       | Admin Roles      | (Optional) Select one or more administrators for the tenant.                                                                      |

    4. Click **Confirm**.
  </Tab>
</Tabs>

## Edit a tenant

<Tabs>
  <Tab title="StreamNative Console">
    To edit a tenant, follow these steps.

    1. On the left navigation pane, in the **Admin** section, click **Tenants**.

    2. Click **Edit** in the **Action** column.

    3. Select the **Configuration** tab where you can edit the following:

       | Item             | Description                                                                                                                          |
       | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
       | Allowed Clusters | Select one allowed cluster for the tenant.                                                                                           |
       | Admin Roles      | Select one administrator for the tenant. Or, you can click **Add Role** and then select the available administrators for the tenant. |

    4. To add more namespaces for the tenant, click the link on the tenant name and then click **New Namespace**. For details about how to create a namespace, see [create a namespace](/cloud/manage-data-streams/namespace#create-a-namespace).
  </Tab>
</Tabs>

## Delete a tenant

<Note title="Note">
  You cannot delete a tenant if there are resources associated with the tenant.
</Note>

<Tabs>
  <Tab title="StreamNative Console">
    To delete a tenant, follow these steps.

    1. On the left navigation pane, in the **Admin** section, click **Tenants**.

    2. Click **Edit** in the **Action** column.

    3. Select the **Configuration** tab.

    4. Click **Delete Tenant**. A dialog box displays asking, *Are you sure you want to delete this?*

    5. Enter the tenant name and then click **Confirm**.
  </Tab>
</Tabs>

## Next step

* [Work with namespaces](/cloud/manage-data-streams/namespace)

## Related content

* Learn more information about managing tenants using Pulsar Admin API, see [Managing Tenants](https://pulsar.apache.org/docs/admin-api-tenants/).


# Work with Topics in StreamNative Cloud
Source: https://docs.streamnative.io/cloud/manage-data-streams/topic



<Note title="Note">
  This document introduces the instructions for working with topics on StreamNative Console. The details may vary depending on the specific product and version number that you use.
</Note>

## Topic Overview

A **topic** is a named channel used to deliver messages published by producers to consumers. After creating a namespace, you can create one or more topics for the namespace. It's recommended to have at least one partition per topic so that you can add more partitions in the future. If there are zero partitions (a non-partitioned topic), you will not be able to add more partitions to the topic after it's created.

As in other pub-sub systems, topics in Pulsar are named channels for transmitting messages from producers to consumers. Pulsar supports persistent and non-persistent topics. By default, a persistent topic is created if you don't specify a topic type. With persistent topics, all messages are durably persisted on disks (if the broker is not standalone, messages are durably persisted on multiple disks), whereas data for non-persistent topics is not persisted to storage disks.

### Non-persistent topics

Pulsar also supports non-persistent topics, which are topics on which messages are never persisted to disk and live only in memory. When using non-persistent delivery, stopping a Pulsar broker or disconnecting a subscriber to a topic means that all in-transit messages are lost on that non-persistent topic. In non-persistent topics, brokers immediately deliver messages to all connected subscribers without persisting them in BookKeeper.

### Partitioned topics

Normal topics are served only by a single broker that limits the maximum throughput of the topic. Partitioned topics are a special type of topic handled by multiple brokers, allowing for higher throughput.

A partitioned topic is actually implemented as N internal topics, where N is the number of partitions. When publishing messages to a partitioned topic, each message is routed to one of several brokers. The distribution of partitions across brokers is handled automatically by Pulsar.

It's recommended to have at least one partition per topic so that you can add more partitions in the future. If there are zero partitions (a non-partitioned topic), you will not be able to add more partitions to the topic after it is created.

## Create a topic

<Tabs>
  <Tab title="StreamNative Console">
    To create a topic, follow these steps.

    1. Navigate to the desired **Namespace Dashboard** page by [switching to the namespace workspace](/cloud/get-started/cloud-console#switch-a-namespace).

    2. On the left navigation pane, under **Resources**, click **Topics**.

    3. Click **New Topic**.

       <img alt="screenshot of new topic dialog box" />

    4. Configure the topic, as outlined in the following table.

       | Item       | Description                                                                                                                                                                                                                                                                    |
       | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
       | Persistent | Configure the type of the topic. <br />- Persistent: messages in a persistent topic are durably persisted on the storage disk. <br />- Non-persistent: messages in a non-persistent topic are not persisted on the storage disk. <br /> By default, it is set to *Persistent*. |
       | Topic Name | Enter a name for the topic. It is a string of up to 40 characters, supporting lowercase letters (a-z), numeric characters (0-9), and the special character hyphen (-).                                                                                                         |
       | Partitions | (Optional) Configure the number of partitions for a partitioned topic. You can have from 1 up to 100 partitions per topic. It's recommended to have at least 1 partition.                                                                                                      |

    5. Click **Confirm**.
  </Tab>
</Tabs>

<Note title="Note">
  Currently, snctl does not support creating topics.
</Note>

## Manage topics

This section describes how to manage topics through the StreamNative Console.

* For how to use snctl to manage topics, see [snctl command reference](https://doc-references.streamnative.io/snctl/latest/index.html#pulsar).
* For how to use pulsarctl to manage topics, see [pulsarctl command reference](https://doc-references.streamnative.io/pulsarctl/latest/index.html#topics).

### Perform basic operations for topics

<Tabs>
  <Tab title="StreamNative Console">
    To perform basic operations for a topic, follow these steps.

    1. On the left navigation pane, under **Resources**, click **Topics**.

    2. Click the topic name link to display detailed information about that topic. You will arrive to the Dashboard with general metrics information about the topic

       <img alt="screenshot of topic dashboard" />

    If you go on the detail tabs you can perform the following operations over the partitions:

    * Unload the topic: click **Unload** to unload the topic.
    * Compact the topic: click **Compact** to compact the topic.
    * Create new subscriptions: click **New Subscription** and a dialog box displays. Enter a name for the subscription and then click **Confirm**.
    * Review other subscriptions
    * Check detailed information about the topic: Cursors, Segments, Producers, Consumers or Stats

    <br />
  </Tab>
</Tabs>

### Create schema for topics

<Tabs>
  <Tab title="StreamNative Console">
    <Note title="Note">
      Currently, only Avro, JSON and Protobuf schema are supported.
    </Note>

    To configure schema for topics, follow these steps.

    1. On the left navigation pane, under **Resources**, click **Topics**.

    2. Click the topic name link.

    3. Select the **Schema** tab. Click to Create a Pulsar schema.

       <img alt="screenshot of the create schema page" />

    4. Select a schema type.

    5. Configure the key and the value and then click **Confirm**.
  </Tab>
</Tabs>

### Check messages

<Tabs>
  <Tab title="StreamNative Console">
    1. On the left navigation pane, under **Resources**, click **Topics**.

    2. Click the link of the topic name to display the dashboard about the topic.

    3. Select the **MESSAGES** tab, choose the partition and subscription to select the position to peek and the number of messages. Push Confirm to see the content of the messages.

    4. Then you can check the information about messages in this topic.

       | Item       | Description            |
       | ---------- | ---------------------- |
       | Message ID | Internal Message ID    |
       | Message    | Content of the message |
       | Properties | The message properties |

    <br />
  </Tab>
</Tabs>

### Configure topic policies, authorization and replicated clusters

<Tabs>
  <Tab title="StreamNative Console">
    To configure policies for topics, follow these steps.

    1. On the left navigation pane, under **Resources**, click **Topics**.

    2. Click the link of the topic name.

    3. Select the **CONFIGURATION** tab and configure authorization policies for the topic.

       <img alt="screenshot of the topic policies tab" />

    4. There you can see the sections for replicate clusters, authorization and different policies
  </Tab>
</Tabs>

### View topic statistics

<Tabs>
  <Tab title="StreamNative Console">
    You can find the latest statistics for a topic and its connected producers and consumers on StreamNative Console, for example, whether the topic has received messages, whether there's a backlog, and so on.

    To view the statistics of a topic, follow these steps.

    1. On the left navigation pane, under **Resources**, click **Topics**.

    2. Click the link of the topic name.

    3. Near the refresh there are 3 vertical dots where you can choose to see the **stats**.

    4. To view the latest stats to monitor your cluster in real time, click the symbol **THREE VERTICAL DOTS** and choose **stats**.
  </Tab>
</Tabs>

## Delete topics

<Tabs>
  <Tab title="StreamNative Console">
    To delete a topic, follow these steps.

    1. On the left navigation pane, under **Resources**, click **Topics**.

    2. Hover over the topic name. And a **delete symbol** will appear at the end of the line.

    3. Click over it.

    4. A dialog box displays asking, *Are you sure you want to delete this?*

    5. Enter the topic name and then click **Confirm**.
  </Tab>
</Tabs>

## Related content

* Learn more information about managing topics using Pulsar Admin API, see [Managing Topics](https://pulsar.apache.org/docs/admin-api-topics/).


# Choose Kafka or Pulsar
Source: https://docs.streamnative.io/cloud/overview/choose-kafka-or-pulsar

Choose the right protocol for your workload. Kafka for event streaming, Pulsar for messaging and multi-protocol workloads.

StreamNative Cloud offers two data streaming services: **Kafka Service** and **Pulsar Service**. Both run on the [URSA engine](/cloud/overview/data-streaming-engine). Choose based on your workload pattern and existing ecosystem.

<Note>
  If you are evaluating the native Pulsar protocol on the Cost-Optimized profile, note that support is coming after the Apache Pulsar 5.0 release. Today, Cost-Optimized Pulsar Clusters expose the Kafka-compatible protocol. See [Cluster Profiles Overview](/cloud/clusters/cluster-profiles-overview) for the full capability matrix.
</Note>

## Quick decision

<Tabs>
  <Tab title="Choose Kafka if">
    * You have **existing Kafka workloads** or Kafka client libraries in production.
    * Your primary pattern is **event streaming**: event sourcing, log aggregation, CDC, clickstream, or telemetry.
    * You want **native Kafka API behavior** with full compatibility for producers, consumers, consumer groups, and Kafka Connect.
    * You are **migrating from Amazon MSK, Confluent Cloud, or self-managed Kafka** and want a drop-in replacement.
  </Tab>

  <Tab title="Choose Pulsar if">
    * You need **messaging and streaming in a single system** (unified model).
    * Your primary pattern is **task queues, job dispatch, or request-reply** where messages are dispatched, acknowledged, and removed.
    * You need **multi-tenancy with namespace isolation** to separate workloads for different teams or environments on the same cluster.
    * You need **built-in geo-replication** across regions with automatic failover.
    * You want **multi-protocol access** (Pulsar + Kafka + MQTT + REST) on the same cluster.
  </Tab>
</Tabs>

## Feature comparison

| Capability                         | Kafka Cluster                     | Pulsar Cluster                                                   |
| ---------------------------------- | --------------------------------- | ---------------------------------------------------------------- |
| **Primary protocol**               | Kafka (native)                    | Pulsar (native)                                                  |
| **Multi-protocol on same cluster** | Kafka only                        | Pulsar, Kafka (via KSN), MQTT (via MoP), REST                    |
| **Multi-tenancy**                  | Topic-level ACLs                  | Tenant and namespace isolation                                   |
| **Geo-replication**                | UniLink                           | Built-in, active-active across regions                           |
| **Queue semantics**                | Queues for Kafka                  | Shared subscriptions with per-message acknowledgment             |
| **Consumer model**                 | Partition failover + queue        | Flexible subscriptions (exclusive, shared, failover, key-shared) |
| **Lakehouse integration**          | Built-in via URSA (both profiles) | Built-in via URSA (both profiles)                                |
| **Status**                         | Public Preview                    | GA                                                               |

## Messaging vs. streaming

The core difference between the two services maps to two data movement patterns: messaging and streaming.

### Messaging (Pulsar strength)

Messaging is **consumer-centric delivery**. Messages are dispatched to consumers, acknowledged individually, and removed from the backlog once processed. This model works well when you need:

* **Competing consumers**: Multiple workers sharing a workload through shared subscriptions.
* **Per-message acknowledgment**: Track processing at the individual message level.
* **Flexible routing**: Route messages to different consumers based on subscription type (exclusive, shared, failover, key-shared).

Pulsar Service is designed for this pattern. Its subscription model gives you fine-grained control over how messages are distributed and acknowledged.

<Note>
  The messaging features in Pulsar have seen widespread adoption and have been running in production for many years. Queue support for Kafka is relatively new — evaluate carefully before adopting it for production messaging workloads.
</Note>

### Streaming (Kafka strength)

Streaming is **log-centric processing**. Events are appended to a durable, ordered log. Consumers read at their own pace using offsets and can replay the log at any time. This model works well when you need:

* **Event replay**: Reprocess historical data by resetting offsets.
* **Ordered processing**: Maintain strict ordering within partitions.
* **Decoupled consumers**: Multiple independent consumer groups reading the same log without coordination.

Kafka Service is designed for this pattern. Its consumer group model gives each consumer group an independent view of the log.

## Powered by URSA engine

Both services are built on the Lakestream architectural paradigm and powered by the URSA engine, which provides:

* **Multiple WAL options**: Apache BookKeeper for Pulsar low-latency, KRaft + local disks for Kafka low-latency, and object storage for cost-optimized profiles.
* **Leaderless brokers** (on Cost-Optimized profiles): Any broker handles any partition, eliminating leader bottlenecks.
* **Compute/storage separation**: Provides the flexibility to choose between local disks for low latency and shared storage (lakehouse storage) for cost-efficiency.
* **Lakehouse-native storage**: Data stored in open formats (Iceberg, Delta) for direct query by analytics engines — available on all profiles.
* **Cost reduction**: Up to 95% lower infrastructure costs on Cost-Optimized profiles by eliminating inter-AZ replication.

To learn more about the architecture, see [Lakestream Architecture](/cloud/overview/lakestream-overview) and the [Data Streaming Engine](/cloud/overview/data-streaming-engine) overview.

## Next steps

<CardGroup>
  <Card title="Get Started with Kafka Service" icon="chart-line" href="/kafka/overview">
    Native Apache Kafka on Ursa. 100% Kafka compatible, leaderless, and lakehouse-native.
  </Card>

  <Card title="Explore Pulsar Service" icon="wave-pulse" href="/cloud/overview/cloud-overview">
    Apache Pulsar with multi-protocol support for messaging and streaming workloads.
  </Card>
</CardGroup>


# Welcome to StreamNative Cloud
Source: https://docs.streamnative.io/cloud/overview/cloud-overview



StreamNative Cloud offers two fully managed data streaming services: **[Kafka Service](/kafka/overview)** for event streaming workloads, and **Pulsar Service** for messaging and multi-protocol workloads. Both services run on the [URSA engine](/cloud/overview/data-streaming-engine) and the [Lakestream architecture](/cloud/overview/lakestream-overview), with cluster profiles that trade off latency and cost through different WAL and metadata configurations.

<Tip>
  See [Cluster Profiles Overview](/cloud/clusters/cluster-profiles-overview) for the full protocol × profile capability matrix, including which features are available today versus coming soon on each cluster type.
</Tip>

StreamNative Cloud incorporates all the features of Apache Pulsar and Apache Kafka, plus StreamNative's tooling to remove the complexity of managing streaming platforms. Our managed deployment options provide a turnkey solution for organizations transitioning to a "streaming first" architecture.

You can choose between two deployment options to easily and safely connect to your existing tech stack:

* **Fully Hosted**: StreamNative clusters hosted on StreamNative's public cloud account, available on AWS, GCP, and Azure. You can choose between [Serverless](/cloud/clusters/cluster-types#serverless-clusters) and [Dedicated](/cloud/clusters/cluster-types#dedicated-clusters) clusters.

* **Bring Your Own Cloud**: On your public cloud account (AWS, GCP, or Azure), managed by StreamNative. You can choose between [BYOC](/cloud/clusters/cluster-types#byoc-clusters) and [BYOC Pro](/cloud/clusters/cluster-types#byoc-pro-clusters).

These products offer a flexible model of clusters either running in StreamNative's cloud or in your organization's own cloud accounts.

## Fully Hosted

The StreamNative Cloud Fully Hosted deployment is ideal for standalone teams who value speed and ease of use over granular control of cloud infrastructure. Use the hosted deployment if you don't have existing infrastructure, lack experience in managing infrastructure, or need to quickly spin up a new project.

With the hosted deployment, you can focus on implementation instead of infrastructure management.

* Limited infrastructure decisions and maintenance are required.
* You have full control of the implementation.
* You do not need DevOps resources.

## Bring Your Own Cloud (BYOC)

The StreamNative Cloud BYOC deployment is ideal for customers who already have their own public cloud infrastructure and are capable of managing their infrastructure, maintaining security protocols, and adhering to data retention policies.

With the BYOC deployment, you can control your data, security, and costs.

* You have full data ownership.
* You manage security protocols.
* You have flexible data retention options.
* You can expect a lower total cost of ownership (TCO) due to optimized network costs.

StreamNative Cloud deployments enable developers to focus on building applications, instead of managing and maintaining complex systems and data services. Developers can spin up a Kafka or Pulsar streaming service in the public cloud in minutes.

StreamNative specializes in cloud-native messaging and event streaming solutions. We are the original creators of Apache Pulsar and the developers of the Lakestream architecture, a cloud-native approach that unifies streaming and lakehouse storage.

StreamNative Cloud provides a managed, fully hosted data streaming platform that allows users to easily build streaming applications without managing the underlying infrastructure. StreamNative Cloud takes care of all the operational aspects, including provisioning, monitoring, scaling, and maintenance of the clusters, allowing developers to focus on building their applications and processing data.

<Tip>
  **New to StreamNative?** Check [Choose Kafka or Pulsar](/cloud/overview/choose-kafka-or-pulsar) to decide which protocol fits your workload, or jump straight to [Kafka Service](/kafka/overview) or [Pulsar Service](/cloud/overview/concepts-overview).
</Tip>

## Next

* [StreamNative Cloud Concepts](/cloud/overview/concepts-overview)

## References

* [BYOC², Portable Data Plane, and the vision of making data streaming accessible and affordable](https://streamnative.io/blog/byoc2-portable-data-plane-and-the-vision-of-making-data-streaming-accessible-and-affordable)


# StreamNative Cloud Concepts
Source: https://docs.streamnative.io/cloud/overview/concepts-overview



StreamNative Cloud offers a veriety of resources to help you build and manage messaging and data streaming applications. This page provides an overview about the StreamNative Cloud resources, that you can use to organize your Pulsar clusters and other StreamNative Cloud resources.

## Organizations

The top-level resource in StreamNative Cloud is the organization. An organization is logical grouping of resources that you can use to manage access to your StreamNative Cloud resources and to organize your resources in a way that make sense for your organization. Most users will only need on organization, but you can create multiple organizations if you need to separate resources for different departments or teams.

To learn more about organizations, see [Organizations](/cloud/security/access/resource-hierarchy/organizations).

## Instances

Within each StreamNative Cloud organization, you can have one or more [Instances](https://pulsar.apache.org/docs/concepts-architecture-overview/#clusters).
An Instance represents an environment within a cloud provider that can contain multiple clusters and deployed components, such as Connectors, Functions, and SQL workspaces. Different departments or teams can use separate instances to isolate their resources from each other.

An Instance can be either fully **Hosted** on StreamNative's cloud account or deployed on your public cloud account via the Bring-Your-Own-Cloud (BYOC) option. An instance can contain one or more clusters.

To learn more about instances, see [Instances](/cloud/clusters/manage-instances/instance).

### Clusters

Within each Instance, you can have one or more Clusters. Each cluster is deployed in a cloud region within the cloud provider that the Instance is configured to use. Each cluster exposes different service endpoints that allow client libraries to connect and perform various operations - producing and consuming messages, running functions and connectors, executing SQL queries, managing Flink jobs, and handling other resources.

Clusters within an instance can replicate among themselves using [geo-replication](https://pulsar.apache.org/docs/concepts-replication/).

To learn more about clusters, see [Clusters](/cloud/clusters/manage-clusters/cluster).

## Users

Within each StreamNative Cloud organization, you can invite one or more users. Each user represents the identify of a person who can be authenticated and granted access to StreamNative Cloud resources and Pulsar resources within each Pulsar cluster.

To learn more about users, see [Users](/cloud/security/authentication/user-accounts).

## Service Accounts

Within each StreamNative Cloud organization, you can create one or more service accounts. Each service account represents an application programmatically accessing StreamNative Cloud resources and Pulsar resources within each Pulsar cluster.

A Service Account can be used across multiple Pulsar instances. However, the authentication credentials or API keys are different when they are used in different Pulsar instances.

To learn more about service accounts, see [Service Accounts](/cloud/security/authentication/service-accounts/service-accounts).

## Secrets

Within each StreamNative Cloud organization, you can create one or more secrets to store and manage sensitive data such as passwords, tokens, and private keys. A Secret may contain numerous keys. You can create Secrets and refer to them in connectors and Pulsar Functions. Currently, secrets are shared across multiple Pulsar instances within an organization.

To learn more about secrets, see [Secrets](/cloud/security/secret).

## Interact with the Cloud Resources

StreamNative Cloud gives you two ways to interact with the resources.

### StreamNative Cloud Console

<span />

The StreamNative Cloud console provides a web-based, graphical user interface that you can use to manage your StreamNative Cloud organizations and resources. When you use the StreamNative Cloud console, you either create a new organization or choose an existing organization, and then use the resources that you create in the context of that organization.

### Command-line interface

<span />

If you prefer to work at the command line, you can perform most StreamNative Cloud tasks by using [snctl](/tools/cli/snctl/snctl-overview) to interact with the StreamNative Cloud resources within an organization. The CLI let you manage development workflow and StreamNative Cloud resources in a terminal window.

### Infrastructure-as-Code (IaC) tools

<span />

If you prefer to manage and provision the StreamNative Cloud resources through Infrastructure-as-Code (IaC) tools such as Terraform, you can declare those StreamNative Cloud resources using [Terraform Provider](https://github.com/streamnative/terraform-provider-streamnative). With these tools, you can create configuration files that contain your StreamNative Cloud resource specifications, which make it easier to edit and distribute.

## Interact with StreamNative Clusters

StreamNative Cloud gives you multiple ways to interact with the StreamNative Clusters.

### StreamNative Cloud Console

<span />

In addition to managing StreamNative Cloud organizations and resources, you can also use StreamNative Cloud console to manage resources, configuration, and policies within a StreamNative Cluster.

### Command-line interface

<span />

If you prefer to work at the command line, you can interact with the StreamNative Cluster by using [pulsarctl](/tools/cli/pulsarctl/pulsarctl-overview). The CLIs let you manage the Pulsar resources in a terminal window.

### Infrastructure-as-Code (IaC) tools

<span />

If you prefer to manage and provision the Pulsar resources through Infrastructure-as-Code (IaC) tools such as Terraform and Kubernetes, you can declare those Pulsar resources using [Terraform Provider](https://github.com/streamnative/terraform-provider-pulsar) or [Pulsar Resource Operators](https://github.com/streamnative/pulsar-resources-operator). With these tools, you can create configuration files that contain your Pulsar resource specifications, which make it easier to eidt and distribute.

### Client libraries

StreamNative supports various client libraries that enable you to easily interact with the StreamNative Clusters to produce & consume messages, running connectors & functions, and create & manage resources.

* [Build Applications with Pulsar Clients](/cloud/build/pulsar-clients/qs-connect)
* [Build Applications with Kafka Clients](/cloud/build/kafka-clients/kafka-on-cloud)
* [Connect to External Systems](/cloud/connect/connector-index)
* [Process Data Streams with Pulsar Functions](/cloud/process/pulsar-functions/functions-index)


# Data Streaming Engine
Source: https://docs.streamnative.io/cloud/overview/data-streaming-engine



StreamNative Cloud runs all cluster profiles on the **URSA engine** — a cloud-native data streaming engine at the heart of the [Lakestream architecture](/cloud/overview/lakestream-overview). URSA supports multiple write-ahead log (WAL) implementations and metadata stores, so each cluster profile can be tuned for latency or cost without switching engines.

<Note>
  Every StreamNative Cloud cluster — Pulsar or Kafka, Latency-Optimized or Cost-Optimized — runs on URSA. The profile determines the WAL, metadata store, and exposed protocols. Native Pulsar protocol support on the Cost-Optimized profile is coming after the Apache Pulsar 5.0 release; today, Cost-Optimized Pulsar Clusters expose the Kafka-compatible protocol.
</Note>

## URSA engine

URSA is StreamNative's unified stream storage engine, recognized with the **VLDB 2025 Best Industry Paper** award. It provides the storage layer for Lakestream and delivers:

* **Native Kafka support** for Kafka Clusters.
* **Native Pulsar support** for Pulsar Clusters (on the Latency-Optimized profile today; on Cost-Optimized after Apache Pulsar 5.0).
* **Multiple WAL options** — Apache BookKeeper for Pulsar low-latency, KRaft + local disks for Kafka low-latency, and object storage (Amazon S3, Google Cloud Storage, Azure Blob Storage) for cost-optimized profiles.
* **Multiple metadata stores** — ZooKeeper, Oxia, or KRaft, depending on cluster type and profile.
* **Lakehouse integration** across all profiles, with data available in Iceberg and Delta Lake formats on open object storage.

### Cluster profile configurations

Each cluster profile maps to a specific WAL, metadata store, and protocol surface:

| Profile / Cluster type       | WAL                                                          | Metadata store                       | Protocols                                                                                    | Caveats                                                           |
| ---------------------------- | ------------------------------------------------------------ | ------------------------------------ | -------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- |
| **Latency-Optimized Pulsar** | Apache BookKeeper                                            | ZooKeeper (default); Oxia on request | Pulsar (native); Kafka via [KSN](/kafka/kafka-cluster-vs-ksn) with full Kafka feature parity | —                                                                 |
| **Cost-Optimized Pulsar**    | Object Storage (Amazon S3, Google Cloud Storage, Azure Blob) | Oxia                                 | Kafka-compatible only                                                                        | Native Pulsar protocol coming after the Apache Pulsar 5.0 release |
| **Latency-Optimized Kafka**  | Local disk (KRaft + ISR)                                     | KRaft                                | Kafka (native)                                                                               | —                                                                 |
| **Cost-Optimized Kafka**     | Object Storage (Amazon S3, Google Cloud Storage, Azure Blob) | Oxia                                 | Kafka (native)                                                                               | Kafka transactions and topic compaction coming soon               |

See [Cluster Profiles Overview](/cloud/clusters/cluster-profiles-overview) for deployment-option availability (Serverless, Dedicated, BYOC).

### Classic configuration (legacy naming)

**Classic** is a historical label for the Apache Pulsar deployment configuration that uses ZooKeeper for metadata and Apache BookKeeper for WAL. It is the default configuration for Latency-Optimized Pulsar Clusters today and remains the reference implementation for low-latency Pulsar workloads. In current documentation, this configuration is an option within the URSA engine — not a separate engine.

If you see "Classic Engine" referenced elsewhere in StreamNative materials, it refers to this Latency-Optimized Pulsar configuration (BookKeeper WAL + ZooKeeper metadata). The Kafka protocol is available on this configuration through the KSN protocol handler with full Kafka feature parity.

### Ursa Stream Storage

At the heart of the URSA engine is the concept of **Ursa Stream Storage** — a headless, multi-modal data storage layer built on lakehouse formats.

For Cost-Optimized profiles, Ursa Stream Storage uses a WAL implementation based on object storage. This design writes records directly to object storage services like Amazon S3, bypassing BookKeeper and eliminating the need for inter-broker replication. Brokers are stateless and leaderless, meaning any broker can handle produce or fetch requests for any partition. This eliminates inter-AZ replication traffic and reduces network costs by up to 95%, at the cost of higher end-to-end latency (typically sub-second, tunable down to \~200 ms).

<img alt="Ursa Cost-Optimized Storage" />

For Latency-Optimized profiles, URSA uses disk-based WALs (Apache BookKeeper for Pulsar, KRaft + local disks for Kafka) to deliver single-digit to tens-of-milliseconds end-to-end latency.

## Compare cluster profiles

Use this summary to pick the profile that matches your workload. For the full feature matrix, see [Cluster Profiles Overview](/cloud/clusters/cluster-profiles-overview).

| Feature                  | Latency-Optimized Pulsar                               | Cost-Optimized Pulsar                               | Latency-Optimized Kafka          | Cost-Optimized Kafka                                |
| ------------------------ | ------------------------------------------------------ | --------------------------------------------------- | -------------------------------- | --------------------------------------------------- |
| **Pulsar protocol**      | Yes (native)                                           | Coming after Pulsar 5.0                             | N/A                              | N/A                                                 |
| **Kafka protocol**       | Yes (via KSN, full feature parity)                     | Yes (Kafka-compatible)                              | Yes (native)                     | Yes (native)                                        |
| **Storage backend**      | Local disk (BookKeeper)                                | Object storage                                      | Local disk (KRaft + ISR)         | Object storage                                      |
| **Metadata store**       | ZooKeeper (default), Oxia on request                   | Oxia                                                | KRaft                            | Oxia                                                |
| **End-to-end latency**   | Single-digit to tens of ms                             | Sub-second (tunable to \~200 ms)                    | Single-digit to tens of ms       | Sub-second (tunable to \~200 ms)                    |
| **Inter-AZ replication** | Required                                               | Eliminated (direct to object storage)               | Required                         | Eliminated (direct to object storage)               |
| **Lakehouse storage**    | Built in (Iceberg, Delta)                              | Built in (Iceberg, Delta)                           | Built in (Iceberg, Delta)        | Built in (Iceberg, Delta)                           |
| **Caveats**              | —                                                      | Native Pulsar protocol not yet available            | —                                | Kafka transactions and topic compaction coming soon |
| **Best for**             | Real-time messaging, mission-critical Pulsar workloads | Lakehouse ingestion, analytics, Kafka-API workloads | Real-time Kafka, fraud detection | Event streaming, log aggregation, CDC               |

## Choose the right profile for your workload

* **Latency-Optimized Pulsar** — when you need native Pulsar protocol (flexible subscriptions, multi-tenancy, geo-replication) with sub-10 ms latency.
* **Cost-Optimized Pulsar** — when you need Kafka-compatible access to Pulsar Clusters with object-storage economics, and you can wait for native Pulsar protocol on this profile.
* **Latency-Optimized Kafka** — when you need native Apache Kafka with full feature support at sub-10 ms latency.
* **Cost-Optimized Kafka** — when you need native Apache Kafka at up to 95% lower infrastructure cost, and your workload does not require Kafka transactions or topic compaction yet.


# Kafka and Pulsar Concepts Map
Source: https://docs.streamnative.io/cloud/overview/kafka-pulsar-concepts

Map concepts between Apache Kafka and Apache Pulsar. Understand how topics, consumer groups, and partitions translate between protocols.

If you know Kafka and encounter Pulsar terms in our docs (or vice versa), use this reference to translate between the two protocols.

## Core concepts mapping

The following table maps common Kafka concepts to their Pulsar equivalents. Where a concept exists in one protocol but not the other, the cell is marked with a dash.

| Kafka Concept                      | Pulsar Equivalent        | Notes                                                                                                                                                                   |
| ---------------------------------- | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Topic                              | Topic                    | Same concept. Both protocols use topics as the primary unit for organizing messages.                                                                                    |
| Partition                          | Partition                | Pulsar topics also have partitions. Both protocols use partitions for parallelism and ordering.                                                                         |
| Consumer Group                     | Subscription (Failover)  | A Failover subscription provides one active consumer with automatic failover to standby consumers, similar to how Kafka consumer groups assign partitions to consumers. |
| Offset                             | Message ID / Cursor      | Pulsar uses message IDs for position tracking instead of numeric offsets.                                                                                               |
| —                                  | Tenant                   | Pulsar's top-level isolation boundary. Kafka Service does not use tenants.                                                                                              |
| —                                  | Namespace                | Pulsar's logical grouping under a tenant. Kafka Service does not use namespaces.                                                                                        |
| Bootstrap Server                   | Service URL              | The connection endpoint for the cluster.                                                                                                                                |
| Producer                           | Producer                 | Same concept. Both protocols use producers to publish messages to topics.                                                                                               |
| Consumer                           | Consumer                 | Same concept. Both protocols use consumers to read messages from topics.                                                                                                |
| Kafka Connect                      | Pulsar IO                | Connector frameworks for integrating external systems. StreamNative supports both.                                                                                      |
| Kafka Streams                      | Pulsar Functions         | Stream processing frameworks. Kafka Streams is a client library; Pulsar Functions run server-side.                                                                      |
| StreamNative Kafka Schema Registry | Built-in Schema Registry | StreamNative Kafka Schema Registry is compatible with Confluent Schema Registry.                                                                                        |
| Retention Policy                   | Retention Policy         | Both Kafka and Pulsar have similar retention policy concepts for controlling how long messages are stored.                                                              |

## Key differences

### Multi-tenancy

Pulsar provides built-in multi-tenancy through tenants and namespaces. Tenants serve as the top-level isolation boundary, and namespaces group related topics under a tenant. This hierarchy enables fine-grained access control and resource isolation.

Kafka Service on StreamNative Cloud works with topics directly and does not expose the tenant or namespace layer to users.

### Subscription types

Pulsar offers four subscription types that control how messages are delivered to consumers:

* **Exclusive**: Only one consumer can attach to a subscription.
* **Shared**: Multiple consumers receive messages in round-robin fashion.
* **Failover**: One active consumer with automatic failover to standby consumers (similar to Kafka consumer groups).
* **Key\_Shared**: Messages with the same key are delivered to the same consumer while allowing multiple consumers on a subscription.

Kafka uses consumer groups with partition-based assignment. Each partition is assigned to exactly one consumer within a group. Kafka also supports **queues** as a newer delivery model that allows multiple consumers to share work without partition affinity, similar to Pulsar's shared subscriptions.

### Geo-replication

Pulsar has built-in cross-cluster replication that synchronizes data across clusters in different regions. You configure replication at the namespace or topic level.

For Kafka protocol on StreamNative Cloud, use [Universal Linking](/cloud/universal-linking/unilink-overview) to replicate data across clusters.

### Cluster types

StreamNative Cloud offers two cluster types:

* **Kafka Clusters**: Run native Apache Kafka. Only Kafka clients can connect to Kafka Clusters.
* **Pulsar Clusters**: Run native Apache Pulsar. Both Pulsar clients and Kafka clients (through KSN) can connect to Pulsar Clusters that have KSN enabled.

<Tip>
  On StreamNative Cloud, both cluster types run on the Ursa Engine. The differences are at the API and cluster type level, not the storage level. You can choose the cluster type that best fits your application without affecting durability, performance, or operational overhead.
</Tip>


# Lakestream Architecture
Source: https://docs.streamnative.io/cloud/overview/lakestream-overview

Lakestream is the cloud-native architecture that unifies streaming and lakehouse storage. It powers both StreamNative Kafka Service and Pulsar Service.

Lakestream is the architectural paradigm that unifies real-time data streaming with lakehouse storage. It separates data, metadata, and protocol into three independent layers, enabling both Kafka and Pulsar to run on the same lakehouse-native foundation.

StreamNative's [URSA engine](/cloud/overview/data-streaming-engine) is the implementation of the storage layer of Lakestream. URSA supports multiple WAL options — Apache BookKeeper and local disk (KRaft + ISR) for latency-optimized profiles, object storage for cost-optimized profiles. It was recognized with the **VLDB 2025 Best Industry Paper** award for its novel approach to lakehouse-native stream storage. Read more in the blog post: [Ursa Wins VLDB 2025 Best Industry Paper](https://streamnative.io/blog/ursa-wins-vldb-2025-best-industry-paper-the-first-lakehouse-native-streaming-engine-for-kafka).

## Architecture

Lakestream separates the streaming stack into three independent layers:

<img alt="Lakestream Architecture" />

### Protocol Layer (stateless serving)

Brokers are **stateless and leaderless**. Any broker can handle produce or fetch requests for any partition. There is no leader election, no partition rebalancing, and no broker-to-broker replication. This means:

* Compute scales independently from storage
* Brokers can be added or removed without data migration
* No cross-AZ replication traffic between brokers

### Metadata Layer (catalog)

**Oxia** replaces ZooKeeper as the metadata store. It provides scalable, strongly consistent metadata management without the operational complexity of ZooKeeper clusters. The **Iceberg Catalog** tracks table metadata for lakehouse integration.

### Data Layer (Ursa Stream Storage)

Data writes directly to object storage (S3, GCS, or Azure Blob Storage) using a Write-Ahead Log (WAL) implementation. The storage layer provides the flexibility to choose between local disks for low latency and shared storage (lakehouse storage) for cost-efficiency. This design:

* Supports both disk-based and diskless storage modes
* Stores data in open table formats (Iceberg, Delta Lake)
* Makes every stream simultaneously queryable as a lakehouse table

## Lakestream vs Traditional Architectures

The following diagram shows how streaming architecture has evolved from monolithic designs to the Lakestream paradigm:

<img alt="Streaming Architecture Evolution: From Monolith to Lakestream" />

## Key benefits

<CardGroup>
  <Card title="Leaderless and Stateless" icon="circle-nodes">
    No leader elections, no partition rebalancing, no broker disks. Brokers are stateless and interchangeable.
  </Card>

  <Card title="Up to 95% Cost Reduction" icon="piggy-bank">
    Data writes directly to object storage, eliminating expensive cross-AZ replication between brokers.
  </Card>

  <Card title="Stream-Table Duality" icon="table">
    Every event written to a topic simultaneously exists as a row in an Iceberg or Delta Lake table. Zero-copy, no ETL.
  </Card>

  <Card title="Open Formats" icon="lock-open">
    Data stored in Iceberg and Delta Lake on your object storage. Query with any engine. No vendor lock-in.
  </Card>
</CardGroup>

## Services powered by Lakestream

<Note>
  Native Pulsar protocol on the Cost-Optimized profile is coming after the Apache Pulsar 5.0 release. Today, Pulsar Clusters on the Cost-Optimized profile expose the Kafka-compatible protocol. See [Cluster Profiles Overview](/cloud/clusters/cluster-profiles-overview) for the full capability matrix.
</Note>

Lakestream powers both streaming services on StreamNative Cloud:

<CardGroup>
  <Card title="Kafka Service" icon="chart-line" href="/kafka/overview">
    Native Apache Kafka API. Ideal for event streaming, log aggregation, CDC, and IoT telemetry.
  </Card>

  <Card title="Pulsar Service" icon="wave-pulse" href="/cloud/overview/cloud-overview">
    Apache Pulsar with multi-protocol support. Ideal for messaging, queuing, and multi-tenant workloads.
  </Card>
</CardGroup>

## Learn more

* [Ursa Engine technical details](/cloud/overview/data-streaming-engine) for storage engine architecture, cluster profiles, and feature comparison
* [Choose Kafka or Pulsar](/cloud/overview/choose-kafka-or-pulsar) to decide which protocol fits your workload
* [Ursa: A Lakehouse-Native Data Streaming Engine for Kafka](https://vldb.org/pvldb/volumes/18/paper/Ursa%3A%20A%20Lakehouse-Native%20Data%20Streaming%20Engine%20for%20Kafka) — VLDB 2025 Best Industry Paper


# Built-in UDFs
Source: https://docs.streamnative.io/cloud/process/pfsql/pfsql-built-in-udfs



<Note title="Note">
  This feature is currently in alpha. If you want to try it out or have any questions, [submit a ticket](https://support.streamnative.io/hc/en-us/requests/new) to the support team.
</Note>

## Built-in Common UDFs

### Data Type Conversion Function

The pfSQL `cast` function is used to convert a value from one data type to another. Currently, it supports 6 data types, including `INT32`, `INT64`, `FLOAT`, `DOUBLE`, `STRING`, and `BOOLEAN`.

The syntax of the `cast` function is:

```sql theme={null}
SELECT cast(field1, to=int32) from topic
```

The `cast` function takes two parameters, the first one is the field name, and the second one is the target data type. The target data type is specified by the `to` keyword, and it can be one of the following values:

* "int32"
* "int64"
* "float"
* "double"
* "string"
* "boolean"

The conversion rules to be followed are shown in the following table.

|         | int32                                                                | int64                                                                | float                                                               | double                 | string                    | boolean                                                                                               |
| ------- | -------------------------------------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------- | ---------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------- |
| int32   | No need to cast                                                      | Cast directly                                                        | Cast directly                                                       | Cast directly          | `String.valueOf`          | `!=0 : true, ==0: false`                                                                              |
| int64   | Out of the range of INT32: throw Exception, otherwise cast directly  | No need to cast                                                      | Cast directly                                                       | Cast directly          | `String.valueOf`          | `!=0L : true, ==0L: false`                                                                            |
| float   | Out of the range of INT32: throw Exception, otherwise `Math.round()` | Out of the range of INT64: throw Exception, otherwise `Math.round()` | No need to cast                                                     | Cast directly          | `String.valueOf`          | `!=0.0f : true, ==0.0f: false`                                                                        |
| double  | Out of the range of INT32: throw Exception, otherwise `Math.round()` | Out of the range of INT64: throw Exception, otherwise `Math.round()` | Out of the range of FLOAT: throw Exception, otherwise cast directly | No need to cast        | `String.valueOf`          | `!=0.0 : true, ==0.0: false`                                                                          |
| string  | `Integer.parseInt()`                                                 | `Long.parseLong()`                                                   | `Float.parseFloat()`                                                | `Double.parseDouble()` | No need to cast           | `text.toLowerCase ==”true”` : true, `text.toLowerCase == “false”` : false, otherwise: throw Exception |
| boolean | `bool ? 1 : 0`                                                       | `bool ? 1L : 0L`                                                     | `bool ? 1.0f : 0.0f`                                                | `bool ? 1.0 : 0.0`     | `bool ? "true" : "false"` | No need to cast                                                                                       |

### Const Value Function

The pfSQL `const` function is used to return a constant value. The syntax of the `const` function is:

```sql theme={null}
SELECT const(value=val, type=typ) from topic
```

The `const` function takes two parameters, the first one is the constant value, and the second one is the data type of the constant value. The data type is specified by the `type` keyword, and it can be one of the following values:

* "int32"
* "int64"
* "float"
* "double"
* "string"
* "boolean"

The following example shows how to use the `const` function:

```sql theme={null}
SELECT const(value=1, type=int32) from topic
```

The above SQL statement returns a constant value of type `int32` with value `1`.

## Built-in Math UDFs

Currently, pfSQL supports the following mathmatical functions. The behavior of these mathmatical functions is identical to the behavior of the corresponding functions in the Java Math standard library.

| Function Name | Allowed input schema types     | Output schema type     | attributes parameter | Corresponding Java Math function                                      |
| ------------- | ------------------------------ | ---------------------- | -------------------- | --------------------------------------------------------------------- |
| sin           | INT32 / INT64 / FLOAT / DOUBLE | DOUBLE                 | -                    | `Math#sin(double)`                                                    |
| cos           | INT32 / INT64 / FLOAT / DOUBLE | DOUBLE                 | -                    | `Math#cos(double)`                                                    |
| tan           | INT32 / INT64 / FLOAT / DOUBLE | DOUBLE                 | -                    | `Math#tan(double)`                                                    |
| asin          | INT32 / INT64 / FLOAT / DOUBLE | DOUBLE                 | -                    | `Math#asin(double)`                                                   |
| acos          | INT32 / INT64 / FLOAT / DOUBLE | DOUBLE                 | -                    | `Math#acos(double)`                                                   |
| atan          | INT32 / INT64 / FLOAT / DOUBLE | DOUBLE                 | -                    | `Math#atan(double)`                                                   |
| sinh          | INT32 / INT64 / FLOAT / DOUBLE | DOUBLE                 | -                    | `Math#sinh(double)`                                                   |
| cosh          | INT32 / INT64 / FLOAT / DOUBLE | DOUBLE                 | -                    | `Math#cosh(double)`                                                   |
| tanh          | INT32 / INT64 / FLOAT / DOUBLE | DOUBLE                 | -                    | `Math#tanh(double)`                                                   |
| degrees       | INT32 / INT64 / FLOAT / DOUBLE | DOUBLE                 | -                    | `Math#toDegrees(double)`                                              |
| radians       | INT32 / INT64 / FLOAT / DOUBLE | DOUBLE                 | -                    | `Math#toRadians(double)`                                              |
| abs           | INT32 / INT64 / FLOAT / DOUBLE | Same type as the input | -                    | `Math#abs(int) / Math#abs(long) / Math#abs(float) / Math#abs(double)` |
| ceil          | INT32 / INT64 / FLOAT / DOUBLE | DOUBLE                 | -                    | `Math#ceil(double)`                                                   |
| floor         | INT32 / INT64 / FLOAT / DOUBLE | DOUBLE                 | -                    | `Math#floor(double)`                                                  |
| exp           | INT32 / INT64 / FLOAT / DOUBLE | DOUBLE                 | -                    | `Math#exp(double)`                                                    |
| ln            | INT32 / INT64 / FLOAT / DOUBLE | DOUBLE                 | -                    | `Math#log(double)`                                                    |
| log10         | INT32 / INT64 / FLOAT / DOUBLE | DOUBLE                 | -                    | `Math#log10(double)`                                                  |
| sqrt          | INT32 / INT64 / FLOAT / DOUBLE | DOUBLE                 | -                    | `Math#sqrt(double)`                                                   |
| sign          | INT32 / INT64 / FLOAT / DOUBLE | DOUBLE                 | -                    | `Math#signum(double)`                                                 |

The syntax of the mathmatical functions is:

```sql theme={null}
SELECT field1, sin(field2), cos(field3), tan(field4) as tan from topic
```

## Limitations

1. The UDF only support in SELECT statement, we are working on it to extend the UDF to WHERE and other statements.
2. The UDF only support passing full path of single field in SELECT statement yet.


# Get started with pfSQL
Source: https://docs.streamnative.io/cloud/process/pfsql/pfsql-get-started



<Note title="Note">
  This feature is currently in alpha. If you want to try it out or have any questions, [submit a ticket](https://support.streamnative.io/hc/en-us/requests/new) to the support team.
</Note>

This tutorial walks you through an example of using SQL queries to create a source connector in Pulsar, covering filtering, routing, and transformation operations on the generated data.

## Step1: Create a source connector

Assume you want to create a data generator source connector named “pipeline-source” that outputs data to the “pipeline-datagen-source” topic with a message-sending rate of 500ms.

<Tabs>
  <Tab title="StreamNative Cloud Console">
    Enter the following SQL when [creating a query](/cloud/process/pfsql/pfsql-work-with-cloud-console#create-a-query).

    ```sql theme={null}
    CREATE SOURCE `pipeline-source` FROM `data-generator` OUTPUT `pipeline-datagen-source` WITH("configs/sleepBetweenMessages"='500')
    ```
  </Tab>

  <Tab title="pfSQL CLI">
    Run the command to submit this filtering query to the pfSQL gateway.

    ```bash theme={null}
    pfsql query run --query "CREATE SOURCE \`pipeline-source\` FROM \`data-generator\` OUTPUT \`pipeline-datagen-source\` WITH(\"configs/sleepBetweenMessages\"='500')"
    ```

    If you want to submit the query to a specific tenant and namespace rather than the default one, you can use `pfsql run query --query '$query' -p tenant=$tenant -p namespace=$namespace` to pass the information via CLI properties.

    <Note title="Note">
      Use a backslash `(\)` to escape single quotes or double quotes. For more information, see [String identifiers](/cloud/process/pfsql/pfsql-understand#string-identifiers).
    </Note>
  </Tab>
</Tabs>

## Step2: Use queries to process messages

After creating a source connector, you can select either of the following ways to continue processing these messages.

* Filter messages
* Route messages
* Transform messages

### Filter messages

To filter the data from the “pipeline-datagen-source” topic and inserts it into another topic according to specific age criteria (greater than or equal to 18), you can do the following:

<Tabs>
  <Tab title="StreamNative Cloud Console">
    Enter the following SQL when [creating a query](/cloud/process/pfsql/pfsql-work-with-cloud-console#create-a-query).

    ```sql theme={null}
    INSERT INTO `pipeline-datagen-age-filter` SELECT * FROM `pipeline-datagen-source` WHERE `age` >= 18
    ```
  </Tab>

  <Tab title="pfSQL CLI">
    Run the command to submit this filtering query to the pfSQL gateway.

    ```bash theme={null}
    pfsql query run --query \
          "INSERT INTO \`pipeline-datagen-age-filter\` \
           SELECT * FROM \`pipeline-datagen-source\` \
           WHERE \`age\` >= 18"
    ```

    <Note title="Note">
      You can use the `--preview` flag to preview the query result (for Avro and JSON schemas only) and push it to the console.
    </Note>

    Once the filtering query is submitted, you will get a query ID with the `pfsql-pfr` prefix. With the query ID, you can get the status and stats of the query with `pfsql query status ${queryId}` and `pfsql query stats ${queryId}`.

    You may also use `pfsql query preview ${queryId}` to preview the query result (for Avro and JSON schemas only) and push it to the console.
  </Tab>
</Tabs>

After the query is deployed, you can see the filtered data in the “pipeline-datagen-age-filter” topic.

### Route messages

To route the messages from the “pipeline-datagen-source” topic to multiple topics according to specific age criteria, you can implement the following logic:
If the `age` field is less than 18, the messages go to the “pipeline-datagen-age-routing-1” topic.
If it is between 18 and 60 (exclusive), the messages go to the “pipeline-datagen-age-routing-2” topic.
Otherwise, the messages are routed to the “pipeline-datagen-age-routing-3” topic.

<Tabs>
  <Tab title="StreamNative Cloud Console">
    Enter the following SQL when [creating a query](/cloud/process/pfsql/pfsql-work-with-cloud-console#create-a-query).

    ```sql theme={null}
    INSERT MULTI IF `age` < 18 THEN INTO `pipeline-datagen-age-routing-1`, IF `age` >= 18 AND `age` < 60 THEN INTO `pipeline-datagen-age-routing-2`, ELSE INTO `pipeline-datagen-age-routing-3` SELECT * FROM `pipeline-datagen-source`
    ```
  </Tab>

  <Tab title="pfSQL CLI">
    Run the command to submit this filtering query to the pfSQL gateway.

    ```bash theme={null}
    pfsql query run --query \
      "INSERT MULTI \
             IF \`age\` < 18 THEN INTO \`pipeline-datagen-age-routing-1\`, \
             IF \`age\` >= 18 AND \`age\` < 60 THEN INTO\`pipeline-datagen-age-routing-2\`, \
            ELSE INTO \`pipeline-datagen-age-routing-3\` \
      SELECT * FROM \`pipeline-datagen-source\`"

    ```

    <Note title="Note">
      You can use the `--preview` flag to preview the query result (for Avro and JSON schemas only) and push it to the console.
    </Note>

    Once the filtering query is submitted, you will get a query ID with the `pfsql-pfr` prefix. With the query ID, you can get the status and stats of the query with `pfsql query status ${queryId}` and `pfsql query stats ${queryId}`.

    You may also use `pfsql query preview ${queryId}` to preview the query result (for Avro and JSON schemas only) and push it to the console.
  </Tab>
</Tabs>

After the query is deployed, you can see the routed data in the “pipeline-datagen-age-routing-1”, “pipeline-datagen-age-routing-2”, and “pipeline-datagen-age-routing-3” topics.

### Transform messages

To extract a field from the “pipeline-datagen-source” topic and insert the field into another topic according to specific age criteria, you can do the following:

<Tabs>
  <Tab title="StreamNative Cloud Console">
    Enter the following SQL when [creating a query](/cloud/process/pfsql/pfsql-work-with-cloud-console#create-a-query).

    ```sql theme={null}
    INSERT INTO `pipeline-datagen-age-select` SELECT `age` FROM `pipeline-datagen-source` WHERE `age` >= 18
    ```
  </Tab>

  <Tab title="pfSQL CLI">
    Run the command to submit this filtering query to the pfSQL gateway.

    ```bash theme={null}
    pfsql query run --query "INSERT INTO \`pipeline-datagen-age-select\` SELECT \`age\` FROM \`pipeline-datagen-source\` WHERE \`age\` >= 18"
    ```

    <Note title="Note">
      You can use the `--preview` flag to preview the query result (for Avro and JSON schemas only) and push it to the console.
    </Note>

    Once the filtering query is submitted, you will get a query ID with the `pfsql-pfr` prefix. With the query ID, you can get the status and stats of the query with `pfsql query status ${queryId}` and `pfsql query stats ${queryId}`.

    You may also use `pfsql query preview ${queryId}` to preview the query result (for Avro and JSON schemas only) and push it to the console.
  </Tab>
</Tabs>

After the query is deployed, you can see the transformed data in the “pipeline-datagen-age-select” topic.

## Step3: Access query results

You can use the ‘pulsar-client’ tool and run the following command to subscribe to the output topic and receive the processed messages.

For example, to access the query results after transforming the messages, you can run the following command:

```bash theme={null}
pulsar-client consume -s my-subscription -p Earliest -n 0 -t persistent://public/default/pipeline-datagen-age-select
```

You can also use the `pfsql query preview` command to preview the query result (for Avro and JSON schemas only) and push it to the console.

## Step4: Clean up queries

After running the SQL queries, you can delete queries that you don’t need anymore.

<Tabs>
  <Tab title="StreamNative Cloud Console">
    Please refer to [Delete a query](/cloud/process/pfsql/pfsql-work-with-cloud-console#delete-a-query).
  </Tab>

  <Tab title="pfSQL CLI">
    1. Run the `pfsql query list` command to get a list of all queries that have been executed.
    2. Copy the returned query ID of each query you want to delete.
    3. Run the `pfsql query delete ${queryId}` command to delete each query one by one. Replace `${queryId}` with the query ID that you copied in step 2.

    For example, if the query IDs are `pfsql-pfr-1k1u04hs8d5k5-e1023304` and `pfsql-pfr-1h7tpcxofd3kz-ef81fe6c`, you can run the following commands to delete them:

    ```bash theme={null}
    pfsql query delete pfsql-pfr-1k1u04hs8d5k5-e1023304
    pfsql query delete pfsql-pfr-1h7tpcxofd3kz-ef81fe6c
    ```
  </Tab>
</Tabs>

## What’s next?

* [Understand pfSQL](/cloud/process/pfsql/pfsql-understand)
* [Built-in UDFs](/cloud/process/pfsql/pfsql-built-in-udfs)


# pfSQL (Alpha) Overview
Source: https://docs.streamnative.io/cloud/process/pfsql/pfsql-overview



<Note title="Note">
  This feature is currently in Private Preview. If you want to try it out or have any questions, [submit a ticket](https://support.streamnative.io/hc/en-us/requests/new) to the support team.
</Note>

## What is pfSQL?

Built on top of [Pulsar functions](/cloud/process/pulsar-functions/functions-overview), pfSQL is a lightweight SQL-like tool that simplifies real-time data processing on StreamNative Cloud.

Unlike standard SQL, pfSQL queries streaming data, meaning that data is constantly flowing through them, either being read, or transformed and redirected. With pfSQL, you can set up a processing pipeline by writing queries with custom processing logic, where several operations like filtering, routing, and field projection can be chained and performed sequentially.

pfSQL allows you to query the following fields of a message with

* custom const values in string, number, or boolean format.
* message key (if any)
* message property
* message payload

## What can pfSQL do?

### Route traffic to different topics

Routing traffic to specific destinations for further analysis can be used for analytics, reporting, and lightweight application development. With pfSQL, you can route the query results to one or multiple Pulsar topics based on keys, properties, or fields.

### Filter traffic to save bandwidth

Select a subset of messages that meet specific criteria. For example, filtering based on certain key, property, and field values or exact field matches.

### Project payload fields for privacy purposes

Build a new message by transforming the format of a message, for example, extracting specific fields, and assembling and renaming them in the output result for downstream consumption.

## What’s next?

* [Work with Cloud Console](/cloud/process/pfsql/pfsql-work-with-cloud-console)
* [Work with pfSQL CLI](/cloud/process/pfsql/pfsql-work-with-cli)
* [Get started with pfSQL](/cloud/process/pfsql/pfsql-get-started)
* [Understand pfSQL](/cloud/process/pfsql/pfsql-understand)
* [Built-in UDFs](/cloud/process/pfsql/pfsql-built-in-udfs)


# Understand pfSQL
Source: https://docs.streamnative.io/cloud/process/pfsql/pfsql-understand



<Note title="Note">
  This feature is currently in alpha. If you want to try it out or have any questions, [submit a ticket](https://support.streamnative.io/hc/en-us/requests/new) to the support team.
</Note>

This section walks you through the basic pfSQL elements, syntax and typical examples.

## pfSQL identifiers

pfSQL identifiers are used to reference objects, such as Pulsar topics, structural data fields, functions, or connectors.

pfSQL supports the following three types of identifiers.

### Unquoted identifiers

Unquoted identifiers must begin with a letter or underscore (\_) and cannot contain extended characters or blank spaces.

Some examples of valid unquoted identifiers include:

* `customer`
* `order_details`
* `_id`
* `email_address`

For example, in a SQL query, you might reference a column using an unquoted identifier like this:

```sql theme={null}
INSERT INTO `output-topic` SELECT customer_name, order_date FROM `input-topic`;
```

In this example, `customer_name` and `order_date` are both unquoted identifiers that refer to column names.

<Note title="Note">
  Don’t use reserved keywords (such as SELECT, INSERT, and so on) to name columns or tables. Instead, you can use backquotes (\`) to enclose reserved keywords or use a different name for your object.
</Note>

### Backquoted Identifiers

Backquoted identifiers are case-sensitive and can start with or contain any valid characters, including:

* Numbers
* Special characters (., ', !, @, #, \$, %, ^, &, \*, and so on)
* Extended ASCII and non-ASCII characters
* Blank spaces
* Reserved keywords

It is highly recommended to use backquoted identifiers in the whole query, including the reference of Pulsar topics, structural data fields, and so on.

### String identifiers

Unlike unquoted and backquoted identifiers, string identifiers are string-type consts, such as text and dates.

String identifiers can contain any combination of characters, including whitespace, punctuation, and special characters.

In general, each string identifier must be enclosed by double quotes (“) or single quotes ('). For example, 'foo' in the following query is a string that represents the text “foo”.

```sql theme={null}
INSERT INTO `output-topic` SELECT customer_name, order_date FROM `input-topic` WHERE customer_name = 'foo';
```

If you want to include a single quote or double quote within the string itself, you need to escape it using a backslash ().

The following is an example of using a backslash to escape a single quote.

```sql theme={null}
INSERT INTO `output-topic` SELECT customer_name, order_date FROM `input-topic` WHERE description = “Apple\’s product”;
```

## Logical/Boolean operators

The following table outlines the SQL logical/boolean operators that pfSQL supports.

| Operator | Description                             | Example                                         |
| -------- | --------------------------------------- | ----------------------------------------------- |
| AND      | Returns TRUE if both operands are TRUE. | `KEY = ‘key1’ AND PROPERTIES[ROUTING] = ‘true’` |
| OR       | Returns TRUE if either operand is TRUE. | `KEY = ‘key1’ OR KEY = ‘key2’`                  |
| NOT      | Returns TRUE if the operand is FALSE.   | `NOT KEY = ‘key1’`                              |

## Comparison operators

The following table outlines the SQL comparison operators that pfSQL supports.

| Operator | Description                                                                                         | Example  | Result |
| -------- | --------------------------------------------------------------------------------------------------- | -------- | ------ |
| `=`      | Checks if the values of two operands are equal or not.                                              | `1 = 2`  | FALSE  |
| `!=`     | Checks if the values of two operands are equal or not.                                              | `1 != 2` | TRUE   |
| `>`      | Checks if the value of the left operand is greater than the value of the right operand.             | `1 > 2`  | FALSE  |
| `<`      | Checks if the value of the left operand is less than the value of the right operand.                | `1 < 2`  | TRUE   |
| `>=`     | Checks if the value of the left operand is greater than or equal to the value of the right operand. | `1 >= 2` | FALSE  |
| `<=`     | Checks if the value of the left operand is less than or equal to the value of the right operand.    | `1 <= 2` | TRUE   |

## Arithmetic operators

### Unary operators

Unary arithmetic operators are used to perform arithmetic operations on a single operand. The unary arithmetic operators are:

* `+` (unary plus)
* `-` (unary minus)

The unary plus operator does not change the sign of the operand. It is included for completeness.

The unary minus operator changes the sign of the operand.

Supported input data types: `INT32`, `INT64`, `FLOAT`, `DOUBLE`

Output data type is the same as the input data type.

### Binary operators

Binary arithmetic operators are used to perform arithmetic operations on two operands. The binary arithmetic operators are:

* `+` (addition)
* `-` (subtraction)
* `*` (multiplication)
* `/` (division)
* `%` (modulo)

Supported input data types: `INT32`, `INT64`, `FLOAT`, `DOUBLE`

Output data type is following the precedence rules:

* If both operands are `INT32`, the output data type is `INT32`.
* If both operands are `INT64`, the output data type is `INT64`.
* If both operands are `FLOAT`, the output data type is `FLOAT`.
* If both operands are `DOUBLE`, the output data type is `DOUBLE`.
* If one operand is `INT32` and the other operand is `INT64`, the output data type is `INT64`.
* If one operand is `INT32` and the other operand is `FLOAT`, the output data type is `FLOAT`.
* If one operand is `INT32` and the other operand is `DOUBLE`, the output data type is `DOUBLE`.
* If one operand is `INT64` and the other operand is `FLOAT`, the output data type is `FLOAT`.
* If one operand is `INT64` and the other operand is `DOUBLE`, the output data type is `DOUBLE`.
* If one operand is `FLOAT` and the other operand is `DOUBLE`, the output data type is `DOUBLE`.

The modulo operator is only supported for `INT32` and `INT64` operands. The output data type is the same as the input data type.

## pfSQL statements

The basic statements of pfSQL include `SELECT`, `INSERT INTO`, and `INSERT MULTI`. You can use a combination of them for routing, filtering, and projection purposes.

<Note title="Note">
  The statements are case-insensitive. It’s highly recommended to use upper cases for reserved keywords, such as SELECT, INSERT, and so on.
</Note>

### SELECT

The `SELECT` statement is used to select data from a topic. It is the key statement of a pfSQL query to define the message scope and processing conditions.

**Syntax scheme**

```sql theme={null}
SELECT `selectItem0`[, `selectItem1`, ...]
    FROM `fromTopic0`[, `fromTopic1`, ...]
    [WHERE {query_where}]
    [OPTIONS {query_options}]
```

<Note title="Note">
  1. Both `WHERE` clause and `OPTION` are optional.
  2. `WHERE` clause allows you to define a set of conditions that the data needs to match in order to be returned. If you don’t want to add any conditions to limit your query results, you can skip it. For more information about the supported operators, see Logical/Boolean operators.
</Note>

**Example**

The following example represents selecting the qualified messages from the input topic. Any messages that do not satisfy the two conditions are excluded.

```sql theme={null}
SELECT * FROM `input_topic` WHERE `field0` = "1" AND `field1` = "demo"
```

#### OPTIONS

On top of the SELECT statement, you can use `OPTIONS` for specific use cases.

##### KeyValue Schema Support

KeyValue is a schema type introduced by the Apache Pulsar, and for more details, please refer to [https://pulsar.apache.org/docs/3.1.x/schema-understand/#keyvalue-schema](https://pulsar.apache.org/docs/3.1.x/schema-understand/#keyvalue-schema).

When using pfSQL to query data in KeyValue schema, you need to use either `OPTIONS UNWRAP KEY` or `OPTIONS UNWRAP VALUE` to unwrap the KeyValue schema, or `OPTIONS MERGE` to merge the KeyValue schema into a single message.

For example, if you have a topic with KeyValue schema, and you want to query the key and value separately, you can use the following SQL:

```sql theme={null}
SELECT * FROM `input` OPTIONS UNWRAP KEY;

SELECT * FROM `input` OPTIONS UNWRAP VALUE;
```

<Note title="Note">
  With `UNWRAP KEY`, the SELECT statement only affects the message keys. Take the `KeyValue<AVRO, AVRO>` message for example, `SELECT * FROM \`topic\` OPTIONS UNWRAP KEY`only selects the message key and ignores the message value. Similarly,`UNWRAP VALUE\` takes the message value as the output message and ignores the key.
</Note>

##### Passing Pulsar Function Configs

You can pass Pulsar Function configs to pfSQL by using `OPTIONS`. Available configs are limited to the following:

* `processingGuarantees`
* `cleanupSubscription`
* `subscriptionPosition`
* `deadLetterTopic`
* `maxMessageRetries`
* `retainOrdering`
* `retainKeyOrdering`
* `subName`

For example, if you want to pass `subscriptionPosition`, `subName` and `cleanupSubscription` to pfSQL, you can use the following SQL:

```sql theme={null}
SELECT * FROM `input` OPTIONS ('subscriptionPosition'=Latest, 'cleanupSubscription'=true, 'subName'='test-sub-name');
```

##### Using them together

You can use all the above features together. For example, if you want to query data from a topic with KeyValue schema, and you want to pass some configs as well, you can use the following SQL:

```sql theme={null}
SELECT * FROM `input` OPTIONS ('subscriptionPosition'=Latest, 'cleanupSubscription'=true, 'subName'='test-sub-name') UNWRAP KEY;
```

### INSERT INTO

`INSERT INTO` routes messages from one topic to another, followed by a `SELECT` statement to specify the data source with possible filtering/transformation conditions.

**Syntax scheme**

```sql theme={null}
INSERT INTO `topic_name`
SELECT {statement}
```

**Example**

The following example represents routing all messages from the input topic to the output topic.

```sql theme={null}
INSERT INTO `output_topic`
SELECT * FROM `input_topic`
```

### INSERT MULTI

`INSERT MULTI` routes messages from a topic to multiple ones based on specific criteria, followed by a `SELECT` statement to specify the data source with possible filtering/transformation conditions.

**Syntax scheme**

```sql theme={null}
INSERT MULTI
IF {when_condition} THEN INTO `topic_name`,
...
ELSE INTO `topic_name`
SELECT {statement}
```

**Example**

The following example represents routing qualified messages from the input topic to three output topics based on the value of the key and the routing property.

```sql theme={null}
INSERT MULTI
IF KEY="us" THEN INTO `output_topic0`,
IF KEY="cn" THEN INTO `output_topic1`,
ELSE INTO `output_topic2`
SELECT * FROM `input_topic` WHERE KEY = "us" OR KEY = "cn" AND PROPERTIES[ROUTING] = "true"
```

### CREATE SOURCE

To feed data from external systems into your Pulsar cluster, you can use the `CREATE SOURCE` statement to create a Pulsar source connector.

<Note title="Note">
  You may still using `snctl`, `pulsar-admin`, `pulsarctl`, `terraform`, or `Cloud Console` to manage source connectors, `CREATE SOURCE` is just another way to create source connectors.
</Note>

**Syntax scheme**

```sql theme={null}
CREATE SOURCE `source_name` [("source_property_name"="source_property_value", ...)]
    FROM ( "source_type" | "source_package_url" )
    OUTPUT "output_topic_name"
```

**Example**

```sql theme={null}
CREATE SOURCE `source-connector`("configs/sleepBetweenMessages"="500") FROM `data-generator` OUTPUT `topic0`
```

### CREATE SINK

To feed data from your Pulsar cluster into external systems, use the `CREATE SINK` statement to create a Pulsa sink connector.

<Note title="Note">
  You may still using `pulsar-admin`, `pulsarctl`, `terraform`, or `Cloud Console` to manage sink connectors, `CREATE SINK` is just another way to create sink connectors.
</Note>

**Syntax scheme**

```sql theme={null}
CREATE SINK `sink_name`[("sink_property_name"="sink_property_value", ...)]
    FROM ( `sink_type` | `sink_package_url` )
    INPUT `input_topic`, ...
```

**Example**

```sql theme={null}
CREATE SINK `sink-connector`("topic_to_serde_className/input1"="serde1", "topictoserdeclassName/input2"="serde2") FROM `data-generator` INPUT `topic0`
```

### Pulsar Schema in pfSQL

pfSQL loads the schema of the input record and creates an output record with the same schema type. For example, if the input topic is in the AVRO schema, then the output topic is in the AVRO schema as well.

When using the `SELECT` statement to transform messages, the generated messages use the same schema type as the input topic but with new schema info based on the transformation rules.

pfSQL currently supports the schema types of AVRO, JSON, and Key/Value when accessing the payload fields. Protobuf and Protobuf Native is not supported yet.

## What’s next

* [Built-in UDFs](/cloud/process/pfsql/pfsql-built-in-udfs)


# Work with pfSQL CLI
Source: https://docs.streamnative.io/cloud/process/pfsql/pfsql-work-with-cli



<Note title="Note">
  This feature is currently in alpha. If you want to try it out or have any questions, [submit a ticket](https://support.streamnative.io/hc/en-us/requests/new) to the support team.
</Note>

pfSQL CLI is a command line tool developed in Rust to perform pfSQL queries, interacting with pfSQL endpoints. This section walks you through the steps to set up pfSQL CLI via Homebrew or Docker.

## Prerequisites

1. To fully use the feature provided by pfSQL, you will need to follow [Set up your environment](/cloud/process/pulsar-functions/function-setup) to set up your Pulsar Functions environment first.
2. Get the pfSQL gateway’s service URL on StreamNative Cloud Console. It shares the same service URL as your Pulsar cluster.
3. Get your service account's token or OAuth2 credential file, see [Get the service account key file or token](/cloud/security/authentication/service-accounts/service-accounts#get-the-service-account-key-file-or-token) for more details.

## Setup pfSQL CLI via Homebrew

### Install pfSQL CLI

```bash theme={null}
brew tap streamnative/streamnative
brew install pfsql-cli
```

### Connect to pfSQL gateway

After installing pfSQL CLI via Homebrew, you can pass the pfSQL connection configs to `pfsql` either through the environment variables or through the command lines.

<Note title="Note">
  1. The configurations passed through command lines can overwrite those passed through environment variables. In other words, if you pass the configurations through both options, those passed through command lines are used.
  2. If you connect to your pfSQL gateway through command lines, you need to assemble the command for connecting to the pfSQL gateway into each of your pfSQL commands.
</Note>

#### Option1: Connect through environment variables

To pass the connection configs through the environment variables, run the following command based on the authentication provider you use.

<Tabs>
  <Tab title="OAuth2">
    ```bash theme={null}
    export PFSQL_BACKEND="${serviceUrl}"
    export PFSQL_AUTH_PROVIDER="oauth2"
    export PFSQL_AUTH_PARAMETERS="{\"credentials_url\":\"file://$OAUTH2_FILE_PATH\", \"issuer_url\":\"$ISSUER_URL\", \"audience\":\"$AUDIENCE\"}"
    pfsql info
    ```

    * `PFSQL_BACKEND`: the HTTP service URL of your Pulsar Cluster.
    * `PFSQL_AUTH_PROVIDER`: use "oauth2" as the authentication type.
    * `PFSQL_AUTH_PARAMETERS`:
      * `credentials_url`: the path to your downloaded OAuth2 credential file. It supports the following pattern formats:
        * `file://path/to/file`
        * `data:application/json;base64,<base64-encoded value>`
      * `issuer_url`: the URL of your OAuth2 authentication provider. You can get the value from your downloaded OAuth2 credential file.
      * `audience`: the Uniform Resource Name, which is a combination of the `urn:sn:pulsar`, the organization name, and the Pulsar instance name, in this format `urn:sn:pulsar:<org_name>:<instance_name>`.

    Output of `pfsql info` would be like:

    ```json theme={null}
    {
      "status": "RUNNING",
      "runtimes": {
        "pulsar_function": "RUNNING"
      },
      "version": "v0.18.0",
      "whoAmI": "test@streamnative.dev"
    }
    ```
  </Tab>

  <Tab title="Token">
    ```bash theme={null}
    export PFSQL_BACKEND="${serviceUrl}"
    export PFSQL_AUTH_PROVIDER="jwt"
    export PFSQL_AUTH_PARAMETERS="${APIKEY}"
    pfsql info
    ```

    * `PFSQL_BACKEND`: the HTTP service URL of your Pulsar Cluster.
    * `PFSQL_AUTH_PROVIDER`: use "jwt" as the authentication type.
    * `PFSQL_AUTH_PARAMETERS`: a API Key generated from StreamNative Cloud Console with a selected service account.
  </Tab>
</Tabs>

#### Option2: Connect through CLI options

To pass the connection configs through command line options, run the following command based on the authentication provider you use.

<Tabs>
  <Tab title="OAuth2">
    ```bash theme={null}
    pfsql -b ${serviceUrl} --auth-provider oauth2 --auth-parameters '{"credentials_url":"file://PATH", "issuer_url":"issuer_url", "audience":"urn:sn:pulsar:org:instance"}' info
    ```

    * `-b`: the HTTP service URL of your Pulsar Cluster.
    * `--auth-provider`: use "oauth2" as the authentication type.
    * `--auth-parameters`: the parameters for OAuth2 authentication. It supports the following pattern formats:
      * `credentials_url`: the path to your downloaded OAuth2 credential file. It supports the following pattern formats:
        * `file://path/to/file`
        * `data:application/json;base64,<base64-encoded value>`
      * `issuer_url`: the URL of your OAuth2 authentication provider. You can get the value from your downloaded OAuth2 credential file.
      * `audience`: the Uniform Resource Name, which is a combination of the `urn:sn:pulsar`, the organization name, and the Pulsar instance name, in this format `urn:sn:pulsar:<org_name>:<instance_name>`.

    Output of `pfsql info` would be like:

    ```json theme={null}
    {
      "status": "RUNNING",
      "runtimes": {
        "pulsar_function": "RUNNING"
      },
      "version": "v0.18.0",
      "whoAmI": "test@streamnative.dev"
    }
    ```
  </Tab>

  <Tab title="Token">
    ```bash theme={null}
    pfsql -b ${serviceUrl} --auth-provider jwt --auth-parameters "${APIKEY}" info
    ```

    * `-b`: the HTTP service URL of your Pulsar Cluster.
    * `--auth-provider`: use "jwt" as the authentication type.
    * `--auth-parameters`: an API Key generated from StreamNative Cloud Console with a selected service account.
  </Tab>
</Tabs>

## Setup pfSQL CLI via Docker

### Install pfSQL CLI via Docker

```bash theme={null}
docker pull docker.cloudsmith.io/streamnative/pfsql/pfsql-cli:0.18.0
```

### Connect to pfSQL gateway

Connect to the pfSQL gateway by passing the connection configs to your docker container. The `pfsql` execuable is located as `/pfsql` in the `pfsql-cli` docker image.

<Tabs>
  <Tab title="OAuth2">
    ```bash theme={null}
    docker run -it –rm -e PFSQL_BACKEND="${serviceUrl}" -e PFSQL_AUTH_PROVIDER=oauth2 -e PFSQL_AUTH_PARAMETERS='{"credentials_url":"file:///tmp/credentials.json", "issuer_url":"issuer_url", "audience":"urn:sn:pulsar:org:instance"}' -v ${CREDENTIL_PATH}:/tmp/credentials.json docker.cloudsmith.io/streamnative/pfsql/pfsql-cli:0.18.0 bash
    # after the container running, and you could call `/pfsql` from the container’s terminal
    /pfsql info
    ```
  </Tab>

  <Tab title="Token">
    ```bash theme={null}
    docker run -it –rm -e PFSQL_BACKEND="${serviceUrl}" -e PFSQL_AUTH_PROVIDER=jwt -e PFSQL_AUTH_PARAMETERS=”${APIKEY}” docker.cloudsmith.io/streamnative/pfsql/pfsql-cli:0.18.0 bash
    # after the container running, and you could call `/pfsql` from the container’s terminal
    /pfsql info
    ```
  </Tab>
</Tabs>

For more information about the parameters, see [Setup pfSQL CLI via Homebrew](/cloud/process/pfsql/pfsql-work-with-cli#setup-pfsql-cli-via-homebrew).

## pfSQL CLI Referneces

### Usage

```bash theme={null}
pfsql [OPTIONS] -b <BACKEND_URL> <SUBCOMMAND>
```

#### Options

The following table outlines the options you can use with pfSQL.

| Option              | Required | Description                                                                                               |
| ------------------- | -------- | --------------------------------------------------------------------------------------------------------- |
| `-b`                | Yes      | Specify the pfSQL gateway service URL. You can use the value of the `PFSQL_BACKEND` environment variable. |
| `--auth-provider`   | No       | Specify the authentication provider. Available values are `jwt` and `oauth2`.                             |
| `--auth-parameters` | No       | Specify the authentication parameters.                                                                    |
| `-v`                | No       | Display more output per occurrence.                                                                       |
| `-q`                | No       | Display less output per occurrence.                                                                       |
| `-h`                | No       | Print help information.                                                                                   |
| `-V`                | No       | Print version information.                                                                                |

#### Subcommands

The following table outlines the subcommands you can use with pfSQL.

| Subcommand     | Description                                                                                                                                                                                           |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `info`         | Get the pfSQL gateway status and version.                                                                                                                                                             |
| `query`        | Manage pfSQL queries. You can use it to execute a pfSQL query or to list the available queries. For more details, see [Query subcommands](/cloud/process/pfsql/pfsql-work-with-cli#query-subcommands) |
| `udf`          | Manage pfSQL UDFs.                                                                                                                                                                                    |
| `health-check` | Check the health status of the pfSQL gateway.                                                                                                                                                         |

#### Query subcommands

The following table outlines the subcommands you can use with `pfsql query`.

| Subcommand             | Description               |
| ---------------------- | ------------------------- |
| `pfsql query metadata` | Get query metadata.       |
| `pfsql query list`     | List all queries.         |
| `pfsql query status`   | Get query status.         |
| `pfsql query stats`    | Get query stats.          |
| `pfsql query run`      | Submit a new query.       |
| `pfsql query pause`    | Pause a query.            |
| `pfsql query resume`   | Resume a query.           |
| `pfsql query delete`   | Delete a query.           |
| `pfsql query preview`  | Preview a query’s result. |

## What's next

* [Get started with pfSQL](/cloud/process/pfsql/pfsql-get-started)
* [Understand pfSQL](/cloud/process/pfsql/pfsql-understand)
* [Built-in UDFs](/cloud/process/pfsql/pfsql-built-in-udfs)


# Work with pfSQL on Cloud Console
Source: https://docs.streamnative.io/cloud/process/pfsql/pfsql-work-with-cloud-console



<Note title="Note">
  This feature is currently in alpha. If you want to try it out or have any questions, [submit a ticket](https://support.streamnative.io/hc/en-us/requests/new) to the support team.
</Note>

This section describes how to work with pfSQL on StremNative Cloud's Cloud Console. You can create and delete a query.

## Prerequisites

To fully use the feature provided by pfSQL, you will need to follow [Set up your environment](/cloud/process/pulsar-functions/function-setup) to set up your Pulsar Functions environment first.

## Create a query

1. On the left navigation pane, click pfSQL.

2. On the pfSQL page, click New.

   1. Enter a query name with a description.
   2. Select a service account from the drop-down list.
   3. In the Query area, enter your query.

   <img alt="screenshot of create a query" />

3. Click Deploy.

By default, a newly created query is in running status. You can continue to pause/resume it based on your needs.

## Delete a query

1. On the left navigation pane, click pfSQL.

2. On the pfSQL page, click the Ellipsis (...) icon in the row of the target query and select Delete.

3. Type the query name to confirm and click Confirm.

<img alt="screenshot of delete a query" />

## Troubleshoot a query

1. On the left navigation pane, click pfSQL.

<img alt="screenshot of pfsql overview" />

2. On the pfSQL page, click the target query to enter its overview page.

<img alt="screenshot of query details" />

3. Click the Logs tab to check out more details.

## What's next

* [Work with pfSQL CLI](/cloud/process/pfsql/pfsql-work-with-cli)
* [Get started with pfSQL](/cloud/process/pfsql/pfsql-get-started)
* [Understand pfSQL](/cloud/process/pfsql/pfsql-understand)
* [Built-in UDFs](/cloud/process/pfsql/pfsql-built-in-udfs)


# Custom Runner Images for BYOC Pro
Source: https://docs.streamnative.io/cloud/process/pulsar-functions/develop-functions/function-custom-images



<Note title="BYOC Pro Feature">
  This feature is available for **BYOC Pro clusters only** with trusted mode enabled. To enable trusted mode, [submit a support ticket](https://support.streamnative.io/hc/en-us/requests/new) through StreamNative support.
</Note>

BYOC Pro users can build and use custom runner images for Pulsar functions and IO connectors when trusted mode is enabled. This capability provides the following benefits:

* Include custom dependencies and libraries
* Use specific versions of runtime environments
* Customize the execution environment for your workloads
* Package application-specific configurations

<Warning title="Configuration Impact">
  Custom runner images provide low-level control over the execution environment. **Incorrect image configuration affects the normal operation of Pulsar functions and IO connectors.** Test thoroughly in development environments before deploying to production.
</Warning>

## Prerequisites

* **BYOC Pro cluster** with trusted mode enabled
* Docker installed and configured
* Access to a Docker registry (Docker Hub, private registry, etc.)
* Basic understanding of Dockerfile syntax and container concepts

## Base Images

StreamNative provides official base images for different runtime environments:

### Java Functions

* **Registry**: [streamnative/pulsar-functions-pulsarctl-java-runner](https://hub.docker.com/r/streamnative/pulsar-functions-pulsarctl-java-runner)

```dockerfile theme={null}
FROM streamnative/pulsar-functions-pulsarctl-java-runner:4.0.5.2
```

### Python Functions

* **Registry**: [streamnative/pulsar-functions-pulsarctl-python-runner](https://hub.docker.com/r/streamnative/pulsar-functions-pulsarctl-python-runner)

```dockerfile theme={null}
FROM streamnative/pulsar-functions-pulsarctl-python-runner:4.0.5.2
```

<Note title="Version Selection">
  Use the appropriate base image version that matches your StreamNative Cloud cluster version. Check the [release notes](/release-notes/) for compatibility information.
</Note>

## Building Custom Images

### 1. Create Dockerfile

Create a Dockerfile that extends the StreamNative base image:

```dockerfile theme={null}
FROM streamnative/pulsar-functions-pulsarctl-java-runner:4.0.5.2

# Install additional system packages if needed
USER root
RUN apt-get update && apt-get install -y your-package
USER $UID
# Copy your function JAR and dependencies
COPY --chown=$UID:$GID example-function.jar /pulsar/
COPY --chown=$UID:$GID dependencies/ /pulsar/lib/
```

<Warning title="File Ownership">
  Always use `--chown=$UID:$GID` when copying files. The `$UID` and `$GID` environment variables are provided by StreamNative base images and ensure proper file permissions. **Do not change these values.**
</Warning>

### 2. Build and Push Image

```bash theme={null}
# Build the Docker image
docker build -t your-registry/custom-function:v1.0 .

# Push to your registry
docker push your-registry/custom-function:v1.0
```

### 3. Deploy Function with Custom Image

Use the `runnerImage` parameter in [`custom-runtime-options`](/cloud/process/pulsar-functions/function-config#trusted-mode-configuration):

```bash theme={null}
snctl pulsar admin functions create \
  --tenant your-tenant \
  --namespace your-namespace \
  --name custom-function \
  --jar example-function.jar \
  --classname com.example.MyFunction \
  --inputs input-topic \
  --output output-topic \
  --custom-runtime-options '{"runnerImage": "your-registry/custom-function:v1.0"}'
```

## Examples

### Java Function with Custom Dependencies

```dockerfile theme={null}
FROM streamnative/pulsar-functions-pulsarctl-java-runner:4.0.5.2

# Copy function JAR and custom dependencies
COPY --chown=$UID:$GID target/my-function.jar /pulsar/
COPY --chown=$UID:$GID target/lib/*.jar /pulsar/lib/

# Copy additional configuration files
COPY --chown=$UID:$GID config/logging.properties /pulsar/conf/
```

## Configuration with Other Trusted Mode Options

Combine custom images with other trusted mode configurations:

```json theme={null}
{
  "runnerImage": "your-registry/custom-function:v1.0",
  "javaOPTs": ["-Dmy.custom.config=value"]
}
```

## Best Practices

### Security

* Use minimal base images to reduce attack surface
* Scan images for vulnerabilities before deployment
* Use private registries for proprietary code
* Follow principle of least privilege for container permissions

### Performance

* Optimize image layers for efficient caching
* Remove unnecessary files and packages
* Use multi-stage builds for smaller final images
* Consider image pull time in high-throughput scenarios

### Maintenance

* Tag images with semantic versions
* Maintain compatibility with StreamNative base image updates
* Document custom dependencies and configurations
* Test images thoroughly before production deployment

## Troubleshooting

### Common Issues

**Image Pull Failures**

* Verify registry credentials and access permissions
* Check network connectivity from cluster to registry
* Ensure image tag exists and is properly formatted

**Permission Errors**

* Verify `--chown=$UID:$GID` is used for all COPY operations
* Check that files have correct permissions within the image
* Ensure service account has proper Kubernetes permissions

**Runtime Failures**

* Validate base image compatibility with cluster version
* Check that required dependencies are properly installed
* Verify classpath and module path configurations

For additional troubleshooting support, [contact StreamNative support](https://support.streamnative.io/hc/en-us/requests/new) with your custom image configuration details.


# Develop Pulsar Functions in Golang(Private Preview)
Source: https://docs.streamnative.io/cloud/process/pulsar-functions/develop-functions/function-develop-golang



This section introduces how to develop and pacakge Golang Pulsar functions to use on StreamNative cloud.

We provide a different Golang runtime other than the community one, and it's still in private preview stage, If you want to try it out or have any questions, please [submit a ticket](https://support.streamnative.io/hc/en-us/requests/new) to the support team.

## Develop

We provide a GO [SDK](https://github.com/streamnative/pulsar-function-go) for developing Golang Pulsar Functions.

The following examples use Pulsar Functions SDK for the Golang language.

```go theme={null}
package main

import (
	"context"
	"fmt"
	"strings"

	"github.com/streamnative/pulsar-function-go/pf"
	"github.com/sirupsen/logrus"
)

func HandleExclamation(ctx context.Context, in []byte) ([]byte, error) {
	// 1. unmarshal []byte to your struct, use any schema you want
	payload := string(in)

	// 2. do your logic
	if fc, ok := pf.FromContext(ctx); ok {
		for _, word := range strings.Split(payload, " ") {
			// 2.1 Incr and Get Counter from state store
			_ = fc.IncrCounter(word, 1)
			count, _ := fc.GetCounter(word)
			// 2.2 Sending logs to a Pulsar topic
			logrus.Infof("got word: %s for %d times", word, count)
		}
        // 2.3 Get user-defined configurations
        cfg := fc.GetUserConfValue("configKey")
		// 2.4 Get secret configurations
        sec, err := fc.GetSecret("secretKey")
        if err == nil {
            msg := fmt.Sprintf("config: %v, secret: %s", cfg, *sec)
			// 2.5 Publish to any topic
            _, _ = fc.Publish("persistent://public/default/test-exec-package-serde-extra", []byte(msg))
        }
	}
	data := payload + "!"

	// 3. marshal your struct to []byte
	return []byte(data), nil
}

func main() {
	pf.Start(HandleExclamation)
}
```

To get more examples, please refer to [examples](https://github.com/streamnative/pulsar-function-go/tree/master/examples)

### Feature Matrix

The StreamNative's Golang runtime doesn't support full features comparing to Java runtime, and it's still in developing, below is the matrix:

#### Input Arguments

| Input                       | Java | Go(Pulsar) | Python | Go(StreamNative) |
| :-------------------------- | :--- | :--------- | :----- | :--------------- |
| Custom SerDe                | ✅    | ❌          | ✅      | **?**            |
| Schema - Avro               | ✅    | ❌          | ✅      | **?**            |
| Schema - JSON               | ✅    | ❌          | ✅      | **?**            |
| Schema - Protobuf           | ✅    | ❌          | ❌      | **?**            |
| Schema - KeyValue           | ✅    | ❌          | ❌      | **?**            |
| Schema - AutoSchema         | ✅    | ❌          | ❌      | **?**            |
| Scehma - Protobuf Native    | ✅    | ❌          | ❌      | **?**            |
| e-2-e encryption            | ✅    | ❌          | ✅      | ✅                |
| maxMessageRetries           | ✅    | ❌          | ❌      | ✅                |
| dead-letter policy          | ✅    | ❌          | ❌      | ✅                |
| SubscriptionName            | ✅    | ✅          | ✅      | ✅                |
| SubscriptionType            | ✅    | ✅          | ✅      | ✅                |
| SubscriptionInitialPosition | ✅    | ❌          | ✅      | ✅                |
| AutoAck                     | ✅    | ✅          | ✅      | ✅                |

<Note title="Note">
  Users can implement the Schema themselves since we are passing and expecting \[]byte to/from the users' function, so leave **?** here.
</Note>

#### Output Arguments

| Output                   | Java | Go(Pulsar) | Python | Go(StreamNative) |
| :----------------------- | :--- | :--------- | :----- | :--------------- |
| Custom SerDe             | ✅    | ❌          | ✅      | **?**            |
| Schema - Avro            | ✅    | ❌          | ✅      | **?**            |
| Schema - JSON            | ✅    | ❌          | ✅      | **?**            |
| Schema - Protobuf        | ✅    | ❌          | ❌      | **?**            |
| Schema - KeyValue        | ✅    | ❌          | ❌      | **?**            |
| Schema - AutoSchema      | ✅    | ❌          | ❌      | **?**            |
| Schema - Protobuf Native | ✅    | ❌          | ❌      | **?**            |
| useThreadLocalProducers  | ✅    | ❌          | ❌      | ✅                |
| Key-based Batcher        | ✅    | ✅          | ✅      | ✅                |
| e-2-e encryption         | ✅    | ❌          | ✅      | ✅                |
| Compression              | ✅    | ✅          | ✅      | ✅                |

#### Context

| Context               | Java | Go(Pulsar) | Python | Go(StreamNative) |
| :-------------------- | :--- | :--------- | :----- | :--------------- |
| InputTopics           | ✅    | ✅          | ✅      | ✅                |
| OutputTopic           | ✅    | ✅          | ✅      | ✅                |
| CurrentRecord         | ✅    | ✅          | ✅      | ✅                |
| OutputSchemaType      | ✅    | ❌          | ✅      | ✅                |
| Tenant                | ✅    | ✅          | ✅      | ✅                |
| Namespace             | ✅    | ✅          | ✅      | ✅                |
| FunctionName          | ✅    | ✅          | ✅      | ✅                |
| FunctionId            | ✅    | ✅          | ✅      | ✅                |
| InstanceId            | ✅    | ✅          | ✅      | ✅                |
| NumInstances          | ✅    | ❌          | ✅      | ✅                |
| FunctionVersion       | ✅    | ✅          | ✅      | ✅                |
| PulsarAdminClient     | ✅    | ❌          | ❌      | ❌                |
| GetLogger             | ✅    | ❌          | ✅      | ✅                |
| RecordMetrics         | ✅    | ✅          | ✅      | ❌                |
| UserConfig            | ✅    | ✅          | ✅      | ✅                |
| Secrets               | ✅    | ❌          | ✅      | ✅                |
| State                 | ✅    | ❌          | ❌      | ✅                |
| Publish               | ✅    | ✅          | ✅      | ✅                |
| ConsumerBuilder       | ✅    | ❌          | ❌      | ❌                |
| Seek / Pause / Resume | ✅    | ❌          | ❌      | ❌                |
| PulsarClient          | ✅    | ❌          | ❌      | ❌                |

#### Other

| Other            | Java | Go(Pulsar) | Python | Go(StreamNative) |
| :--------------- | :--- | :--------- | :----- | :--------------- |
| Resources        | ✅    | ✅          | ✅      | ✅                |
| At-most-once     | ✅    | ✅          | ✅      | ✅                |
| At-least-once    | ✅    | ✅          | ✅      | ✅                |
| Effectively-once | ✅    | ❌          | ✅      | ❌                |

## Package

For Golang, we need to compile the function file to an executable one:

1. Prepare your function file:

   ```go theme={null}
   package main

   import (
       "context"
       "github.com/streamnative/pulsar-function-go/pf"
   )

   func HandleExclamation(ctx context.Context, in []byte) ([]byte, error) {
       return []byte(string(in) + "!"), nil
   }

   func main() {
       pf.Start(HandleExclamation)
   }
   ```

2. Compile

   ```bash theme={null}
   go mod init func
   go mod tidy
   GO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -o exclamation exclamation.go
   ```

## Deploy

After creating a cluster, set up your environment and develop\&package your function, you can use the `snctl`, `pulsarctl`, `pulsar-admin` command, the REST API, or `terraform` to deploy a Pulsar function to your cluster.

You can create a Golang Pulsar function by using a local compiled Golang file or an uploaded Pulsar functions package(recommend).

### (Optional) Upload your function file to Pulsar

It's recommended to upload your function file to Pulsar before you create a function. Since you can add a version suffix to the package.

<Tabs>
  <Tab title="snctl">
    Upload packages

    ```bash theme={null}
    snctl pulsar admin packages upload function://${tenant}/${namespace}/${package_name} \
    --path ${file_path} \
    --description "${description}" \
    --properties fileName=${file_name}
    ```

    You should see the following output:

    ```bash theme={null}
    The package 'function://${tenant}/${namespace}/${package_name}' uploaded from path '${file_path}' successfully
    ```
  </Tab>

  <Tab title="Pulsarctl">
    You need to set the context for Pulsarctl first:

    ```bash theme={null}
    # create a context
    pulsarctl context set ${context-name}  \
    --admin-service-url ${admin-service-url} \
    --issuer-endpoint ${issuerUrl} \
    --audience urn:sn:pulsar:${orgName}:${instanceName} \
    --key-file ${privateKey}

    # activate oauth2
    pulsarctl oauth2 activate
    ```

    <Note title="Note">
      Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).

      * `context-name`: any name you want
      * `admin-service-url`: the HTTP service URL of your Pulsar cluster.
      * `privateKey`: the path to the downloaded OAuth2 key file.
      * `issuerUrl`: the URL of the OAuth2 issuer.
      * `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#uniform-resource-name-urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
        * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
        * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
    </Note>

    Upload packages

    ```bash theme={null}
    pulsarctl packages upload function://${tenant}/${namespace}/${package_name} \
    --path ${file_path} \
    --description "${description}" \
    --properties fileName=${file_name}
    ```

    You should see the following output:

    ```bash theme={null}
    The package 'function://${tenant}/${namespace}/${package_name}' uploaded from path '${file_path}' successfully
    ```
  </Tab>

  <Tab title="Pulsar-admin">
    ```bash theme={null}
    ./bin/pulsar-admin \
        --admin-url "${WEB_SERVICE_URL}" \
        --auth-plugin org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 \
        --auth-params '{"privateKey":"file://${privateKey}","issuerUrl":"${issuerUrl}","audience":"urn:sn:pulsar:${orgName}:${instanceName}}' \
        packages upload function://${tenant}/${namespace}/${package_name} \
        --path ${file_path} \
        --description "${description}" \
        --properties fileName=${file_name}
    ```

    <Note title="Note">
      Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).

      * `admin-url`: the HTTP service URL of your Pulsar cluster.
      * `privateKey`: the path to the downloaded OAuth2 key file.
      * `issuerUrl`: the URL of the OAuth2 issuer.
      * `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#uniform-resource-name-urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
        * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
        * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
    </Note>

    You should see the following output:

    ```bash theme={null}
    The package 'function://${tenant}/${namespace}/${package_name}' uploaded from path '${file_path}' successfully
    ```
  </Tab>
</Tabs>

### Create

<Tabs>
  <Tab title="snctl">
    ```bash theme={null}
    snctl pulsar admin functions create \
    --tenant public \
    --namespace default \
    --name function1 \
    --inputs persistent://public/default/test-go-input \
    --output persistent://public/default/test-go-output \
    --classname exclamation \
    --go function://public/default/go-exclamation@v0.1 \
    --custom-runtime-options '{"genericKind": "executable"}'
    ```

    <Note title="Note">
      We are using a different Golang runtime, need to specify the `--custom-runtime-options '{"genericKind": "executable"}'` to make it work.
    </Note>

    You should see something like this:

    ```bash theme={null}
    Created function1 successfully
    ```
  </Tab>

  <Tab title="Pulsarctl">
    ```bash theme={null}
    pulsarctl functions create \
    --tenant public \
    --namespace default \
    --name function1 \
    --inputs persistent://public/default/test-go-input \
    --output persistent://public/default/test-go-output \
    --classname exclamation \
    --go function://public/default/go-exclamation@v0.1 \
    --custom-runtime-options '{"genericKind": "executable"}'
    ```

    <Note title="Note">
      We are using a different Golang runtime, need to specify the `--custom-runtime-options '{"genericKind": "executable"}'` to make it work.
    </Note>

    You should see something like this:

    ```bash theme={null}
    Created function1 successfully
    ```
  </Tab>

  <Tab title="Pulsar-admin">
    ```bash theme={null}
    ./bin/pulsar-admin \
    --admin-url "${WEB_SERVICE_URL}" \
    --auth-plugin org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 \
    --auth-params '{"privateKey":"file:///YOUR-KEY-FILE-PATH","issuerUrl":"https://auth.streamnative.cloud/","audience":"urn:sn:pulsar:${orgName}:${instanceName}}' \
    functions create \
    --tenant public \
    --namespace default \
    --name function1 \
    --inputs persistent://public/default/test-go-input \
    --output persistent://public/default/test-go-output \
    --classname exclamation \
    --go function://public/default/go-exclamation@v0.1 \
    --custom-runtime-options '{"genericKind": "executable"}'
    ```

    <Note title="Note">
      We are using a different Golang runtime, need to specify the `--custom-runtime-options '{"genericKind": "executable"}'` to make it work.
    </Note>

    You should see something like this:

    ```bash theme={null}
    Created successfully
    ```
  </Tab>

  <Tab title="Terraform">
    Create your terraform yaml file:

    ```yaml theme={null}
    terraform {
      required_providers {
        pulsar = {
          version = "0.2.0"
          source = "registry.terraform.io/streamnative/pulsar"
        }
      }
    }

    provider "pulsar" {
      web_service_url = "{$admin-url}"
      api_version     = "3"
      audience = "urn:sn:pulsar:${orgName}:${instanceName}}"
      issuer_url = "${issuerUrl}"
      key_file_path = "${privateKey}"
    }

    // Note: function resource requires v3 api.
    resource "pulsar_function" "function-1" {
      provider = pulsar

      name = "function1"
      tenant = "public"
      namespace = "default"
      parallelism = 1

      processing_guarantees = "ATLEAST_ONCE"

      go = "function://public/default/go-exclamation@v0.1"
      classname = "exclamation.ExclamationFunction"

      inputs = ["persistent://public/default/test-go-input"]

      output = "persistent://public/default/test-go-output"

      subscription_name = "test-sub"
      subscription_position = "Latest"
      cleanup_subscription = true
      skip_to_latest = true
      forward_source_message_property = true
      retain_key_ordering = true
      auto_ack = true
      max_message_retries = 100
      dead_letter_topic = "public/default/dlt"
      log_topic = "public/default/lt"
      timeout_ms = 6666

      secrets = jsonencode(
      {
        "SECRET1": {
           "path": "sectest",
           "key": "hello"
        }
      })
      custom_runtime_options = jsonencode(
      {
         "genericKind": "executable",
          "env": {
              "HELLO": "WORLD"
          },
          "snServiceAccount": "${SERVICE_ACCOUNT}"
      })
    }
    ```

    <Note title="Note">
      We are using a different Golang runtime, need to specify the `--custom-runtime-options '{"genericKind": "executable"}'` to make it work.
    </Note>

    Init the terraform provider in the same dir of your `.tf` file if you haven't done it:

    ```bash theme={null}
    terraform init
    ```

    You should see something like this:

    ```bash theme={null}
    Initializing the backend...

    Initializing provider plugins...
    - Finding streamnative/pulsar versions matching "0.2.0"...
    - Installing streamnative/pulsar v0.2.0...
    - Installed streamnative/pulsar v0.2.0 (self-signed, key ID 3105E1011F3C3671)

    Partner and community providers are signed by their developers.
    If you'd like to know more about provider signing, you can read about it here:
    https://www.terraform.io/docs/cli/plugins/signing.html

    Terraform has created a lock file .terraform.lock.hcl to record the provider
    selections it made above. Include this file in your version control repository
    so that Terraform can guarantee to make the same selections by default when
    you run "terraform init" in the future.

    Terraform has been successfully initialized!

    You may now begin working with Terraform. Try running "terraform plan" to see
    any changes that are required for your infrastructure. All Terraform commands
    should now work.

    If you ever set or change modules or backend configuration for Terraform,
    rerun this command to reinitialize your working directory. If you forget, other
    commands will detect it and remind you to do so if necessary.
    ```

    Create the function:

    ```bash theme={null}
    terraform apply
    ```

    You should see something like:

    ```bash theme={null}
    Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
      + create

    Terraform will perform the following actions:

      # pulsar_function.function-1 will be created
      + resource "pulsar_function" "function-1" {
          + auto_ack                        = true
          + classname                       = "exclamation.ExclamationFunction"
          + cleanup_subscription            = true
          + cpu                             = 0.5
          + custom_runtime_options          = jsonencode(
                {
                  + genericKind = "executable",
                  + env = {
                      + HELLO = "WORLD"
                    },
                  + snServiceAccount = "${SERVICE_ACCOUNT}"
                }
            )
          + dead_letter_topic               = "public/default/dlt"
          + disk_mb                         = 128
          + forward_source_message_property = true
          + id                              = (known after apply)
          + inputs                          = [
              + "persistent://public/default/test-go-input",
            ]
          + go                              = "function://public/default/go-exclamation@v0.1"
          + log_topic                       = "public/default/lt"
          + max_message_retries             = 100
          + name                            = "function1"
          + namespace                       = "default"
          + output                          = "persistent://public/default/test-go-output"
          + parallelism                     = 1
          + processing_guarantees           = "ATLEAST_ONCE"
          + ram_mb                          = 128
          + retain_key_ordering             = true
          + secrets                         = jsonencode(
                {
                  + SECRET1 = {
                      + key  = "hello"
                      + path = "sectest"
                    }
                }
            )
          + skip_to_latest                  = true
          + subscription_name               = "test-sub"
          + subscription_position           = "Latest"
          + tenant                          = "public"
          + timeout_ms                      = 6666
        }

    Plan: 1 to add, 0 to change, 0 to destroy.

    Do you want to perform these actions?
      Terraform will perform the actions described above.
      Only 'yes' will be accepted to approve.

      Enter a value:
    ```

    After enter "yes", you should see the following:

    ```bash theme={null}
    pulsar_function.function-1: Creating...
    pulsar_function.function-1: Creation complete after 1s [id=public/default/function1]

    Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
    ```

    <Note title="Note">
      Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).

      * `admin-url`: the HTTP service URL of your Pulsar cluster.
      * `privateKey`: the path to the downloaded OAuth2 key file.
      * `issuerUrl`: the URL of the OAuth2 issuer.
      * `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#uniform-resource-name-urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
        * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
        * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
    </Note>
  </Tab>

  <Tab title="REST API">
    If you would like to create a function configuration using the REST API you can
    do so using CURL.

    ```bash theme={null}
    curl -X POST ${WEB_SERVICE_URL}/admin/v3/functions/public/default/${FUNCTION_NAME} \
      -H 'Authorization: Bearer ${TOKEN}' \
      -H "Content-Type: multipart/form-data" \
      -F 'functionConfig={"name": "${FUNCTION_NAME}", "tenant": "public", "namespace": "default", "runtime": "GO", "go": "function://public/default/${PACKAGE_NAME}@${PACKAGE_VERSION}", "output": "public/default/output-test", "inputs": ["public/default/input"], "className": "exclamation", "customRuntimeOptions": "{\"genericKind\": \"executable\"}"};type=application/json' \
      -F 'url=function://public/default/${PACKAGE_NAME}@${PACKAGE_VERSION}'
    ```

    <Note title="Note">
      We are using a different Golang runtime, need to specify the `--custom-runtime-options '{"genericKind": "executable"}'` to make it work.
    </Note>

    <Note title="Note">
      The function is assumed to be already uploaded at this point. If you have not
      uploaded the function, change the `url` parameter to be your local filepath.
      This will look something like the following.

      ```bash theme={null}
      -F 'url=file://$YOUR_LCOAL_FUNCTION_FILE'
      ```
    </Note>

    You should see something like this:

    ```bash theme={null}
    Created successfully
    ```

    <Note title="Note">
      Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).

      * `WEB_SERVICE_URL`: the HTTPS service URL of your Pulsar cluster.
      * `TOKEN`: a valid token to interact with your Pulsar cluster.
      * `FUNCTION_NAME`: the name of your function.
      * `FUNCTION_VERSION`: the version of your function you want to deploy e.g. 1.12.
    </Note>
  </Tab>
</Tabs>

For details about Pulsar function configurations, see [Pulsar function configurations](/cloud/process/pulsar-functions/function-config).

## What’s next?

* Learn how to develop [NodeJs functions](/cloud/process/pulsar-functions/develop-functions/function-develop-nodejs).
* Learn how to develop [WASM functions](/cloud/process/pulsar-functions/develop-functions/function-develop-wasm).
* Learn how to [manage functions](/cloud/process/pulsar-functions/function-manage).
* Learn how to [configure stateful functions](/cloud/process/pulsar-functions/function-state).
* Learn how to [monitor functions](/cloud/process/pulsar-functions/function-monitoring).
* Reference [common configurations](/cloud/process/pulsar-functions/function-config).


# Develop Pulsar Functions in Java
Source: https://docs.streamnative.io/cloud/process/pulsar-functions/develop-functions/function-develop-java



This section introduces how to develop and pacakge Java Pulsar functions to use on StreamNative cloud.

## Develop

StreamNative supports all features for the Java functions, please refer to: [Develop Functions](https://pulsar.apache.org/docs/3.2.x/functions-develop/)
to learn how to develop a Java Function.

## Package

Please refer to: [Pacakge Java Functions](https://pulsar.apache.org/docs/3.2.x/functions-package-java/)

## Deploy

After creating a cluster, set up your environment and develop\&package your function, you can use the `snctl`, `pulsarctl`, `pulsar-admin` command, the REST API, or `terraform` to deploy a Pulsar function to your cluster.

You can create a java Pulsar function by using a local JAR/NAR package or an uploaded Pulsar functions package(recommend).

### (Optional) Upload your function file to Pulsar

It's recommended to upload your function file to Pulsar before you create a function. Since you can add a version suffix to the package.

<Tabs>
  <Tab title="snctl">
    Upload packages

    ```bash theme={null}
    snctl pulsar admin packages upload function://${tenant}/${namespace}/${package_name} \
    --path ${file_path} \
    --description "${description}" \
    --properties fileName=${file_name}
    ```

    You should see the following output:

    ```bash theme={null}
    The package 'function://${tenant}/${namespace}/${package_name}' uploaded from path '${file_path}' successfully
    ```
  </Tab>

  <Tab title="Pulsarctl">
    You need to set the context for Pulsarctl first:

    ```bash theme={null}
    # create a context
    pulsarctl context set ${context-name}  \
    --admin-service-url ${admin-service-url} \
    --issuer-endpoint ${issuerUrl} \
    --audience urn:sn:pulsar:${orgName}:${instanceName} \
    --key-file ${privateKey}

    # activate oauth2
    pulsarctl oauth2 activate
    ```

    <Note title="Note">
      Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).

      * `context-name`: any name you want
      * `admin-service-url`: the HTTP service URL of your Pulsar cluster.
      * `privateKey`: the path to the downloaded OAuth2 key file.
      * `issuerUrl`: the URL of the OAuth2 issuer.
      * `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#uniform-resource-name-urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
        * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
        * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
    </Note>

    Upload packages

    ```bash theme={null}
    pulsarctl packages upload function://${tenant}/${namespace}/${package_name} \
    --path ${file_path} \
    --description "${description}" \
    --properties fileName=${file_name}
    ```

    You should see the following output:

    ```bash theme={null}
    The package 'function://${tenant}/${namespace}/${package_name}' uploaded from path '${file_path}' successfully
    ```
  </Tab>

  <Tab title="Pulsar-admin">
    ```bash theme={null}
    ./bin/pulsar-admin \
        --admin-url "${WEB_SERVICE_URL}" \
        --auth-plugin org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 \
        --auth-params '{"privateKey":"file://${privateKey}","issuerUrl":"${issuerUrl}","audience":"urn:sn:pulsar:${orgName}:${instanceName}}' \
        packages upload function://${tenant}/${namespace}/${package_name} \
        --path ${file_path} \
        --description "${description}" \
        --properties fileName=${file_name}
    ```

    <Note title="Note">
      Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).

      * `admin-url`: the HTTP service URL of your Pulsar cluster.
      * `privateKey`: the path to the downloaded OAuth2 key file.
      * `issuerUrl`: the URL of the OAuth2 issuer.
      * `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#uniform-resource-name-urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
        * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
        * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
    </Note>

    You should see the following output:

    ```bash theme={null}
    The package 'function://${tenant}/${namespace}/${package_name}' uploaded from path '${file_path}' successfully
    ```
  </Tab>
</Tabs>

### Create

You need to package your Java function as a `.jar` or `.nar` file (and upload it to Pulsar) first.

<Tabs>
  <Tab title="snctl">
    ```bash theme={null}
    snctl pulsar admin functions create \
    --tenant public \
    --namespace default \
    --name function1 \
    --inputs persistent://public/default/test-java-input \
    --output persistent://public/default/test-java-output \
    --classname org.apache.pulsar.functions.api.examples.ExclamationFunction \
    --jar function://public/default/exclamation@v0.1 \
    --sn-service-account $SERVICE_ACCOUNT
    ```

    <Note>
      Only Pulsar 4.0.x clusters running version 4.0.10.6 or later, or Pulsar 4.2.x clusters running version 4.2.1.4 or later, support the `--sn-service-account` flag.

      You can use `--as-service-account` instead of `--sn-service-account` if you are using other versions of Pulsar clusters.
    </Note>

    You should see something like this:

    ```bash theme={null}
    Created function1 successfully
    ```
  </Tab>

  <Tab title="Pulsarctl">
    ```bash theme={null}
    pulsarctl functions create \
    --tenant public \
    --namespace default \
    --name function1 \
    --inputs persistent://public/default/test-java-input \
    --output persistent://public/default/test-java-output \
    --classname org.apache.pulsar.functions.api.examples.ExclamationFunction \
    --jar function://public/default/exclamation@v0.1
    ```

    You should see something like this:

    ```bash theme={null}
    Created function1 successfully
    ```
  </Tab>

  <Tab title="Pulsar-admin">
    ```bash theme={null}
    ./bin/pulsar-admin \
    --admin-url "${WEB_SERVICE_URL}" \
    --auth-plugin org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 \
    --auth-params '{"privateKey":"file:///YOUR-KEY-FILE-PATH","issuerUrl":"https://auth.streamnative.cloud/","audience":"urn:sn:pulsar:${orgName}:${instanceName}}' \
    functions create \
    --tenant public \
    --namespace default \
    --name function1 \
    --inputs persistent://public/default/test-java-input \
    --output persistent://public/default/test-java-output \
    --classname org.apache.pulsar.functions.api.examples.ExclamationFunction \
    --jar function://public/default/exclamation@v0.1
    ```

    You should see something like this:

    ```bash theme={null}
    Created successfully
    ```
  </Tab>

  <Tab title="Terraform">
    Create your terraform yaml file:

    ```yaml theme={null}
    terraform {
      required_providers {
        pulsar = {
          version = "0.2.0"
          source = "registry.terraform.io/streamnative/pulsar"
        }
      }
    }

    provider "pulsar" {
      web_service_url = "{$admin-url}"
      api_version     = "3"
      audience = "urn:sn:pulsar:${orgName}:${instanceName}}"
      issuer_url = "${issuerUrl}"
      key_file_path = "${privateKey}"
    }

    // Note: function resource requires v3 api.
    resource "pulsar_function" "function-1" {
      provider = pulsar

      name = "function1"
      tenant = "public"
      namespace = "default"
      parallelism = 1

      processing_guarantees = "ATLEAST_ONCE"

      jar = "function://public/default/exclamation@v0.1"
      classname = "org.apache.pulsar.functions.api.examples.ExclamationFunction"

      inputs = ["persistent://public/default/test-java-input"]

      output = "persistent://public/default/test-java-output"

      subscription_name = "test-sub"
      subscription_position = "Latest"
      cleanup_subscription = true
      skip_to_latest = true
      forward_source_message_property = true
      retain_key_ordering = true
      auto_ack = true
      max_message_retries = 100
      dead_letter_topic = "public/default/dlt"
      log_topic = "public/default/lt"
      timeout_ms = 6666

      secrets = jsonencode(
      {
        "SECRET1": {
           "path": "sectest",
           "key": "hello"
        }
      })
      custom_runtime_options = jsonencode(
      {
          "env": {
              "HELLO": "WORLD"
          },
          "snServiceAccount": "${SERVICE_ACCOUNT}"
      })
    }
    ```

    Init the terraform provider in the same dir of your `.tf` file if you haven't done it:

    ```bash theme={null}
    terraform init
    ```

    You should see something like this:

    ```bash theme={null}
    Initializing the backend...

    Initializing provider plugins...
    - Finding streamnative/pulsar versions matching "0.2.0"...
    - Installing streamnative/pulsar v0.2.0...
    - Installed streamnative/pulsar v0.2.0 (self-signed, key ID 3105E1011F3C3671)

    Partner and community providers are signed by their developers.
    If you'd like to know more about provider signing, you can read about it here:
    https://www.terraform.io/docs/cli/plugins/signing.html

    Terraform has created a lock file .terraform.lock.hcl to record the provider
    selections it made above. Include this file in your version control repository
    so that Terraform can guarantee to make the same selections by default when
    you run "terraform init" in the future.

    Terraform has been successfully initialized!

    You may now begin working with Terraform. Try running "terraform plan" to see
    any changes that are required for your infrastructure. All Terraform commands
    should now work.

    If you ever set or change modules or backend configuration for Terraform,
    rerun this command to reinitialize your working directory. If you forget, other
    commands will detect it and remind you to do so if necessary.
    ```

    Create the function:

    ```bash theme={null}
    terraform apply
    ```

    You should see something like:

    ```bash theme={null}
    Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
      + create

    Terraform will perform the following actions:

      # pulsar_function.function-1 will be created
      + resource "pulsar_function" "function-1" {
          + auto_ack                        = true
          + classname                       = "org.apache.pulsar.functions.api.examples.ExclamationFunction"
          + cleanup_subscription            = true
          + cpu                             = 0.5
          + custom_runtime_options          = jsonencode(
                {
                  + env = {
                      + HELLO = "WORLD"
                    },
                  + snServiceAccount = "${SERVICE_ACCOUNT}"
                }
            )
          + dead_letter_topic               = "public/default/dlt"
          + disk_mb                         = 128
          + forward_source_message_property = true
          + id                              = (known after apply)
          + inputs                          = [
              + "persistent://public/default/test-java-input",
            ]
          + jar                             = "function://public/default/exclamation@v0.1"
          + log_topic                       = "public/default/lt"
          + max_message_retries             = 100
          + name                            = "function1"
          + namespace                       = "default"
          + output                          = "persistent://public/default/test-java-output"
          + parallelism                     = 1
          + processing_guarantees           = "ATLEAST_ONCE"
          + ram_mb                          = 128
          + retain_key_ordering             = true
          + secrets                         = jsonencode(
                {
                  + SECRET1 = {
                      + key  = "hello"
                      + path = "sectest"
                    }
                }
            )
          + skip_to_latest                  = true
          + subscription_name               = "test-sub"
          + subscription_position           = "Latest"
          + tenant                          = "public"
          + timeout_ms                      = 6666
        }

    Plan: 1 to add, 0 to change, 0 to destroy.

    Do you want to perform these actions?
      Terraform will perform the actions described above.
      Only 'yes' will be accepted to approve.

      Enter a value:
    ```

    After enter "yes", you should see the following:

    ```bash theme={null}
    pulsar_function.function-1: Creating...
    pulsar_function.function-1: Creation complete after 1s [id=public/default/function1]

    Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
    ```

    <Note title="Note">
      Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).

      * `admin-url`: the HTTP service URL of your Pulsar cluster.
      * `privateKey`: the path to the downloaded OAuth2 key file.
      * `issuerUrl`: the URL of the OAuth2 issuer.
      * `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#uniform-resource-name-urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
        * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
        * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
    </Note>
  </Tab>

  <Tab title="REST API">
    If you would like to create a function configuration using the REST API you can
    do so using CURL.

    ```bash theme={null}
      curl -X POST ${WEB_SERVICE_URL}/admin/v3/functions/public/default/${FUNCTION_NAME} \
      -H 'Authorization: Bearer ${TOKEN}' \
      -H "Content-Type: multipart/form-data" \
      -F 'functionConfig={"name": "${FUNCTION_NAME}", "tenant": "public", "namespace": "default", "runtime": "JAVA", "jar": "function://public/default/${PACKAGE_NAME}@${PACKAGE_VERSION}", "output": "public/default/output-test", "inputs": ["public/default/input"], "className": "org.apache.pulsar.functions.api.examples.ExclamationFunction"};type=application/json' \
      -F 'url=function://public/default/${PACKAGE_NAME}@${PACKAGE_VERSION}'
    ```

    <Note title="Note">
      The function is assumed to be already uploaded at this point. If you have not
      uploaded the function, change the `url` parameter to be your local filepath.
      This will look something like the following.

      ```bash theme={null}
      -F 'url=file://$YOUR_LCOAL_JAR_OR_NAR_FILE'
      ```
    </Note>

    You should see something like this:

    ```bash theme={null}
    Created successfully
    ```

    <Note title="Note">
      Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).

      * `WEB_SERVICE_URL`: the HTTPS service URL of your Pulsar cluster.
      * `TOKEN`: a valid token to interact with your Pulsar cluster.
      * `FUNCTION_NAME`: the name of your function.
      * `FUNCTION_VERSION`: the version of your function you want to deploy e.g. 1.12.
    </Note>
  </Tab>
</Tabs>

For details about Pulsar function configurations, see [Pulsar function configurations](#pulsar-function-configurations).

## What’s next?

* Learn how to develop [Python functions](/cloud/process/pulsar-functions/develop-functions/function-develop-python).
* Learn how to develop [Golang functions](/cloud/process/pulsar-functions/develop-functions/function-develop-golang).
* Learn how to develop [NodeJs functions](/cloud/process/pulsar-functions/develop-functions/function-develop-nodejs).
* Learn how to develop [WASM functions](/cloud/process/pulsar-functions/develop-functions/function-develop-wasm).
* Learn how to [manage functions](/cloud/process/pulsar-functions/function-manage).
* Learn how to [configure stateful functions](/cloud/process/pulsar-functions/function-state).
* Learn how to [monitor functions](/cloud/process/pulsar-functions/function-monitoring).
* Reference [common configurations](/cloud/process/pulsar-functions/function-config).


# Develop Pulsar Functions in NodeJs(Private Preview)
Source: https://docs.streamnative.io/cloud/process/pulsar-functions/develop-functions/function-develop-nodejs



This section introduces how to develop and pacakge NodeJs Pulsar functions to use on StreamNative cloud.

The NodeJs runtime is still in private preview stage, If you want to try it out or have any questions, please [submit a ticket](https://support.streamnative.io/hc/en-us/requests/new) to the support team.

## Develop

The StreamNative cloud will find the `process` method from the NodeJs function file, below is an example:

```node theme={null}
async function process(message, context) {
  // 1. Get the logger to send logs to a Pulsar topic(when log-topic is set)
  const logger = context.getLogger()
  for (let word of message.split(' ')) {
    // 2. Incre and Get counter from state store
    await context.incrementCounter(word, 1)
    let count = await context.getCounter(word)
    logger.info(`got word: ${word} for ${count['value']} times`)
  }
  // 3. Get user-defined configurations
  let cfg = context.getUserConfigValue('configKey')
  // 4. Get secret configurations
  let sec = context.getSecret('secretKey')
  let result = `config: ${cfg}, secret: ${sec}`
  // 5. Publish messages to any Pulsar topic
  await context.publish(
    'persistent://public/default/test-node-package-serde-extra',
    result
  )
  // 6. Return the processing result
  return message.concat('!')
}
```

It can also handle `Avro` and `Json` schema record, for `Avro` scheme input and output:

```node theme={null}
function process(params) {
  params['grade']['int'] = params['grade']['int'] + 1
  return params
}

const definitions = {
  name: 'Student',
  type: 'record',
  fields: [
    { name: 'name', type: ['null', 'string'] },
    { name: 'age', type: ['null', 'int'] },
    { name: 'grade', type: ['null', 'int'] },
  ],
}

module.exports.definitions = definitions
```

For `Json` input and output:

```node theme={null}
function process(params) {
  params['grade'] = params['grade'] + 1
  return params
}
```

To get more examples, please refer to [examples](https://github.com/streamnative/pulsar-function-examples/tree/main/generic-runtime/nodejs)

### Feature Matrix

The NodeJs runtime doesn't support full features comparing to Java runtime, and it's still in developing, below is the matrix:

#### Input Arguments

| Input                       | Java | Go(Pulsar) | Python | NodeJs |
| :-------------------------- | :--- | :--------- | :----- | :----- |
| Custom SerDe                | ✅    | ❌          | ✅      | ✅      |
| Schema - Avro               | ✅    | ❌          | ✅      | ✅      |
| Schema - JSON               | ✅    | ❌          | ✅      | ✅      |
| Schema - Protobuf           | ✅    | ❌          | ❌      | ❌      |
| Schema - KeyValue           | ✅    | ❌          | ❌      | ❌      |
| Schema - AutoSchema         | ✅    | ❌          | ❌      | ❌      |
| Scehma - Protobuf Native    | ✅    | ❌          | ❌      | ❌      |
| e-2-e encryption            | ✅    | ❌          | ✅      | ✅      |
| maxMessageRetries           | ✅    | ❌          | ❌      | ✅      |
| dead-letter policy          | ✅    | ❌          | ❌      | ✅      |
| SubscriptionName            | ✅    | ✅          | ✅      | ✅      |
| SubscriptionType            | ✅    | ✅          | ✅      | ✅      |
| SubscriptionInitialPosition | ✅    | ❌          | ✅      | ✅      |
| AutoAck                     | ✅    | ✅          | ✅      | ✅      |

#### Output Arguments

| Output                   | Java | Go(Pulsar) | Python | NodeJs |
| :----------------------- | :--- | :--------- | :----- | :----- |
| Custom SerDe             | ✅    | ❌          | ✅      | ✅      |
| Schema - Avro            | ✅    | ❌          | ✅      | ✅      |
| Schema - JSON            | ✅    | ❌          | ✅      | ✅      |
| Schema - Protobuf        | ✅    | ❌          | ❌      | ❌      |
| Schema - KeyValue        | ✅    | ❌          | ❌      | ❌      |
| Schema - AutoSchema      | ✅    | ❌          | ❌      | ❌      |
| Schema - Protobuf Native | ✅    | ❌          | ❌      | ❌      |
| useThreadLocalProducers  | ✅    | ❌          | ❌      | ✅      |
| Key-based Batcher        | ✅    | ✅          | ✅      | ✅      |
| e-2-e encryption         | ✅    | ❌          | ✅      | ✅      |
| Compression              | ✅    | ✅          | ✅      | ✅      |

#### Context

| Context               | Java | Go(Pulsar) | Python | NodeJs |
| :-------------------- | :--- | :--------- | :----- | :----- |
| InputTopics           | ✅    | ✅          | ✅      | ✅      |
| OutputTopic           | ✅    | ✅          | ✅      | ✅      |
| CurrentRecord         | ✅    | ✅          | ✅      | ✅      |
| OutputSchemaType      | ✅    | ❌          | ✅      | ✅      |
| Tenant                | ✅    | ✅          | ✅      | ✅      |
| Namespace             | ✅    | ✅          | ✅      | ✅      |
| FunctionName          | ✅    | ✅          | ✅      | ✅      |
| FunctionId            | ✅    | ✅          | ✅      | ✅      |
| InstanceId            | ✅    | ✅          | ✅      | ✅      |
| NumInstances          | ✅    | ❌          | ✅      | ✅      |
| FunctionVersion       | ✅    | ✅          | ✅      | ✅      |
| PulsarAdminClient     | ✅    | ❌          | ❌      | ❌      |
| GetLogger             | ✅    | ❌          | ✅      | ✅      |
| RecordMetrics         | ✅    | ✅          | ✅      | ❌      |
| UserConfig            | ✅    | ✅          | ✅      | ✅      |
| Secrets               | ✅    | ❌          | ✅      | ✅      |
| State                 | ✅    | ❌          | ❌      | ✅      |
| Publish               | ✅    | ✅          | ✅      | ✅      |
| ConsumerBuilder       | ✅    | ❌          | ❌      | ❌      |
| Seek / Pause / Resume | ✅    | ❌          | ❌      | ❌      |
| PulsarClient          | ✅    | ❌          | ❌      | ❌      |

#### Other

| Other            | Java | Go(Pulsar) | Python | NodeJs |
| :--------------- | :--- | :--------- | :----- | :----- |
| Resources        | ✅    | ✅          | ✅      | ✅      |
| At-most-once     | ✅    | ✅          | ✅      | ✅      |
| At-least-once    | ✅    | ✅          | ✅      | ✅      |
| Effectively-once | ✅    | ❌          | ✅      | ❌      |

## Package

You can either provide a single `.js` file or package your function into a `.zip` file when creating NodeJs functions.

A zip file should contain:

1. the `pacakge.json` file
2. other source code file

Like:

```
Archive:  pad.zip
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
     107  Defl:N       89  17% 2024-03-01 01:07 918cdb62  index.js
      94  Defl:N       78  17% 2024-03-01 01:07 2ed1db00  package.json
sidebarTitle: Develop NodeJs Functions
--------          -------  ---                            -------
     201              167  17%                            2 files
```

Below is an example of the `package.json`:

```json theme={null}
{
  "name": "test",
  "main": "index.js",
  "dependencies": {
    "left-pad": "1.3.0"
  }
}
```

The "main" file is required for such `package.json`:

```node theme={null}
function process(message) {
  const leftPad = require('left-pad')
  return leftPad(message, 20, 'x')
}
```

The StreamNative Cloud will install dependencies specified in the `package.json` and handle your function.

## Deploy

After creating a cluster, set up your environment and develop\&package your function, you can use the `snctl`, `pulsarctl`, `pulsar-admin` command, the REST API, or `terraform` to deploy a Pulsar function to your cluster.

You can create a NodeJs Pulsar function by using a local `.js` or `.zip` file or an uploaded Pulsar functions package(recommend).

### (Optional) Upload your function file to Pulsar

It's recommended to upload your function file to Pulsar before you create a function. Since you can add a version suffix to the package.

<Tabs>
  <Tab title="snctl">
    Upload packages

    ```bash theme={null}
    snctl pulsar admin packages upload function://${tenant}/${namespace}/${package_name} \
    --path ${file_path} \
    --description "${description}" \
    --properties fileName=${file_name}
    ```

    You should see the following output:

    ```bash theme={null}
    The package 'function://${tenant}/${namespace}/${package_name}' uploaded from path '${file_path}' successfully
    ```
  </Tab>

  <Tab title="Pulsarctl">
    You need to set the context for Pulsarctl first:

    ```bash theme={null}
    # create a context
    pulsarctl context set ${context-name}  \
    --admin-service-url ${admin-service-url} \
    --issuer-endpoint ${issuerUrl} \
    --audience urn:sn:pulsar:${orgName}:${instanceName} \
    --key-file ${privateKey}

    # activate oauth2
    pulsarctl oauth2 activate
    ```

    <Note title="Note">
      Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).

      * `context-name`: any name you want
      * `admin-service-url`: the HTTP service URL of your Pulsar cluster.
      * `privateKey`: the path to the downloaded OAuth2 key file.
      * `issuerUrl`: the URL of the OAuth2 issuer.
      * `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#uniform-resource-name-urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
        * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
        * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
    </Note>

    Upload packages

    ```bash theme={null}
    pulsarctl packages upload function://${tenant}/${namespace}/${package_name} \
    --path ${file_path} \
    --description "${description}" \
    --properties fileName=${file_name}
    ```

    You should see the following output:

    ```bash theme={null}
    The package 'function://${tenant}/${namespace}/${package_name}' uploaded from path '${file_path}' successfully
    ```
  </Tab>

  <Tab title="Pulsar-admin">
    ```bash theme={null}
    ./bin/pulsar-admin \
        --admin-url "${WEB_SERVICE_URL}" \
        --auth-plugin org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 \
        --auth-params '{"privateKey":"file://${privateKey}","issuerUrl":"${issuerUrl}","audience":"urn:sn:pulsar:${orgName}:${instanceName}}' \
        packages upload function://${tenant}/${namespace}/${package_name} \
        --path ${file_path} \
        --description "${description}" \
        --properties fileName=${file_name}
    ```

    <Note title="Note">
      Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).

      * `admin-url`: the HTTP service URL of your Pulsar cluster.
      * `privateKey`: the path to the downloaded OAuth2 key file.
      * `issuerUrl`: the URL of the OAuth2 issuer.
      * `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#uniform-resource-name-urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
        * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
        * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
    </Note>

    You should see the following output:

    ```bash theme={null}
    The package 'function://${tenant}/${namespace}/${package_name}' uploaded from path '${file_path}' successfully
    ```
  </Tab>
</Tabs>

### Create

<Tabs>
  <Tab title="snctl">
    ```bash theme={null}
    snctl pulsar admin functions create \
    --tenant public \
    --namespace default \
    --name function1 \
    --inputs persistent://public/default/test-node-input \
    --output persistent://public/default/test-node-output \
    --classname exclamation \
    --py function://public/default/node-exclamation@v0.1 \
    --custom-runtime-options '{"genericKind": "nodejs"}' \
    --sn-service-account $SERVICE_ACCOUNT
    ```

    <Note>
      Only Pulsar 4.0.x clusters running version 4.0.10.6 or later, or Pulsar 4.2.x clusters running version 4.2.1.4 or later, support the `--sn-service-account` flag.

      You can use `--as-service-account` instead of `--sn-service-account` if you are using other versions of Pulsar clusters.
    </Note>

    <Note title="Note">
      Since Pulsar doesn't support NodeJs runtime, we need to use `--py` to specify the function file and specify the `--custom-runtime-options '{"genericKind": "nodejs"}'` to make it work.
    </Note>

    You should see something like this:

    ```bash theme={null}
    Created function1 successfully
    ```
  </Tab>

  <Tab title="Pulsarctl">
    ```bash theme={null}
    pulsarctl functions create \
    --tenant public \
    --namespace default \
    --name function1 \
    --inputs persistent://public/default/test-node-input \
    --output persistent://public/default/test-node-output \
    --classname exclamation \
    --py function://public/default/node-exclamation@v0.1 \
    --custom-runtime-options '{"genericKind": "nodejs"}'
    ```

    <Note title="Note">
      Since Pulsar doesn't support NodeJs runtime, we need to use `--py` to specify the function file and specify the `--custom-runtime-options '{"genericKind": "nodejs"}'` to make it work.
    </Note>

    You should see something like this:

    ```bash theme={null}
    Created function1 successfully
    ```
  </Tab>

  <Tab title="Pulsar-admin">
    ```bash theme={null}
    ./bin/pulsar-admin \
    --admin-url "${WEB_SERVICE_URL}" \
    --auth-plugin org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 \
    --auth-params '{"privateKey":"file:///YOUR-KEY-FILE-PATH","issuerUrl":"https://auth.streamnative.cloud/","audience":"urn:sn:pulsar:${orgName}:${instanceName}}' \
    functions create \
    --tenant public \
    --namespace default \
    --name function1 \
    --inputs persistent://public/default/test-node-input \
    --output persistent://public/default/test-node-output \
    --classname exclamation \
    --py function://public/default/node-exclamation@v0.1 \
    --custom-runtime-options '{"genericKind": "nodejs"}'
    ```

    <Note title="Note">
      Since Pulsar doesn't support NodeJs runtime, we need to use `--py` to specify the function file and specify the `--custom-runtime-options '{"genericKind": "nodejs"}'` to make it work.
    </Note>

    You should see something like this:

    ```bash theme={null}
    Created successfully
    ```
  </Tab>

  <Tab title="Terraform">
    Create your terraform yaml file:

    ```yaml theme={null}
    terraform {
      required_providers {
        pulsar = {
          version = "0.2.0"
          source = "registry.terraform.io/streamnative/pulsar"
        }
      }
    }

    provider "pulsar" {
      web_service_url = "{$admin-url}"
      api_version     = "3"
      audience = "urn:sn:pulsar:${orgName}:${instanceName}}"
      issuer_url = "${issuerUrl}"
      key_file_path = "${privateKey}"
    }

    // Note: function resource requires v3 api.
    resource "pulsar_function" "function-1" {
      provider = pulsar

      name = "function1"
      tenant = "public"
      namespace = "default"
      parallelism = 1

      processing_guarantees = "ATLEAST_ONCE"

      py = "function://public/default/node-exclamation@v0.1"
      classname = "exclamation.ExclamationFunction"

      inputs = ["persistent://public/default/test-node-input"]

      output = "persistent://public/default/test-node-output"

      subscription_name = "test-sub"
      subscription_position = "Latest"
      cleanup_subscription = true
      skip_to_latest = true
      forward_source_message_property = true
      retain_key_ordering = true
      auto_ack = true
      max_message_retries = 100
      dead_letter_topic = "public/default/dlt"
      log_topic = "public/default/lt"
      timeout_ms = 6666

      secrets = jsonencode(
      {
        "SECRET1": {
           "path": "sectest",
           "key": "hello"
        }
      })
      custom_runtime_options = jsonencode(
      {
         "genericKind": "nodejs",
          "env": {
              "HELLO": "WORLD"
          },
          "snServiceAccount": "${SERVICE_ACCOUNT}"
      })
    }
    ```

    <Note title="Note">
      Since Pulsar doesn't support NodeJs runtime, we need to use `--py` to specify the function file and specify the `--custom-runtime-options '{"genericKind": "nodejs"}'` to make it work.
    </Note>

    Init the terraform provider in the same dir of your `.tf` file if you haven't done it:

    ```bash theme={null}
    terraform init
    ```

    You should see something like this:

    ```bash theme={null}
    Initializing the backend...

    Initializing provider plugins...
    - Finding streamnative/pulsar versions matching "0.2.0"...
    - Installing streamnative/pulsar v0.2.0...
    - Installed streamnative/pulsar v0.2.0 (self-signed, key ID 3105E1011F3C3671)

    Partner and community providers are signed by their developers.
    If you'd like to know more about provider signing, you can read about it here:
    https://www.terraform.io/docs/cli/plugins/signing.html

    Terraform has created a lock file .terraform.lock.hcl to record the provider
    selections it made above. Include this file in your version control repository
    so that Terraform can guarantee to make the same selections by default when
    you run "terraform init" in the future.

    Terraform has been successfully initialized!

    You may now begin working with Terraform. Try running "terraform plan" to see
    any changes that are required for your infrastructure. All Terraform commands
    should now work.

    If you ever set or change modules or backend configuration for Terraform,
    rerun this command to reinitialize your working directory. If you forget, other
    commands will detect it and remind you to do so if necessary.
    ```

    Create the function:

    ```bash theme={null}
    terraform apply
    ```

    You should see something like:

    ```bash theme={null}
    Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
      + create

    Terraform will perform the following actions:

      # pulsar_function.function-1 will be created
      + resource "pulsar_function" "function-1" {
          + auto_ack                        = true
          + classname                       = "exclamation.ExclamationFunction"
          + cleanup_subscription            = true
          + cpu                             = 0.5
          + custom_runtime_options          = jsonencode(
                {
                  + genericKind = "nodejs",
                  + env = {
                      + HELLO = "WORLD"
                    },
                  + snServiceAccount = "${SERVICE_ACCOUNT}"
                }
            )
          + dead_letter_topic               = "public/default/dlt"
          + disk_mb                         = 128
          + forward_source_message_property = true
          + id                              = (known after apply)
          + inputs                          = [
              + "persistent://public/default/test-node-input",
            ]
          + py                              = "function://public/default/node-exclamation@v0.1"
          + log_topic                       = "public/default/lt"
          + max_message_retries             = 100
          + name                            = "function1"
          + namespace                       = "default"
          + output                          = "persistent://public/default/test-node-output"
          + parallelism                     = 1
          + processing_guarantees           = "ATLEAST_ONCE"
          + ram_mb                          = 128
          + retain_key_ordering             = true
          + secrets                         = jsonencode(
                {
                  + SECRET1 = {
                      + key  = "hello"
                      + path = "sectest"
                    }
                }
            )
          + skip_to_latest                  = true
          + subscription_name               = "test-sub"
          + subscription_position           = "Latest"
          + tenant                          = "public"
          + timeout_ms                      = 6666
        }

    Plan: 1 to add, 0 to change, 0 to destroy.

    Do you want to perform these actions?
      Terraform will perform the actions described above.
      Only 'yes' will be accepted to approve.

      Enter a value:
    ```

    After enter "yes", you should see the following:

    ```bash theme={null}
    pulsar_function.function-1: Creating...
    pulsar_function.function-1: Creation complete after 1s [id=public/default/function1]

    Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
    ```

    <Note title="Note">
      Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).

      * `admin-url`: the HTTP service URL of your Pulsar cluster.
      * `privateKey`: the path to the downloaded OAuth2 key file.
      * `issuerUrl`: the URL of the OAuth2 issuer.
      * `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#uniform-resource-name-urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
        * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
        * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
    </Note>
  </Tab>

  <Tab title="REST API">
    If you would like to create a function configuration using the REST API you can
    do so using CURL.

    ```bash theme={null}
    curl -X POST ${WEB_SERVICE_URL}/admin/v3/functions/public/default/${FUNCTION_NAME} \
      -H 'Authorization: Bearer ${TOKEN}' \
      -H "Content-Type: multipart/form-data" \
      -F 'functionConfig={"name": "${FUNCTION_NAME}", "tenant": "public", "namespace": "default", "runtime": "PYTHON", "py": "function://public/default/${PACKAGE_NAME}@${PACKAGE_VERSION}", "output": "public/default/output-test", "inputs": ["public/default/input"], "className": "exclamation", "customRuntimeOptions": "{\"genericKind\": \"nodejs\"}"};type=application/json' \
      -F 'url=function://public/default/${PACKAGE_NAME}@${PACKAGE_VERSION}'
    ```

    <Note title="Note">
      Since Pulsar doesn't support NodeJs runtime, we need to use `--py` to specify the function file and specify the `--custom-runtime-options '{"genericKind": "nodejs"}'` to make it work.
    </Note>

    <Note title="Note">
      The function is assumed to be already uploaded at this point. If you have not
      uploaded the function, change the `url` parameter to be your local filepath.
      This will look something like the following.

      ```bash theme={null}
      -F 'url=file://$YOUR_LCOAL_FUNCTION_FILE'
      ```
    </Note>

    You should see something like this:

    ```bash theme={null}
    Created successfully
    ```

    <Note title="Note">
      Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).

      * `WEB_SERVICE_URL`: the HTTPS service URL of your Pulsar cluster.
      * `TOKEN`: a valid token to interact with your Pulsar cluster.
      * `FUNCTION_NAME`: the name of your function.
      * `FUNCTION_VERSION`: the version of your function you want to deploy e.g. 1.12.
    </Note>
  </Tab>
</Tabs>

For details about Pulsar function configurations, see [Pulsar function configurations](/cloud/process/pulsar-functions/function-config).

## What’s next?

* Learn how to develop [WASM functions](/cloud/process/pulsar-functions/develop-functions/function-develop-wasm).
* Learn how to [manage functions](/cloud/process/pulsar-functions/function-manage).
* Learn how to [configure stateful functions](/cloud/process/pulsar-functions/function-state).
* Learn how to [monitor functions](/cloud/process/pulsar-functions/function-monitoring).
* Reference [common configurations](/cloud/process/pulsar-functions/function-config).


# Develop Pulsar Functions
Source: https://docs.streamnative.io/cloud/process/pulsar-functions/develop-functions/function-develop-overview



<Note title="Note">
  StreamNative Cloud supports:

  1. Java functions
  2. Python functions
  3. Golang functions(Private Preview)
  4. NodeJs functions(Private Preview)
  5. WASM functions(Private Preview)
</Note>

## What's next?

* Learn how to develop [Java functions](/cloud/process/pulsar-functions/develop-functions/function-develop-java).
* Learn how to develop [Python functions](/cloud/process/pulsar-functions/develop-functions/function-develop-python).
* Learn how to develop [Golang functions](/cloud/process/pulsar-functions/develop-functions/function-develop-golang).
* Learn how to develop [NodeJs functions](/cloud/process/pulsar-functions/develop-functions/function-develop-nodejs).
* Learn how to develop [WASM functions](/cloud/process/pulsar-functions/develop-functions/function-develop-wasm).
* Learn how to build [custom runner images](/cloud/process/pulsar-functions/develop-functions/function-custom-images) (BYOC Pro only).
* Learn how to [manage functions](/cloud/process/pulsar-functions/function-manage).
* Learn how to [configure stateful functions](/cloud/process/pulsar-functions/function-state).
* Learn how to [monitor functions](/cloud/process/pulsar-functions/function-monitoring).
* Reference [common configurations](/cloud/process/pulsar-functions/function-config).


# Develop Pulsar Functions in Python
Source: https://docs.streamnative.io/cloud/process/pulsar-functions/develop-functions/function-develop-python



This section introduces how to develop and pacakge Python Pulsar functions to use on StreamNative cloud.

## Develop

StreamNative supports all features for the Python functions, please refer to: [Develop Functions](https://pulsar.apache.org/docs/3.2.x/functions-develop/)
to learn how to develop a Python Function.

## Package

Please refer to: [Pacakge Python Functions](https://pulsar.apache.org/docs/3.2.x/functions-package-python/)

## Deploy

After creating a cluster, set up your environment and develop\&package your function, you can use the `snctl`, `pulsarctl`, `pulsar-admin` command, the REST API, or `terraform` to deploy a Pulsar function to your cluster.

You can create a Python Pulsar function by using a local `.py`, `.pip`, or `.zip` file or an uploaded Pulsar functions package(recommend).

### (Optional) Upload your function file to Pulsar

It's recommended to upload your function file to Pulsar before you create a function. Since you can add a version suffix to the package.

<Tabs>
  <Tab title="snctl">
    Upload packages

    ```bash theme={null}
    snctl pulsar admin packages upload function://${tenant}/${namespace}/${package_name} \
    --path ${file_path} \
    --description "${description}" \
    --properties fileName=${file_name}
    ```

    You should see the following output:

    ```bash theme={null}
    The package 'function://${tenant}/${namespace}/${package_name}' uploaded from path '${file_path}' successfully
    ```
  </Tab>

  <Tab title="Pulsarctl">
    You need to set the context for Pulsarctl first:

    ```bash theme={null}
    # create a context
    pulsarctl context set ${context-name}  \
    --admin-service-url ${admin-service-url} \
    --issuer-endpoint ${issuerUrl} \
    --audience urn:sn:pulsar:${orgName}:${instanceName} \
    --key-file ${privateKey}

    # activate oauth2
    pulsarctl oauth2 activate
    ```

    <Note title="Note">
      Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).

      * `context-name`: any name you want
      * `admin-service-url`: the HTTP service URL of your Pulsar cluster.
      * `privateKey`: the path to the downloaded OAuth2 key file.
      * `issuerUrl`: the URL of the OAuth2 issuer.
      * `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#uniform-resource-name-urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
        * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
        * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
    </Note>

    Upload packages

    ```bash theme={null}
    pulsarctl packages upload function://${tenant}/${namespace}/${package_name} \
    --path ${file_path} \
    --description "${description}" \
    --properties fileName=${file_name}
    ```

    You should see the following output:

    ```bash theme={null}
    The package 'function://${tenant}/${namespace}/${package_name}' uploaded from path '${file_path}' successfully
    ```
  </Tab>

  <Tab title="Pulsar-admin">
    ```bash theme={null}
    ./bin/pulsar-admin \
        --admin-url "${WEB_SERVICE_URL}" \
        --auth-plugin org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 \
        --auth-params '{"privateKey":"file://${privateKey}","issuerUrl":"${issuerUrl}","audience":"urn:sn:pulsar:${orgName}:${instanceName}}' \
        packages upload function://${tenant}/${namespace}/${package_name} \
        --path ${file_path} \
        --description "${description}" \
        --properties fileName=${file_name}
    ```

    <Note title="Note">
      Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).

      * `admin-url`: the HTTP service URL of your Pulsar cluster.
      * `privateKey`: the path to the downloaded OAuth2 key file.
      * `issuerUrl`: the URL of the OAuth2 issuer.
      * `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#uniform-resource-name-urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
        * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
        * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
    </Note>

    You should see the following output:

    ```bash theme={null}
    The package 'function://${tenant}/${namespace}/${package_name}' uploaded from path '${file_path}' successfully
    ```
  </Tab>
</Tabs>

### Create

<Tabs>
  <Tab title="snctl">
    ```bash theme={null}
    snctl pulsar admin functions create \
    --tenant public \
    --namespace default \
    --name function1 \
    --inputs persistent://public/default/test-python-input \
    --output persistent://public/default/test-python-output \
    --classname exclamation.ExclamationFunction \
    --py function://public/default/py-exclamation@v0.1 \
    --sn-service-account $SERVICE_ACCOUNT
    ```

    <Note>
      Only Pulsar 4.0.x clusters running version 4.0.10.6 or later, or Pulsar 4.2.x clusters running version 4.2.1.4 or later, support the `--sn-service-account` flag.

      You can use `--as-service-account` instead of `--sn-service-account` if you are using other versions of Pulsar clusters.
    </Note>

    You should see something like this:

    ```bash theme={null}
    Created function1 successfully
    ```
  </Tab>

  <Tab title="Pulsarctl">
    ```bash theme={null}
    pulsarctl functions create \
    --tenant public \
    --namespace default \
    --name function1 \
    --inputs persistent://public/default/test-python-input \
    --output persistent://public/default/test-python-output \
    --classname exclamation.ExclamationFunction \
    --py function://public/default/py-exclamation@v0.1
    ```

    You should see something like this:

    ```bash theme={null}
    Created function1 successfully
    ```
  </Tab>

  <Tab title="Pulsar-admin">
    ```bash theme={null}
    ./bin/pulsar-admin \
    --admin-url "${WEB_SERVICE_URL}" \
    --auth-plugin org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 \
    --auth-params '{"privateKey":"file:///YOUR-KEY-FILE-PATH","issuerUrl":"https://auth.streamnative.cloud/","audience":"urn:sn:pulsar:${orgName}:${instanceName}}' \
    functions create \
    --tenant public \
    --namespace default \
    --name function1 \
    --inputs persistent://public/default/test-python-input \
    --output persistent://public/default/test-python-output \
    --classname exclamation.ExclamationFunction \
    --py function://public/default/py-exclamation@v0.1
    ```

    You should see something like this:

    ```bash theme={null}
    Created successfully
    ```
  </Tab>

  <Tab title="Terraform">
    Create your terraform yaml file:

    ```yaml theme={null}
    terraform {
      required_providers {
        pulsar = {
          version = "0.2.0"
          source = "registry.terraform.io/streamnative/pulsar"
        }
      }
    }

    provider "pulsar" {
      web_service_url = "{$admin-url}"
      api_version     = "3"
      audience = "urn:sn:pulsar:${orgName}:${instanceName}}"
      issuer_url = "${issuerUrl}"
      key_file_path = "${privateKey}"
    }

    // Note: function resource requires v3 api.
    resource "pulsar_function" "function-1" {
      provider = pulsar

      name = "function1"
      tenant = "public"
      namespace = "default"
      parallelism = 1

      processing_guarantees = "ATLEAST_ONCE"

      py = "function://public/default/py-exclamation@v0.1"
      classname = "exclamation.ExclamationFunction"

      inputs = ["persistent://public/default/test-python-input"]

      output = "persistent://public/default/test-python-output"

      subscription_name = "test-sub"
      subscription_position = "Latest"
      cleanup_subscription = true
      skip_to_latest = true
      forward_source_message_property = true
      retain_key_ordering = true
      auto_ack = true
      max_message_retries = 100
      dead_letter_topic = "public/default/dlt"
      log_topic = "public/default/lt"
      timeout_ms = 6666

      secrets = jsonencode(
      {
        "SECRET1": {
           "path": "sectest",
           "key": "hello"
        }
      })
      custom_runtime_options = jsonencode(
      {
          "env": {
              "HELLO": "WORLD"
          },
          "snServiceAccount": "${SERVICE_ACCOUNT}"
      })
    }
    ```

    Init the terraform provider in the same dir of your `.tf` file if you haven't done it:

    ```bash theme={null}
    terraform init
    ```

    You should see something like this:

    ```bash theme={null}
    Initializing the backend...

    Initializing provider plugins...
    - Finding streamnative/pulsar versions matching "0.2.0"...
    - Installing streamnative/pulsar v0.2.0...
    - Installed streamnative/pulsar v0.2.0 (self-signed, key ID 3105E1011F3C3671)

    Partner and community providers are signed by their developers.
    If you'd like to know more about provider signing, you can read about it here:
    https://www.terraform.io/docs/cli/plugins/signing.html

    Terraform has created a lock file .terraform.lock.hcl to record the provider
    selections it made above. Include this file in your version control repository
    so that Terraform can guarantee to make the same selections by default when
    you run "terraform init" in the future.

    Terraform has been successfully initialized!

    You may now begin working with Terraform. Try running "terraform plan" to see
    any changes that are required for your infrastructure. All Terraform commands
    should now work.

    If you ever set or change modules or backend configuration for Terraform,
    rerun this command to reinitialize your working directory. If you forget, other
    commands will detect it and remind you to do so if necessary.
    ```

    Create the function:

    ```bash theme={null}
    terraform apply
    ```

    You should see something like:

    ```bash theme={null}
    Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
      + create

    Terraform will perform the following actions:

      # pulsar_function.function-1 will be created
      + resource "pulsar_function" "function-1" {
          + auto_ack                        = true
          + classname                       = "exclamation.ExclamationFunction"
          + cleanup_subscription            = true
          + cpu                             = 0.5
          + custom_runtime_options          = jsonencode(
                {
                  + env = {
                      + HELLO = "WORLD"
                    },
                  + snServiceAccount = "${SERVICE_ACCOUNT}"
                }
            )
          + dead_letter_topic               = "public/default/dlt"
          + disk_mb                         = 128
          + forward_source_message_property = true
          + id                              = (known after apply)
          + inputs                          = [
              + "persistent://public/default/test-python-input",
            ]
          + py                              = "function://public/default/py-exclamation@v0.1"
          + log_topic                       = "public/default/lt"
          + max_message_retries             = 100
          + name                            = "function1"
          + namespace                       = "default"
          + output                          = "persistent://public/default/test-python-output"
          + parallelism                     = 1
          + processing_guarantees           = "ATLEAST_ONCE"
          + ram_mb                          = 128
          + retain_key_ordering             = true
          + secrets                         = jsonencode(
                {
                  + SECRET1 = {
                      + key  = "hello"
                      + path = "sectest"
                    }
                }
            )
          + skip_to_latest                  = true
          + subscription_name               = "test-sub"
          + subscription_position           = "Latest"
          + tenant                          = "public"
          + timeout_ms                      = 6666
        }

    Plan: 1 to add, 0 to change, 0 to destroy.

    Do you want to perform these actions?
      Terraform will perform the actions described above.
      Only 'yes' will be accepted to approve.

      Enter a value:
    ```

    After enter "yes", you should see the following:

    ```bash theme={null}
    pulsar_function.function-1: Creating...
    pulsar_function.function-1: Creation complete after 1s [id=public/default/function1]

    Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
    ```

    <Note title="Note">
      Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).

      * `admin-url`: the HTTP service URL of your Pulsar cluster.
      * `privateKey`: the path to the downloaded OAuth2 key file.
      * `issuerUrl`: the URL of the OAuth2 issuer.
      * `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#uniform-resource-name-urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
        * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
        * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
    </Note>
  </Tab>

  <Tab title="REST API">
    If you would like to create a function configuration using the REST API you can
    do so using CURL.

    ```bash theme={null}
    curl -X POST ${WEB_SERVICE_URL}/admin/v3/functions/public/default/${FUNCTION_NAME} \
      -H 'Authorization: Bearer ${TOKEN}' \
      -H "Content-Type: multipart/form-data" \
      -F 'functionConfig={"name": "${FUNCTION_NAME}", "tenant": "public", "namespace": "default", "runtime": "PYTHON", "py": "function://public/default/${PACKAGE_NAME}@${PACKAGE_VERSION}", "output": "public/default/output-test", "inputs": ["public/default/input"], "className": "exclamation.ExclamationFunction"};type=application/json' \
      -F 'url=function://public/default/${PACKAGE_NAME}@${PACKAGE_VERSION}'
    ```

    <Note title="Note">
      The function is assumed to be already uploaded at this point. If you have not
      uploaded the function, change the `url` parameter to be your local filepath.
      This will look something like the following.

      ```bash theme={null}
      -F 'url=file://$YOUR_LCOAL_PYTHON_FILE'
      ```
    </Note>

    You should see something like this:

    ```bash theme={null}
    Created successfully
    ```

    <Note title="Note">
      Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).

      * `WEB_SERVICE_URL`: the HTTPS service URL of your Pulsar cluster.
      * `TOKEN`: a valid token to interact with your Pulsar cluster.
      * `FUNCTION_NAME`: the name of your function.
      * `FUNCTION_VERSION`: the version of your function you want to deploy e.g. 1.12.
    </Note>
  </Tab>
</Tabs>

For details about Pulsar function configurations, see [Pulsar function configurations](/cloud/process/pulsar-functions/function-config).

## What’s next?

* Learn how to develop [Golang functions](/cloud/process/pulsar-functions/develop-functions/function-develop-golang).
* Learn how to develop [NodeJs functions](/cloud/process/pulsar-functions/develop-functions/function-develop-nodejs).
* Learn how to develop [WASM functions](/cloud/process/pulsar-functions/develop-functions/function-develop-wasm).
* Learn how to [manage functions](/cloud/process/pulsar-functions/function-manage).
* Learn how to [configure stateful functions](/cloud/process/pulsar-functions/function-state).
* Learn how to [monitor functions](/cloud/process/pulsar-functions/function-monitoring).
* Reference [common configurations](/cloud/process/pulsar-functions/function-config).


# Develop Pulsar Functions in WASM(Private Preview)
Source: https://docs.streamnative.io/cloud/process/pulsar-functions/develop-functions/function-develop-wasm



This section introduces how to develop and pacakge WASM Pulsar functions to use on StreamNative cloud.

The WASM runtime is still in private preview stage, If you want to try it out or have any questions, please [submit a ticket](https://support.streamnative.io/hc/en-us/requests/new) to the support team.

## Develop

The WASM runtime is using [WasmEdge](https://github.com/WasmEdge/WasmEdge), theoretically, you can use any languages which
can be compiled to a WASM module to write your functions, below is an example using Rust:

* cargo.toml

```toml theme={null}
[package]
name = "excla"
version = "0.1.0"
edition = "2021"

[lib]
crate-type = ["cdylib"]

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
apache-avro = { version = "0.15.0", features = ["bzip", "xz", "snappy", "zstandard"] }
serde = "^1.0"
serde_json = "^1.0"
wasmedge-bindgen = "0.4.1"
wasmedge-bindgen-macro = "0.4.1"
```

* src/lib.rs

```rust theme={null}
#[allow(unused_imports)]
use wasmedge_bindgen::*;
use wasmedge_bindgen_macro::*;

use serde::{Deserialize, Serialize};

#[derive(Debug, Deserialize,  Serialize)]
pub struct Student {
    pub name: Option<String>,
    pub age: Option<i32>,
    pub grade: Option<i32>
}

// use `process_json` as the class name when creating functions
#[wasmedge_bindgen]
pub fn process_json(s: Vec<u8>) -> Vec<u8> {
    let stu = serde_json::from_slice::<Student>(&s[..]).unwrap();
    let stu = Student {
        grade: stu.grade.map(|grade| grade + 1),
        ..stu
    };
    let res = serde_json::to_vec(&stu).unwrap();
    return res
}
```

### Feature Matrix

The WASM runtime doesn't support full features comparing to Java runtime, and it's still in developing, below is the matrix:

#### Input Arguments

| Input                       | Java | Go(Pulsar) | Python | WASM  |
| :-------------------------- | :--- | :--------- | :----- | :---- |
| Custom SerDe                | ✅    | ❌          | ✅      | **?** |
| Schema - Avro               | ✅    | ❌          | ✅      | **?** |
| Schema - JSON               | ✅    | ❌          | ✅      | **?** |
| Schema - Protobuf           | ✅    | ❌          | ❌      | **?** |
| Schema - KeyValue           | ✅    | ❌          | ❌      | **?** |
| Schema - AutoSchema         | ✅    | ❌          | ❌      | **?** |
| Scehma - Protobuf Native    | ✅    | ❌          | ❌      | **?** |
| e-2-e encryption            | ✅    | ❌          | ✅      | ✅     |
| maxMessageRetries           | ✅    | ❌          | ❌      | ✅     |
| dead-letter policy          | ✅    | ❌          | ❌      | ✅     |
| SubscriptionName            | ✅    | ✅          | ✅      | ✅     |
| SubscriptionType            | ✅    | ✅          | ✅      | ✅     |
| SubscriptionInitialPosition | ✅    | ❌          | ✅      | ✅     |
| AutoAck                     | ✅    | ✅          | ✅      | ✅     |

<Note title="Note">
  Users can implement the Schema themselves since we are passing and expecting \[]byte to/from the users' function, so leave **?** here.
</Note>

#### Output Arguments

| Output                   | Java | Go(Pulsar) | Python | WASM  |
| :----------------------- | :--- | :--------- | :----- | :---- |
| Custom SerDe             | ✅    | ❌          | ✅      | **?** |
| Schema - Avro            | ✅    | ❌          | ✅      | **?** |
| Schema - JSON            | ✅    | ❌          | ✅      | **?** |
| Schema - Protobuf        | ✅    | ❌          | ❌      | **?** |
| Schema - KeyValue        | ✅    | ❌          | ❌      | **?** |
| Schema - AutoSchema      | ✅    | ❌          | ❌      | **?** |
| Schema - Protobuf Native | ✅    | ❌          | ❌      | **?** |
| useThreadLocalProducers  | ✅    | ❌          | ❌      | ✅     |
| Key-based Batcher        | ✅    | ✅          | ✅      | ✅     |
| e-2-e encryption         | ✅    | ❌          | ✅      | ✅     |
| Compression              | ✅    | ✅          | ✅      | ✅     |

#### Context

WASM runtime doesn't support the **Context** features at all for now.

#### Other

| Other            | Java | Go(Pulsar) | Python | WASM |
| :--------------- | :--- | :--------- | :----- | :--- |
| Resources        | ✅    | ✅          | ✅      | ✅    |
| At-most-once     | ✅    | ✅          | ✅      | ✅    |
| At-least-once    | ✅    | ✅          | ✅      | ✅    |
| Effectively-once | ✅    | ❌          | ✅      | ❌    |

## Package

You need to compile the function to a `.wasm` module first before creating Pulsar Functions.

```bash theme={null}
cargo build --target wasm32-wasi --release
```

## Deploy

After creating a cluster, set up your environment and develop\&package your function, you can use the `snctl`, `pulsarctl`, `pulsar-admin` command, the REST API, or `terraform` to deploy a Pulsar function to your cluster.

You can create a WASM Pulsar function by using a local `.wasm` file or an uploaded Pulsar functions package(recommend).

### (Optional) Upload your function file to Pulsar

It's recommended to upload your function file to Pulsar before you create a function. Since you can add a version suffix to the package.

<Tabs>
  <Tab title="snctl">
    Upload packages

    ```bash theme={null}
    snctl pulsar admin packages upload function://${tenant}/${namespace}/${package_name} \
    --path ${file_path} \
    --description "${description}" \
    --properties fileName=${file_name}
    ```

    You should see the following output:

    ```bash theme={null}
    The package 'function://${tenant}/${namespace}/${package_name}' uploaded from path '${file_path}' successfully
    ```
  </Tab>

  <Tab title="Pulsarctl">
    You need to set the context for Pulsarctl first:

    ```bash theme={null}
    # create a context
    pulsarctl context set ${context-name}  \
    --admin-service-url ${admin-service-url} \
    --issuer-endpoint ${issuerUrl} \
    --audience urn:sn:pulsar:${orgName}:${instanceName} \
    --key-file ${privateKey}

    # activate oauth2
    pulsarctl oauth2 activate
    ```

    <Note title="Note">
      Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).

      * `context-name`: any name you want
      * `admin-service-url`: the HTTP service URL of your Pulsar cluster.
      * `privateKey`: the path to the downloaded OAuth2 key file.
      * `issuerUrl`: the URL of the OAuth2 issuer.
      * `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#uniform-resource-name-urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
        * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
        * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
    </Note>

    Upload packages

    ```bash theme={null}
    pulsarctl packages upload function://${tenant}/${namespace}/${package_name} \
    --path ${file_path} \
    --description "${description}" \
    --properties fileName=${file_name}
    ```

    You should see the following output:

    ```bash theme={null}
    The package 'function://${tenant}/${namespace}/${package_name}' uploaded from path '${file_path}' successfully
    ```
  </Tab>

  <Tab title="Pulsar-admin">
    ```bash theme={null}
    ./bin/pulsar-admin \
        --admin-url "${WEB_SERVICE_URL}" \
        --auth-plugin org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 \
        --auth-params '{"privateKey":"file://${privateKey}","issuerUrl":"${issuerUrl}","audience":"urn:sn:pulsar:${orgName}:${instanceName}}' \
        packages upload function://${tenant}/${namespace}/${package_name} \
        --path ${file_path} \
        --description "${description}" \
        --properties fileName=${file_name}
    ```

    <Note title="Note">
      Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).

      * `admin-url`: the HTTP service URL of your Pulsar cluster.
      * `privateKey`: the path to the downloaded OAuth2 key file.
      * `issuerUrl`: the URL of the OAuth2 issuer.
      * `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#uniform-resource-name-urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
        * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
        * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
    </Note>

    You should see the following output:

    ```bash theme={null}
    The package 'function://${tenant}/${namespace}/${package_name}' uploaded from path '${file_path}' successfully
    ```
  </Tab>
</Tabs>

### Create

<Tabs>
  <Tab title="snctl">
    ```bash theme={null}
    snctl pulsar admin functions create \
    --tenant public \
    --namespace default \
    --name function1 \
    --inputs persistent://public/default/test-wasm-input \
    --output persistent://public/default/test-wasm-output \
    --classname exclamation \
    --py function://public/default/wasm-exclamation@v0.1 \
    --custom-runtime-options '{"genericKind": "wasm"}' \
    --sn-service-account $SERVICE_ACCOUNT
    ```

    <Note>
      Only Pulsar 4.0.x clusters running version 4.0.10.6 or later, or Pulsar 4.2.x clusters running version 4.2.1.4 or later, support the `--sn-service-account` flag.

      You can use `--as-service-account` instead of `--sn-service-account` if you are using other versions of Pulsar clusters.
    </Note>

    <Note title="Note">
      Since Pulsar doesn't support WASM runtime, we need to use `--py` to specify the function file and specify the `--custom-runtime-options '{"genericKind": "wasm"}'` to make it work.
    </Note>

    You should see something like this:

    ```bash theme={null}
    Created function1 successfully
    ```
  </Tab>

  <Tab title="Pulsarctl">
    ```bash theme={null}
    pulsarctl functions create \
    --tenant public \
    --namespace default \
    --name function1 \
    --inputs persistent://public/default/test-wasm-input \
    --output persistent://public/default/test-wasm-output \
    --classname exclamation \
    --py function://public/default/wasm-exclamation@v0.1 \
    --custom-runtime-options '{"genericKind": "wasm"}'
    ```

    <Note title="Note">
      Since Pulsar doesn't support WASM runtime, we need to use `--py` to specify the function file and specify the `--custom-runtime-options '{"genericKind": "wasm"}'` to make it work.
    </Note>

    You should see something like this:

    ```bash theme={null}
    Created function1 successfully
    ```
  </Tab>

  <Tab title="Pulsar-admin">
    ```bash theme={null}
    ./bin/pulsar-admin \
    --admin-url "${WEB_SERVICE_URL}" \
    --auth-plugin org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 \
    --auth-params '{"privateKey":"file:///YOUR-KEY-FILE-PATH","issuerUrl":"https://auth.streamnative.cloud/","audience":"urn:sn:pulsar:${orgName}:${instanceName}}' \
    functions create \
    --tenant public \
    --namespace default \
    --name function1 \
    --inputs persistent://public/default/test-wasm-input \
    --output persistent://public/default/test-wasm-output \
    --classname exclamation \
    --py function://public/default/wasm-exclamation@v0.1 \
    --custom-runtime-options '{"genericKind": "wasm"}'
    ```

    <Note title="Note">
      Since Pulsar doesn't support WASM runtime, we need to use `--py` to specify the function file and specify the `--custom-runtime-options '{"genericKind": "wasm"}'` to make it work.
    </Note>

    You should see something like this:

    ```bash theme={null}
    Created successfully
    ```
  </Tab>

  <Tab title="Terraform">
    Create your terraform yaml file:

    ```yaml theme={null}
    terraform {
      required_providers {
        pulsar = {
          version = "0.2.0"
          source = "registry.terraform.io/streamnative/pulsar"
        }
      }
    }

    provider "pulsar" {
      web_service_url = "{$admin-url}"
      api_version     = "3"
      audience = "urn:sn:pulsar:${orgName}:${instanceName}}"
      issuer_url = "${issuerUrl}"
      key_file_path = "${privateKey}"
    }

    // Note: function resource requires v3 api.
    resource "pulsar_function" "function-1" {
      provider = pulsar

      name = "function1"
      tenant = "public"
      namespace = "default"
      parallelism = 1

      processing_guarantees = "ATLEAST_ONCE"

      py = "function://public/default/wasm-exclamation@v0.1"
      classname = "exclamation.ExclamationFunction"

      inputs = ["persistent://public/default/test-wasm-input"]

      output = "persistent://public/default/test-wasm-output"

      subscription_name = "test-sub"
      subscription_position = "Latest"
      cleanup_subscription = true
      skip_to_latest = true
      forward_source_message_property = true
      retain_key_ordering = true
      auto_ack = true
      max_message_retries = 100
      dead_letter_topic = "public/default/dlt"
      log_topic = "public/default/lt"
      timeout_ms = 6666

      secrets = jsonencode(
      {
        "SECRET1": {
           "path": "sectest",
           "key": "hello"
        }
      })
      custom_runtime_options = jsonencode(
      {
         "genericKind": "wasm",
          "env": {
              "HELLO": "WORLD"
          },
          "snServiceAccount": "${SERVICE_ACCOUNT}"
      })
    }
    ```

    <Note title="Note">
      Since Pulsar doesn't support WASM runtime, we need to use `--py` to specify the function file and specify the `--custom-runtime-options '{"genericKind": "wasm"}'` to make it work.
    </Note>

    Init the terraform provider in the same dir of your `.tf` file if you haven't done it:

    ```bash theme={null}
    terraform init
    ```

    You should see something like this:

    ```bash theme={null}
    Initializing the backend...

    Initializing provider plugins...
    - Finding streamnative/pulsar versions matching "0.2.0"...
    - Installing streamnative/pulsar v0.2.0...
    - Installed streamnative/pulsar v0.2.0 (self-signed, key ID 3105E1011F3C3671)

    Partner and community providers are signed by their developers.
    If you'd like to know more about provider signing, you can read about it here:
    https://www.terraform.io/docs/cli/plugins/signing.html

    Terraform has created a lock file .terraform.lock.hcl to record the provider
    selections it made above. Include this file in your version control repository
    so that Terraform can guarantee to make the same selections by default when
    you run "terraform init" in the future.

    Terraform has been successfully initialized!

    You may now begin working with Terraform. Try running "terraform plan" to see
    any changes that are required for your infrastructure. All Terraform commands
    should now work.

    If you ever set or change modules or backend configuration for Terraform,
    rerun this command to reinitialize your working directory. If you forget, other
    commands will detect it and remind you to do so if necessary.
    ```

    Create the function:

    ```bash theme={null}
    terraform apply
    ```

    You should see something like:

    ```bash theme={null}
    Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
      + create

    Terraform will perform the following actions:

      # pulsar_function.function-1 will be created
      + resource "pulsar_function" "function-1" {
          + auto_ack                        = true
          + classname                       = "exclamation.ExclamationFunction"
          + cleanup_subscription            = true
          + cpu                             = 0.5
          + custom_runtime_options          = jsonencode(
                {
                  + genericKind = "wasm",
                  + env = {
                      + HELLO = "WORLD"
                    },
                  + snServiceAccount = "${SERVICE_ACCOUNT}"
                }
            )
          + dead_letter_topic               = "public/default/dlt"
          + disk_mb                         = 128
          + forward_source_message_property = true
          + id                              = (known after apply)
          + inputs                          = [
              + "persistent://public/default/test-wasm-input",
            ]
          + py                              = "function://public/default/wasm-exclamation@v0.1"
          + log_topic                       = "public/default/lt"
          + max_message_retries             = 100
          + name                            = "function1"
          + namespace                       = "default"
          + output                          = "persistent://public/default/test-wasm-output"
          + parallelism                     = 1
          + processing_guarantees           = "ATLEAST_ONCE"
          + ram_mb                          = 128
          + retain_key_ordering             = true
          + secrets                         = jsonencode(
                {
                  + SECRET1 = {
                      + key  = "hello"
                      + path = "sectest"
                    }
                }
            )
          + skip_to_latest                  = true
          + subscription_name               = "test-sub"
          + subscription_position           = "Latest"
          + tenant                          = "public"
          + timeout_ms                      = 6666
        }

    Plan: 1 to add, 0 to change, 0 to destroy.

    Do you want to perform these actions?
      Terraform will perform the actions described above.
      Only 'yes' will be accepted to approve.

      Enter a value:
    ```

    After enter "yes", you should see the following:

    ```bash theme={null}
    pulsar_function.function-1: Creating...
    pulsar_function.function-1: Creation complete after 1s [id=public/default/function1]

    Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
    ```

    <Note title="Note">
      Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).

      * `admin-url`: the HTTP service URL of your Pulsar cluster.
      * `privateKey`: the path to the downloaded OAuth2 key file.
      * `issuerUrl`: the URL of the OAuth2 issuer.
      * `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#uniform-resource-name-urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
        * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
        * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
    </Note>
  </Tab>

  <Tab title="REST API">
    If you would like to create a function configuration using the REST API you can
    do so using CURL.

    ```bash theme={null}
    curl -X POST ${WEB_SERVICE_URL}/admin/v3/functions/public/default/${FUNCTION_NAME} \
      -H 'Authorization: Bearer ${TOKEN}' \
      -H "Content-Type: multipart/form-data" \
      -F 'functionConfig={"name": "${FUNCTION_NAME}", "tenant": "public", "namespace": "default", "runtime": "PYTHON", "py": "function://public/default/${PACKAGE_NAME}@${PACKAGE_VERSION}", "output": "public/default/output-test", "inputs": ["public/default/input"], "className": "exclamation", "customRuntimeOptions": "{\"genericKind\": \"wasm\"}"};type=application/json' \
      -F 'url=function://public/default/${PACKAGE_NAME}@${PACKAGE_VERSION}'
    ```

    <Note title="Note">
      Since Pulsar doesn't support WASM runtime, we need to use `--py` to specify the function file and specify the `--custom-runtime-options '{"genericKind": "wasm"}'` to make it work.
    </Note>

    <Note title="Note">
      The function is assumed to be already uploaded at this point. If you have not
      uploaded the function, change the `url` parameter to be your local filepath.
      This will look something like the following.

      ```bash theme={null}
      -F 'url=file://$YOUR_LCOAL_FUNCTION_FILE'
      ```
    </Note>

    You should see something like this:

    ```bash theme={null}
    Created successfully
    ```

    <Note title="Note">
      Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).

      * `WEB_SERVICE_URL`: the HTTPS service URL of your Pulsar cluster.
      * `TOKEN`: a valid token to interact with your Pulsar cluster.
      * `FUNCTION_NAME`: the name of your function.
      * `FUNCTION_VERSION`: the version of your function you want to deploy e.g. 1.12.
    </Note>
  </Tab>
</Tabs>

For details about Pulsar function configurations, see [Pulsar function configurations](/cloud/process/pulsar-functions/function-config).

## What’s next?

* Learn how to [manage functions](/cloud/process/pulsar-functions/function-manage).
* Learn how to [configure stateful functions](/cloud/process/pulsar-functions/function-state).
* Learn how to [monitor functions](/cloud/process/pulsar-functions/function-monitoring).
* Reference [common configurations](/cloud/process/pulsar-functions/function-config).


# Configuration Reference
Source: https://docs.streamnative.io/cloud/process/pulsar-functions/function-config



## Pulsar function configurations

This table lists all fields available for creating a Pulsar function.

| Field                          | Description                                                                                                                                                                                                                                                    | Default          |
| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- |
| `auto-ack`                     | Whether or not the framework acknowledges messages automatically.                                                                                                                                                                                              | true             |
| `classname`                    | The class name of a Pulsar function.                                                                                                                                                                                                                           |                  |
| `CPU`                          | The CPU in cores that need to be allocated per function instance (applicable only to docker runtime).                                                                                                                                                          |                  |
| `custom-runtime-options`       | A string that encodes options to customize the runtime.                                                                                                                                                                                                        |                  |
| `custom-schema-inputs`         | The map of input topics to Schema class names (as a JSON string).                                                                                                                                                                                              |                  |
| `custom-serde-inputs`          | The map of input topics to SerDe class names (as a JSON string).                                                                                                                                                                                               |                  |
| `dead-letter-topic`            | The topic where all messages that were not processed successfully are sent. This parameter is not supported in Python Functions.                                                                                                                               |                  |
| `disk`                         | The disk in bytes that need to be allocated per function instance (applicable only to docker runtime).                                                                                                                                                         |                  |
| `fqfn`                         | The Fully Qualified Function Name (FQFN) for the function.                                                                                                                                                                                                     |                  |
| `function-config-file`         | The path to a YAML config file specifying the configuration of a Pulsar function.                                                                                                                                                                              |                  |
| `go`                           | Path to the main Go executable binary for the function (if the function is written in Go). Go Functions are not supported in StreamNative Cloud.                                                                                                               |                  |
| `inputs`                       | The input topic or topics of a Pulsar function (multiple topics can be specified as a comma-separated list).                                                                                                                                                   |                  |
| `jar`                          | Path to the jar file for the function (if the function is written in Java). It also supports URL-path \[http/https/file (file protocol assumes that file already exists on worker host)] from which worker can download the package.                           |                  |
| `log-topic`                    | The topic to which the logs of a Pulsar function are produced.                                                                                                                                                                                                 |                  |
| `max-message-retries`          | How many times should we try to process a message before giving up.                                                                                                                                                                                            |                  |
| `name`                         | The name of a Pulsar function.                                                                                                                                                                                                                                 |                  |
| `namespace`                    | The namespace of a Pulsar function.                                                                                                                                                                                                                            |                  |
| `output`                       | The output topic of a Pulsar function (If none is specified, no output is written).                                                                                                                                                                            |                  |
| `output-serde-classname`       | The SerDe class to be used for messages output by the function.                                                                                                                                                                                                |                  |
| `parallelism`                  | The parallelism factor of a Pulsar function (i.e. the number of function instances to run).                                                                                                                                                                    |                  |
| `processing-guarantees`        | The processing guarantees (delivery semantics) applied to the function. Available values: \[ATLEAST\_ONCE, ATMOST\_ONCE, EFFECTIVELY\_ONCE].                                                                                                                   | ATLEAST\_ONCE    |
| `py`                           | Path to the main Python file/Python Wheel file for the function (if the function is written in Python).                                                                                                                                                        |                  |
| `ram`                          | The ram in bytes that need to be allocated per function instance (applicable only to process/docker runtime).                                                                                                                                                  |                  |
| `retain-ordering`              | Function consumes and processes messages in order.                                                                                                                                                                                                             |                  |
| `schema-type`                  | The builtin schema type or custom schema class name to be used for messages output by the function.                                                                                                                                                            | `<empty string>` |
| `sliding-interval-count`       | The number of messages after which the window slides.                                                                                                                                                                                                          |                  |
| `sliding-interval-duration-ms` | The time duration after which the window slides.                                                                                                                                                                                                               |                  |
| `subs-name`                    | Pulsar source subscription name if user wants a specific subscription-name for the input-topic consumer.                                                                                                                                                       |                  |
| `tenant`                       | The tenant of a Pulsar function.                                                                                                                                                                                                                               |                  |
| `timeout-ms`                   | The message timeout in milliseconds.                                                                                                                                                                                                                           |                  |
| `topics-pattern`               | The topic pattern to consume from a list of topics under a namespace that matches the pattern. \[--input] and \[--topic-pattern] are mutually exclusive. Add SerDe class name for a pattern in --custom-serde-inputs (only supported in Java Pulsar function). |                  |
| `user-config`                  | User-defined config key/values.                                                                                                                                                                                                                                |                  |
| `window-length-count`          | The number of messages per window.                                                                                                                                                                                                                             |                  |
| `window-length-duration-ms`    | The time duration of the window is milliseconds.                                                                                                                                                                                                               |                  |

## StreamNative Cloud custom runtime options

To facilitate submitting Pulsar functions based on your requirements, Function on Cloud service provides some custom options via `custom-runtime-options`.

<Note title="Note">
  When update functions/sinks/sources with custom runtime options, the original custom runtime options will be replaced by the new ones, so make sure all the wanted fields are passed in the custom runtime options when you do the update.
</Note>

This table lists all fields available for custom options.

| Name                            | Type                  | Default | Description                                                                                                                                                                                                                                                                                                                                                                                                                              |
| ------------------------------- | --------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `clusterName`                   | String                | N/A     | The Pulsar cluster of a Pulsar function, source, or sink.                                                                                                                                                                                                                                                                                                                                                                                |
| `inputTypeClassName`            | String                | `[B`    | The map of input topics to Java class names.                                                                                                                                                                                                                                                                                                                                                                                             |
| `outputTypeClassName`           | String                | `[B`    | The map of output topics to Java class names.                                                                                                                                                                                                                                                                                                                                                                                            |
| `maxReplicas`                   | Integer               | `0`     | The maximum number of Pulsar instances that you want to run for this Pulsar Function. When the value of the `maxReplicas` parameter is greater than the value of `replicas`, it indicates that the Functions controller automatically scales the Pulsar Functions based on the CPU usage. By default, `maxReplicas` is set to 0, which indicates that auto-scaling is disabled.                                                          |
| `env`                           | `Map<String, String>` | N/A     | The environment variables being attached to a Pod that is created by the Function Mesh Operator for the cluster.                                                                                                                                                                                                                                                                                                                         |
| `imagePullSecrets`              | `List<String>`        | N/A     | A list of references to secrets in the same namespace for pulling any of the images used by a Pod.                                                                                                                                                                                                                                                                                                                                       |
| `logLevel`                      | String                | info    | The log levels for Pulsar functions. For details, see [log levels](https://functionmesh.io/docs/next/reference/crd-config/function-crd#log-levels).                                                                                                                                                                                                                                                                                      |
| `logRotationPolicy`             | String                | N/A     | The log rotation policies for Pulsar functions. You can set the log rotation policies based on the time or the log file size. For details, see [log rotation policies](https://functionmesh.io/docs/next/reference/crd-config/function-crd#log-rotation-policies).                                                                                                                                                                       |
| `runnerImageTag`                | String                | N/A     | The tag of the [runner image](https://functionmesh.io/docs/next/reference/crd-config/function-crd#runner-images) that is used to submit a function, source, or sink.                                                                                                                                                                                                                                                                     |
| `hpaSpec` (Preview)             | HPASpec               | N/A     | The Kubernetes HorizontalPodAutoscaler settings. For details, see [Kubernetes documentation](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/). This feature may not ready for your cluster enviroment, please file a ticket if you want this feature enabled.                                                                                                                                                 |
| `logFormat`                     | String                | text    | The log format that defines how the content of a log file should be interpreted. Available options are ` json` and `text`. The log format configurations are only available for the Java and Python runtimes.                                                                                                                                                                                                                            |
| `logTopic`                      | String                | N/A     | Used for sinks/sources since they don't have a log topic argument like functions                                                                                                                                                                                                                                                                                                                                                         |
| `logTopicAgent`                 | String                | runtime | The log agent that defines how StreamNative cloud redirects your functions / connectors log into a Pulsar topic. Available options are `runtime` and `sidecar`. When use `sidecar` all logs (including instance logs) will be sent to the log topic by filebeat (better performance than `runtime`), else will use the Pulsar Functions runtime's log-topic implementation instead.                                                      |
| `genericKind`                   | String                | N/A     | Used for functions written in languages other than Java and Python, available values are `executable`, `nodejs`, `wasm`                                                                                                                                                                                                                                                                                                                  |
| `snServiceAccount`              | String                | N/A     | The StreamNative Cloud service account that the function, source, or sink uses as its runtime identity. You must have the `cloud.serviceaccounts.describe` permission on the specified service account (see [account-admin](/cloud/security/access/rbac/manage-rbac-roles#account-admin)). Only Pulsar 4.0.x clusters running version 4.0.10.6 or later, or Pulsar 4.2.x clusters running version 4.2.1.4 or later, support this config. |
| `terminationGracePeriodSeconds` | Long                  | N/A     | The amount of time that kubernetes will give for a pod before terminating it.                                                                                                                                                                                                                                                                                                                                                            |
| `pauseRollout`                  | Boolean               | N/A     | Whether to pause the rollout of functions/connectors, when set to true, running functions\&connectors will not restart during the upgrade of backend function-mesh operator.                                                                                                                                                                                                                                                             |
| `enableStateStore`              | Boolean               | N/A     | Whether to enable stateful backend for this function, default to false, only available for **Java** function                                                                                                                                                                                                                                                                                                                             |

You can compose the custom runtime options as a JSON string and pass it to the `custom-runtime-options` field. For example:

```json theme={null}
{
  "inputTypeClassName": "java.lang.String",
  "outputTypeClassName": "java.lang.String",
  "maxReplicas": 0
}
```

Then pass it to the `custom-runtime-options` field as follows:

* Using `snctl`:

```bash theme={null}
snctl pulsar admin functions create --custom-runtime-options '{"inputTypeClassName":"java.lang.String","outputTypeClassName":"java.lang.String","maxReplicas":0}' ...
```

* Using `pulsarctl`:

```bash theme={null}
pulsarctl functions create --custom-runtime-options '{"inputTypeClassName":"java.lang.String","outputTypeClassName":"java.lang.String","maxReplicas":0}' ...
```

### Run a function as a service account

To run a function with a specific StreamNative Cloud service account, set the `snServiceAccount` custom runtime option. The selected service account becomes the runtime identity for the function.

With `snctl`, use `--sn-service-account` on `functions create` or `functions update`:

```bash theme={null}
snctl pulsar admin functions create \
  --name function1 \
  --inputs persistent://public/default/input \
  --output persistent://public/default/output \
  --jar function://public/default/exclamation@v0.1 \
  --sn-service-account $SERVICE_ACCOUNT
```

You can also use `--use-sn-service-account` to select the runtime service account interactively.

With `pulsarctl` or `pulsar-admin`, set the same value in `custom-runtime-options`:

```bash theme={null}
pulsarctl functions create \
  --name function1 \
  --inputs persistent://public/default/input \
  --output persistent://public/default/output \
  --jar function://public/default/exclamation@v0.1 \
  --custom-runtime-options "{\"snServiceAccount\":\"${SERVICE_ACCOUNT}\"}"
```

<Note>
  Only Pulsar 4.0.x clusters running version 4.0.10.6 or later, or Pulsar 4.2.x clusters running version 4.2.1.4 or later, support runtime service account binding for functions.
</Note>

## Trusted Mode Configuration

<Note title="BYOC Pro Feature">
  This feature is available for **BYOC Pro clusters only**. To enable Trusted Mode:

  1. [Submit a support ticket](https://support.streamnative.io/hc/en-us/requests/new) through StreamNative support
  2. StreamNative support team will activate trusted mode for your cluster
  3. **Important**: Enabling trusted mode requires a broker restart
</Note>

<Warning title="Service Impact">
  Enabling trusted mode causes a **broker restart**, which temporarily interrupts message processing. Plan this activation during a maintenance window.
</Warning>

When trusted mode is enabled, BYOC Pro users can configure additional advanced options through the `custom-runtime-options` field. These options provide fine-grained control over Kubernetes pod configuration and runtime behavior.

<Warning title="Configuration Impact">
  Trusted mode exposes low-level configuration options. **Incorrect configuration of these parameters may affect the normal operation of Pulsar functions and IO connectors.** Ensure you understand the implications of each setting before applying them to production workloads.
</Warning>

### Trusted Mode Configuration Options

| Field                              | Type                        | Description                                                                                                                                                                                               |
| ---------------------------------- | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `runnerImage`                      | String                      | Custom runner image for the function. Overrides the default StreamNative runner image. For details, see [custom runner images](/cloud/process/pulsar-functions/develop-functions/function-custom-images). |
| `functionSpecVolumeClaimTemplates` | `List<VolumeClaimTemplate>` | Persistent volume claim templates for function pods. Enables persistent storage for functions.                                                                                                            |
| `javaOPTs`                         | `List<String>`              | JVM options for Java functions. Allows tuning of JVM parameters. **Note**: Some system-reserved parameters cannot be overridden (see restrictions below).                                                 |

### Trusted Mode Configuration Example

```json theme={null}
{
  "inputTypeClassName": "java.lang.String",
  "outputTypeClassName": "java.lang.String", 
  "maxReplicas": 3,
  "runnerImage": "my-registry.com/custom-runner:v1.0",
  "javaOPTs": ["-Dmy.custom.property=value"]
}
```

### javaOPTs Restrictions

<Warning title="System Reserved Parameters">
  The following JVM parameters are **system-reserved** and cannot be overridden in `javaOPTs`:

  * `-Dpulsar.functions.extra.dependencies.dir`
  * `-Dpulsar.log`
  * `-Dbk.log`
  * `-Dpulsar.function.log`
  * `-Dpulsar.functions.instance.classpath`
  * `-Djava.io.tmpdir`
  * `-Xbootclasspath`
  * `-Dlog4j.configurationFile`
  * `-XX:InitialRAMPercentage`
  * `-XX:MaxRAMPercentage`
  * `-Xmx`
  * `-Xms`
  * `-XX:MaxDirectMemorySize`
  * `agentlib:jdwp`
  * `-Djava.security.manager`
  * `-Djava.security.policy`

  Attempting to override these parameters will result in configuration errors.
</Warning>

Usage with trusted mode configuration:

```bash theme={null}
snctl pulsar admin functions create \
  --custom-runtime-options '{"javaOPTs":["-Dmy.custom.property=value"]}' \
  --other-function-options...
```


# Manage Functions
Source: https://docs.streamnative.io/cloud/process/pulsar-functions/function-manage



StreamNative Cloud enables you to manage Pulsar Functions by using a variety of tools, including `snctl`, `pulsarctl`, `pulsar-admin`, `REST API`, and `Terraform`.

<Note title="Note">
  If you want to update or delete functions using `snctl`, `pulsarctl` or `pulsar-admin`, make sure you have set up your client tool. For more information, see [set up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).
</Note>

## Update a function

When you want to modify configurations or update resources for functions, you can update functions using multiple tools.

The following example shows how to update the parallelism of the Java exclamation function `exclamation` to `2` using different tools.

<Tabs>
  <Tab title="snctl">
    ```bash theme={null}
    snctl pulsar admin functions update \
    --name exclamation \
    --parallelism 2
    ```

    You should see the following output:

    ```bash theme={null}
    Updated successfully
    ```

    And you can further check the status:

    ```bash theme={null}
    snctl pulsar admin functions status --name exclamation

    {
      "numInstances": 2,
      "numRunning": 2,
      "instances": [
        {
          "instanceId": 0,
          "status": {
            "running": true,
            "error": "",
            "numRestarts": 0,
            "numReceivedFromSource": 1799,
            "numSystemExceptions": 0,
            "latestSystemExceptions": [],
            "numSourceExceptions": 0,
            "latestSourceExceptions": [],
            "numWritten": 1799,
            "lastReceivedTime": 1693946327331,
            "workerId": "test"
          }
        },
        {
          "instanceId": 1,
          "status": {
            "running": true,
            "error": "",
            "numRestarts": 0,
            "numReceivedFromSource": 689,
            "numSystemExceptions": 0,
            "latestSystemExceptions": [],
            "numSourceExceptions": 0,
            "latestSourceExceptions": [],
            "numWritten": 689,
            "lastReceivedTime": 1693946327129,
            "workerId": "test"
          }
        }
      ]
    }

    ```
  </Tab>

  <Tab title="pulsarctl">
    ```bash theme={null}
    pulsarctl functions update \
    --name exclamation \
    --parallelism 2
    ```

    You should see the following output:

    ```bash theme={null}
    Updated successfully
    ```

    And you can further check the status:

    ```bash theme={null}
    pulsarctl functions status --name exclamation

    {
      "numInstances": 2,
      "numRunning": 2,
      "instances": [
        {
          "instanceId": 0,
          "status": {
            "running": true,
            "error": "",
            "numRestarts": 0,
            "numReceivedFromSource": 1799,
            "numSystemExceptions": 0,
            "latestSystemExceptions": [],
            "numSourceExceptions": 0,
            "latestSourceExceptions": [],
            "numWritten": 1799,
            "lastReceivedTime": 1693946327331,
            "workerId": "test"
          }
        },
        {
          "instanceId": 1,
          "status": {
            "running": true,
            "error": "",
            "numRestarts": 0,
            "numReceivedFromSource": 689,
            "numSystemExceptions": 0,
            "latestSystemExceptions": [],
            "numSourceExceptions": 0,
            "latestSourceExceptions": [],
            "numWritten": 689,
            "lastReceivedTime": 1693946327129,
            "workerId": "test"
          }
        }
      ]
    }

    ```
  </Tab>

  <Tab title="pulsar-admin">
    ```bash theme={null}
    pulsar-admin \
        --admin-url "${WEB_SERVICE_URL}" \
        --auth-plugin org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 \
        --auth-params '{"privateKey":"file:///YOUR-KEY-FILE-PATH",
            "issuerUrl":"https://auth.streamnative.cloud/",
            "audience":"urn:sn:pulsar:${orgName}:${instanceName}}'
    functions update \
    --name exclamation \
    --parallelism 2
    ```

    Replace the placeholder variables with the actual values, which you can get when you set up client tools.

    * `admin-url`: the HTTP service URL of your Pulsar cluster.
    * `private_key`: the path to the downloaded OAuth2 key file.
    * `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#uniform-resource-name-urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
      * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
      * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).

    You should see the following output:

    ```bash theme={null}
    Updated successfully
    ```

    And you can further check the status:

    ```bash theme={null}
    pulsar-admin functions status --name test
    {
      "numInstances" : 2,
      "numRunning" : 2,
      "instances" : [ {
        "instanceId" : 0,
        "status" : {
          "running" : true,
          "error" : "",
          "numRestarts" : 0,
          "numReceivedFromSource" : 1287,
          "numSystemExceptions" : 0,
          "latestSystemExceptions" : [ ],
          "numSourceExceptions" : 0,
          "latestSourceExceptions" : [ ],
          "numWritten" : 1287,
          "lastReceivedTime" : 1693946605095,
          "workerId" : "test"
        }
      }, {
        "instanceId" : 1,
        "status" : {
          "running" : true,
          "error" : "",
          "numRestarts" : 0,
          "numReceivedFromSource" : 909,
          "numSystemExceptions" : 0,
          "latestSystemExceptions" : [ ],
          "numSourceExceptions" : 0,
          "latestSourceExceptions" : [ ],
          "numWritten" : 909,
          "lastReceivedTime" : 1693946605023,
          "workerId" : "test"
        }
      } ]
    }
    ```
  </Tab>

  <Tab title="Terraform">
    To update the submitted function, you only need to update the Terraform file and then call the following command.

    ```bash theme={null}
    terraform apply
    ```

    You should see something like following:

    ```bash theme={null}
    pulsar_function.function-1: Refreshing state... [id=public/default/function1]

    Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
      ~ update in-place

    Terraform will perform the following actions:

      # pulsar_function.function-1 will be updated in-place
      ~ resource "pulsar_function" "function-1" {
          ~ custom_runtime_options          = jsonencode(
              ~ {
                  - clusterName         = "oxia-test"
                  - enableStateStore    = false
                  - inputTypeClassName  = "java.lang.String"
                  - logTopicAgent       = "runtime"
                  - managed             = true
                  - maxReplicas         = 0
                  - outputTypeClassName = "java.lang.String"
                  - runnerImage         = "streamnative/pulsar-functions-sn-java-runner:3.1.0.4"
                  - serviceAccountName  = "oxia-test-function-pulsarcluster"
                    # (1 unchanged attribute hidden)
                }
            )
          ~ disk_mb                         = 10240 -> 128
            id                              = "public/default/function1"
            name                            = "function1"
          ~ parallelism                     = 1 -> 2
          ~ ram_mb                          = 140 -> 128
            # (21 unchanged attributes hidden)
        }

    Plan: 0 to add, 1 to change, 0 to destroy.

    Do you want to perform these actions?
      Terraform will perform the actions described above.
      Only 'yes' will be accepted to approve.

      Enter a value:
    ```

    Enter yes and then you should get:

    ```bash theme={null}
    pulsar_function.function-1: Modifying... [id=public/default/function1]
    pulsar_function.function-1: Modifications complete after 0s [id=public/default/function1]

    Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
    ```
  </Tab>

  <Tab title="REST API">
    For the REST API you can simply call CURL to update your function.

    ```bash theme={null}
      curl -X POST ${WEB_SERVICE_URL}/admin/v3/functions/public/default/${FUNCTION_NAME} \
      -H 'Authorization: Bearer ${TOKEN}' \
      -H "Content-Type: multipart/form-data" \
      -F 'functionConfig={"name": "${FUNCTION_NAME}", "parallelism": 2};type=application/json' \
      -F 'url=function://public/default/${FUNCTION_NAME}@${FUNCTION_VERSION}' \
    ```

    <Note title="Note">
      Replace the placeholder variables with the actual values that you can get when [setting up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).

      * `WEB_SERVICE_URL`: the HTTPS service URL of your Pulsar cluster.
      * `TOKEN`: a valid token to interact with your Pulsar cluster.
      * `FUNCTION_NAME`: the name of your function.
      * `FUNCTION_VERSION`: the version of your function you want to deploy e.g. 1.12.
    </Note>
  </Tab>
</Tabs>

## Delete a function

The following example shows how to delete the Java exclamation function `exclamation` using different tools.

<Tabs>
  <Tab title="snctl">
    To delete the function `exclamation`, use the following command.

    ```bash theme={null}
    snctl pulsar admin functions delete --tenant public --namespace default --name exclamation
    ```

    You should see the following output:

    ```bash theme={null}
    Deleted exclamation successfully
    ```

    If you want to verify whether the function has been deleted successfully, run the following command.

    ```bash theme={null}
    snctl pulsar admin functions get --tenant public --namespace default --name exclamation
    ```

    You should see the following output:

    ```bash theme={null}
    [✖]  code: 500 reason: failed to perform the request: responseCode: 404, responseMessage: functions.compute.functionmesh.io "exclamation-XXXXX" not found
    ```
  </Tab>

  <Tab title="pulsarctl">
    To delete the function `exclamation`, use the following command.

    ```bash theme={null}
    pulsarctl functions delete --tenant public --namespace default --name exclamation
    ```

    You should see the following output:

    ```bash theme={null}
    Deleted exclamation successfully
    ```

    If you want to verify whether the function has been deleted successfully, run the following command.

    ```bash theme={null}
    pulsarctl functions get --tenant public --namespace default --name exclamation
    ```

    You should see the following output:

    ```bash theme={null}
    [✖]  code: 500 reason: failed to perform the request: responseCode: 404, responseMessage: functions.compute.functionmesh.io "exclamation-XXXXX" not found
    ```
  </Tab>

  <Tab title="pulsar-admin">
    To delete the function `exclamation`, run the following command.

    ```bash theme={null}
    ./bin/pulsar-admin functions delete  --tenant public --namespace default --name exclamation
    ```

    You should see the following output:

    ```bash theme={null}
    Delete exclamation successfully
    ```

    To verify the function has been deleted, run the following command.

    ```bash theme={null}
    ./bin/pulsar-admin functions get --tenant public --namespace default --name exclamation
    ```

    You should see the following output:

    ```bash theme={null}
    failed to perform the request: responseCode: 404, responseMessage: functions.compute.functionmesh.io "exclamation-XXXXX" not found
    ```
  </Tab>

  <Tab title="Terraform">
    To delete the function `exclamation` with terraform, run the following command and type `yes` on the prompt.

    ```bash theme={null}
    terraform destroy
    ```

    You should see the following output:

    ```bash theme={null}
    pulsar_function.function-1: Refreshing state... [id=public/default/function1]

    Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
      - destroy

    Terraform will perform the following actions:

      # pulsar_function.function-1 will be destroyed
      - resource "pulsar_function" "function-1" {
          - auto_ack                        = true -> null
          - classname                       = "org.apache.pulsar.functions.api.examples.ExclamationFunction" -> null
          - cleanup_subscription            = true -> null
          - cpu                             = 0.5 -> null
          - custom_runtime_options          = jsonencode(
                {
                  - clusterName         = "oxia-test"
                  - enableStateStore    = false
                  - env                 = {
                      - HELLO = "WORLD"
                    }
                  - inputTypeClassName  = "java.lang.String"
                  - logTopicAgent       = "runtime"
                  - managed             = true
                  - maxReplicas         = 0
                  - outputTypeClassName = "java.lang.String"
                  - runnerImage         = "streamnative/pulsar-functions-sn-java-runner:3.1.0.4"
                  - serviceAccountName  = "oxia-test-function-pulsarcluster"
                }
            ) -> null
          - dead_letter_topic               = "public/default/dlt" -> null
          - disk_mb                         = 10240 -> null
          - forward_source_message_property = true -> null
          - id                              = "public/default/function1" -> null
          - inputs                          = [
              - "persistent://public/default/test-java-input",
            ] -> null
          - jar                             = "function://public/default/exclamation@v0.1" -> null
          - log_topic                       = "public/default/lt" -> null
          - max_message_retries             = 100 -> null
          - name                            = "function1" -> null
          - namespace                       = "default" -> null
          - output                          = "persistent://public/default/test-java-output" -> null
          - parallelism                     = 2 -> null
          - processing_guarantees           = "ATLEAST_ONCE" -> null
          - ram_mb                          = 140 -> null
          - retain_key_ordering             = true -> null
          - retain_ordering                 = false -> null
          - secrets                         = jsonencode(
                {
                  - SECRET1 = {
                      - key  = "hello"
                      - path = "sectest"
                    }
                }
            ) -> null
          - skip_to_latest                  = true -> null
          - subscription_name               = "test-sub" -> null
          - subscription_position           = "Latest" -> null
          - tenant                          = "public" -> null
          - timeout_ms                      = 6666 -> null
        }

    Plan: 0 to add, 0 to change, 1 to destroy.

    Do you really want to destroy all resources?
      Terraform will destroy all your managed infrastructure, as shown above.
      There is no undo. Only 'yes' will be accepted to confirm.

      Enter a value:
    ```

    Enter yes and you will get:

    ```bash theme={null}
    pulsar_function.function-1: Destroying... [id=public/default/function1]
    pulsar_function.function-1: Destruction complete after 0s

    Destroy complete! Resources: 1 destroyed.
    ```
  </Tab>
</Tabs>

## What’s next?

* Learn how to [configure stateful functions](/cloud/process/pulsar-functions/function-state).
* Learn how to [monitor functions](/cloud/process/pulsar-functions/function-monitoring).
* Reference [common configurations](/cloud/process/pulsar-functions/function-config).


# Monitor and Troubleshoot Functions
Source: https://docs.streamnative.io/cloud/process/pulsar-functions/function-monitoring



StreamNative Cloud allows you to monitor functions status, logs, and exceptions that are thrown when a function fails to be created, updated, or cannot work.

## View function status

This section describes how to view function status using `snctl`, `pulsarctl`, `pulsar-admin`, and console.

<Note title="Note">
  If you want to monitor functions using `snctl`, `pulsarctl` or `pulsar-admin`, make sure you have set up your client tool. For more information, see [set up client tools](/cloud/process/pulsar-functions/function-setup#set-up-client-tools).
</Note>

The following example introduces how to view the status of the Java exclamation function named `exclamation`.

<Tabs>
  <Tab title="snctl">
    To check the status of the function `exclamation`, run the following command:

    ```bash theme={null}
    snctl pulsar admin functions status --tenant public --namespace default --name exclamation
    ```

    ```bash theme={null}
    {
      "numInstances": 1,
      "numRunning": 1,
      "instances": [
        {
          "instanceId": 0,
          "status": {
            "running": true,
            "error": "",
            "numRestarts": 0,
            "numReceivedFromSource": 2622,
            "numSystemExceptions": 0,
            "latestSystemExceptions": [],
            "numSourceExceptions": 0,
            "latestSourceExceptions": [],
            "numWritten": 2622,
            "lastReceivedTime": 1691532145625,
            "workerId": "test"
          }
      ]
    }
    ```
  </Tab>

  <Tab title="pulsarctl">
    To check the status of the function `exclamation`, run the following command:

    ```bash theme={null}
    pulsarctl functions status --tenant public --namespace default --name exclamation
    ```

    ```bash theme={null}
    {
      "numInstances": 1,
      "numRunning": 1,
      "instances": [
        {
          "instanceId": 0,
          "status": {
            "running": true,
            "error": "",
            "numRestarts": 0,
            "numReceivedFromSource": 2622,
            "numSystemExceptions": 0,
            "latestSystemExceptions": [],
            "numSourceExceptions": 0,
            "latestSourceExceptions": [],
            "numWritten": 2622,
            "lastReceivedTime": 1691532145625,
            "workerId": "test"
          }
      ]
    }
    ```
  </Tab>

  <Tab title="pulsar-admin">
    To check the status of the function `exclamation`, run the following command:

    ```bash theme={null}
    ./bin/pulsar-admin functions status --tenant public --namespace default --name exclamation
    ```

    You should see the following output:

    ```bash theme={null}
    {
      "numInstances" : 1,
      "numRunning" : 1,
      "instances" : [ {
        "instanceId" : 0,
        "status" : {
          "running" : true,
          "error" : "",
          "numRestarts" : 0,
          "numReceivedFromSource" : 433,
          "numSystemExceptions" : 0,
          "latestSystemExceptions" : [ ],
          "numSourceExceptions" : 0,
          "latestSourceExceptions" : [ ],
          "numWritten" : 433,
          "lastReceivedTime" : 1691452485845,
          "workerId" : "test"
        }
      } ]
    }
    ```
  </Tab>

  <Tab title="Console">
    1. From the left navigation pane, under **Resources**, click **Functions**.

    2. Select the function item to view the status and exceptions of the function, as well as the system exceptions.
  </Tab>
</Tabs>

## View function logs

### View the function logs using `snctl`

This section describes how to view function logs using `snctl`.

<Note title="Note">
  This example assumes you have [installed snctl](/tools/cli/snctl/snctl-overview#install-snctl) and [initialized snctl configurations](/tools/cli/snctl/snctl-overview#initialize-snctl-configurationn).
</Note>

You can run the `snctl logs` command to view logs for a specific function. This table outlines the configuration options that are used for viewing function logs. For details about all supported fields, you can use the `snctl logs -h` command to list more information.

| Option                   | Descriptions                                                                                                     |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------- |
| `-c` or `--cluster`      | The name of your Pulsar cluster where the function is created.                                                   |
| `-p` or `--component`    | The type of component to monitor. Available options are `function`, `sink`, and `source`.                        |
| `-f` or `--follow`       | Continuously list the function log history.                                                                      |
| `-h` or `--help`         | Show usage information about the `snctl logs` command.                                                           |
| `-i` or `--instance`     | The name of your Pulsar instance where the function is created.                                                  |
| `--name`                 | The name of your function.                                                                                       |
| `-o` or `--organization` | The name of your organization where the function is created.                                                     |
| `--previous`             | Print the logs that are generated before the configured timestamp.                                               |
| `--pulsar-tenant`        | The name of your Pulsar tenant where the function is created.                                                    |
| `--pulsar-namespace`     | The name of your Pulsar namespace where the function is created.                                                 |
| `--since`                | List logs more recent than the specific time. Available units are `second`, `minute`, and `hour`, such as `24h`. |
| `-s` or `--size`         | Specify how many lines of recent logs to display.                                                                |
| `--timestamp`            | Include timestamps on each line in the log output.                                                               |

The following command example shows how to view up to 60 lines of the `exclamation` function’s logs within the last 5 hours.

```bash theme={null}
snctl logs --since 5h --organization sndev --instance aws --cluster aws --name exclamation --pulsar-tenant public --pulsar-namespace default -p function -f -s 60
```

You should see the following output:

```bash theme={null}
The package 'function://public/default/exclamation@v0.1' downloaded to path 'download/pulsar_functions/api-examples.jar' successfully
shardId=0
Using function root classloader: jdk.internal.loader.ClassLoaders$AppClassLoader@14dad5dc
Using function instance classloader: java.net.URLClassLoader@42dafa95
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/pulsar/lib/org.apache.logging.log4j-log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/pulsar/instances/java-instance.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
Starting function instance...
2023-10-20T07:19:34,786+0000 [main] INFO  org.apache.pulsar.common.nar.FileUtils - Jar file download/pulsar_functions/api-examples.jar does not contain META-INF/bundled-dependencies, it is not a NAR file
2023-10-20T07:19:34,789+0000 [main] INFO  org.apache.pulsar.functions.runtime.thread.ThreadRuntime - Load file as simple JAR file: download/pulsar_functions/api-examples.jar
2023-10-20T07:19:34,791+0000 [main] INFO  org.apache.pulsar.functions.runtime.thread.ThreadRuntime - Initialize function class loader for function exclamation-2598d0c3 at function cache manager, functionClassLoader: org.apache.pulsar.functions.utils.functioncache.FunctionClassLoaders$ParentFirstClassLoader@2eced48b
WARNING: Illegal reflective access by org.apache.pulsar.common.util.netty.DnsResolverUtil (file:/pulsar/lib/io.streamnative-pulsar-common-2.10.5.5.jar) to method sun.net.InetAddressCachePolicy.get()
WARNING: Please consider reporting this to the maintainers of org.apache.pulsar.common.util.netty.DnsResolverUtil
WARNING: All illegal access operations will be denied in a future release
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: An illegal reflective access operation has occurred
2023-10-20T07:19:37,463+0000 [main] INFO  org.apache.pulsar.functions.runtime.JavaInstanceStarter - Starting runtimeSpawner
2023-10-20T07:19:37,462+0000 [main] INFO  org.apache.pulsar.functions.runtime.JavaInstanceStarter - JavaInstance Server started, listening on 9093
2023-10-20T07:19:37,463+0000 [main] INFO  org.apache.pulsar.functions.runtime.RuntimeSpawner - public/default/exclamation-2598d0c3-0 RuntimeSpawner starting function
2023-10-20T07:19:37,466+0000 [main] INFO  org.apache.pulsar.functions.runtime.thread.ThreadRuntime - Load file as simple JAR file: download/pulsar_functions/api-examples.jar
2023-10-20T07:19:37,466+0000 [main] INFO  org.apache.pulsar.common.nar.FileUtils - Jar file download/pulsar_functions/api-examples.jar does not contain META-INF/bundled-dependencies, it is not a NAR file
2023-10-20T07:19:37,466+0000 [main] INFO  org.apache.pulsar.functions.runtime.thread.ThreadRuntime - Initialize function class loader for function exclamation-2598d0c3 at function cache manager, functionClassLoader: org.apache.pulsar.functions.utils.functioncache.FunctionClassLoaders$ParentFirstClassLoader@2eced48b
2023-10-20T07:19:37,473+0000 [main] INFO  org.apache.pulsar.functions.runtime.thread.ThreadRuntime - ThreadContainer starting function with instanceId 0 functionId 0-28d06ebd-df24-4503-8d28-f573034ac7f8 namespace default
2023-10-20T07:19:37,474+0000 [main] INFO  org.apache.pulsar.functions.runtime.JavaInstanceStarter - Starting metrics server on port 9094
userConfig: "{}"
parallelism: 1
  inputSpecs {
  typeClassName: "java.lang.String"
  ram: 1073741824
sink {
  forwardSourceMessageProperty: true
}
  cpu: 1.0
2023-10-20T07:19:37,489+0000 [public/default/exclamation-2598d0c3-0] INFO  org.apache.pulsar.functions.instance.JavaInstanceRunnable - Starting Java Instance exclamation-2598d0c3 :
namespace: "default"
className: "org.apache.pulsar.functions.api.examples.ExclamationFunction"
    }
  }
}
  producerSpec {
}
 Details = tenant: "public"
  typeClassName: "java.lang.String"
    key: "persistent://public/default/test-java-input"
  cleanupSubscription: true
  topic: "persistent://public/default/test-java-output"
  }
componentType: FUNCTION
name: "exclamation-2598d0c3"
autoAck: true
source {
    value {
resources {

2023-10-20T07:19:37,967+0000 [public/default/exclamation-2598d0c3-0] INFO  org.apache.pulsar.functions.sink.PulsarSink - Opening pulsar sink with config: PulsarSinkConfig(processingGuarantees=ATLEAST_ONCE, topic=persistent://public/default/test-java-output, serdeClassName=null, schemaType=null, schemaProperties={}, typeClassName=java.lang.String, forwardSourceMessageProperty=true, producerConfig=ProducerConfig(maxPendingMessages=0, maxPendingMessagesAcrossPartitions=0, useThreadLocalProducers=false, cryptoConfig=null, batchBuilder=))
2023-10-20T07:19:38,177+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x030c5cc3, L:/10.44.3.121:51444 - R:func-on-cloud-test-broker.NAMESPACE.svc.cluster.local.NAMESPACE.svc.cluster.local/10.39.176.165:6650]] Connected to server
2023-10-20T07:19:38,577+0000 [public/default/exclamation-2598d0c3-0] INFO  org.apache.pulsar.functions.sink.PulsarSink - crypto key reader is not provided, not enabling end to end encryption
2023-10-20T07:19:38,682+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - Starting Pulsar producer perf with config: {"topicName":"persistent://public/default/test-java-output","producerName":null,"sendTimeoutMs":0,"blockIfQueueFull":true,"maxPendingMessages":1000,"maxPendingMessagesAcrossPartitions":50000,"messageRoutingMode":"CustomPartition","hashingScheme":"Murmur3_32Hash","cryptoFailureAction":"FAIL","batchingMaxPublishDelayMicros":10000,"batchingPartitionSwitchFrequencyByPublishDelay":10,"batchingMaxMessages":1000,"batchingMaxBytes":131072,"batchingEnabled":true,"chunkingEnabled":false,"compressionType":"LZ4","initialSequenceId":null,"autoUpdatePartitions":true,"autoUpdatePartitionsIntervalSeconds":60,"multiSchema":true,"accessMode":"Shared","lazyStartPartitionedProducers":false,"properties":{"application":"pulsar-function","id":"public/default/exclamation-2598d0c3","instance_hostname":"exclamation-2598d0c3-function-0","instance_id":"0"},"initialSubscriptionName":null}
2023-10-20T07:19:38,705+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - Pulsar client config: {"serviceUrl":"pulsar://func-on-cloud-test-broker.NAMESPACE.svc.cluster.local:6650","authPluginClassName":"org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2","authParams":"*****","authParamMap":null,"operationTimeoutMs":30000,"lookupTimeoutMs":30000,"statsIntervalSeconds":60,"numIoThreads":1,"numListenerThreads":1,"connectionsPerBroker":1,"useTcpNoDelay":true,"useTls":false,"tlsTrustCertsFilePath":null,"tlsAllowInsecureConnection":true,"tlsHostnameVerificationEnable":false,"concurrentLookupRequest":5000,"maxLookupRequest":50000,"maxLookupRedirects":20,"maxNumberOfRejectedRequestPerConnection":50,"keepAliveIntervalSeconds":30,"connectionTimeoutMs":10000,"requestTimeoutMs":60000,"initialBackoffIntervalNanos":100000000,"maxBackoffIntervalNanos":60000000000,"enableBusyWait":false,"listenerName":null,"useKeyStoreTls":false,"sslProvider":null,"tlsTrustStoreType":"JKS","tlsTrustStorePath":null,"tlsTrustStorePassword":null,"tlsCiphers":[],"tlsProtocols":[],"memoryLimitBytes":0,"proxyServiceUrl":null,"proxyProtocol":null,"enableTransaction":false,"dnsLookupBindAddress":null,"dnsLookupBindPort":0,"socks5ProxyAddress":null,"socks5ProxyUsername":null,"socks5ProxyPassword":null}
2023-10-20T07:19:38,810+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xda231b5b, L:/10.44.3.121:58592 - R:func-on-cloud-test-broker-0.func-on-cloud-test-broker-headless.NAMESPACE.svc.cluster.local/240.240.0.86:6650]] Connected to server
2023-10-20T07:19:39,400+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/test-java-output-partition-0] [null] Creating producer on cnx [id: 0xda231b5b, L:/10.44.3.121:58592 - R:func-on-cloud-test-broker-0.func-on-cloud-test-broker-headless.NAMESPACE.svc.cluster.local/240.240.0.86:6650]
2023-10-20T07:19:41,808+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/test-java-output-partition-0] [func-on-cloud-test-5-0] Created producer on cnx [id: 0xda231b5b, L:/10.44.3.121:58592 - R:func-on-cloud-test-broker-0.func-on-cloud-test-broker-headless.NAMESPACE.svc.cluster.local/240.240.0.86:6650]
2023-10-20T07:19:41,811+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.PartitionedProducerImpl - [persistent://public/default/test-java-output] Created partitioned producer
2023-10-20T07:19:41,818+0000 [public/default/exclamation-2598d0c3-0] INFO  org.apache.pulsar.functions.source.SingleConsumerPulsarSource - Opening pulsar source with config: SingleConsumerPulsarSourceConfig(topic=persistent://public/default/test-java-input, consumerConfig=ConsumerConfig(schemaType=null, serdeClassName=null, isRegexPattern=false, schemaProperties={}, consumerProperties={}, receiverQueueSize=null, cryptoConfig=null, poolMessages=false))
  "properties": {}
}
  "timestamp": 0,
  "type": "STRING",
  "name": "String",
2023-10-20T07:19:41,824+0000 [public/default/exclamation-2598d0c3-0] INFO  org.apache.pulsar.functions.source.SingleConsumerPulsarSource - Creating consumer for topic : persistent://public/default/test-java-input, schema : org.apache.pulsar.client.impl.schema.StringSchema@38ebab52, schemaInfo: {
  "schema": "",
2023-10-20T07:19:41,922+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Starting Pulsar consumer status recorder with config: {"topicNames":[],"topicsPattern":null,"subscriptionName":"public/default/exclamation-2598d0c3","subscriptionType":"Shared","subscriptionProperties":null,"subscriptionMode":"Durable","receiverQueueSize":1000,"acknowledgementsGroupTimeMicros":100000,"negativeAckRedeliveryDelayMicros":60000000,"maxTotalReceiverQueueSizeAcrossPartitions":50000,"consumerName":"7330b","ackTimeoutMillis":0,"tickDurationMillis":1000,"priorityLevel":0,"maxPendingChunkedMessage":10,"autoAckOldestChunkedMessageOnQueueFull":false,"expireTimeOfIncompleteChunkedMessageMillis":60000,"cryptoFailureAction":"FAIL","properties":{"application":"pulsar-function","id":"public/default/exclamation-2598d0c3","instance_hostname":"exclamation-2598d0c3-function-0","instance_id":"0"},"readCompacted":false,"subscriptionInitialPosition":"Latest","patternAutoDiscoveryPeriod":60,"regexSubscriptionMode":"PersistentOnly","deadLetterPolicy":null,"retryEnable":false,"autoUpdatePartitions":true,"autoUpdatePartitionsIntervalSeconds":60,"replicateSubscriptionState":false,"resetIncludeHead":false,"batchIndexAckEnabled":false,"ackReceiptEnabled":false,"poolMessages":false,"startPaused":false,"maxPendingChuckedMessage":10}
2023-10-20T07:19:41,961+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Pulsar client config: {"serviceUrl":"pulsar://func-on-cloud-test-broker.NAMESPACE.svc.cluster.local:6650","authPluginClassName":"org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2","authParams":"*****","authParamMap":null,"operationTimeoutMs":30000,"lookupTimeoutMs":30000,"statsIntervalSeconds":60,"numIoThreads":1,"numListenerThreads":1,"connectionsPerBroker":1,"useTcpNoDelay":true,"useTls":false,"tlsTrustCertsFilePath":null,"tlsAllowInsecureConnection":true,"tlsHostnameVerificationEnable":false,"concurrentLookupRequest":5000,"maxLookupRequest":50000,"maxLookupRedirects":20,"maxNumberOfRejectedRequestPerConnection":50,"keepAliveIntervalSeconds":30,"connectionTimeoutMs":10000,"requestTimeoutMs":60000,"initialBackoffIntervalNanos":100000000,"maxBackoffIntervalNanos":60000000000,"enableBusyWait":false,"listenerName":null,"useKeyStoreTls":false,"sslProvider":null,"tlsTrustStoreType":"JKS","tlsTrustStorePath":null,"tlsTrustStorePassword":null,"tlsCiphers":[],"tlsProtocols":[],"memoryLimitBytes":0,"proxyServiceUrl":null,"proxyProtocol":null,"enableTransaction":false,"dnsLookupBindAddress":null,"dnsLookupBindPort":0,"socks5ProxyAddress":null,"socks5ProxyUsername":null,"socks5ProxyPassword":null}
2023-10-20T07:19:41,961+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Pulsar client config: {"serviceUrl":"pulsar://func-on-cloud-test-broker.NAMESPACE.svc.cluster.local:6650","authPluginClassName":"org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2","authParams":"*****","authParamMap":null,"operationTimeoutMs":30000,"lookupTimeoutMs":30000,"statsIntervalSeconds":60,"numIoThreads":1,"numListenerThreads":1,"connectionsPerBroker":1,"useTcpNoDelay":true,"useTls":false,"tlsTrustCertsFilePath":null,"tlsAllowInsecureConnection":true,"tlsHostnameVerificationEnable":false,"concurrentLookupRequest":5000,"maxLookupRequest":50000,"maxLookupRedirects":20,"maxNumberOfRejectedRequestPerConnection":50,"keepAliveIntervalSeconds":30,"connectionTimeoutMs":10000,"requestTimeoutMs":60000,"initialBackoffIntervalNanos":100000000,"maxBackoffIntervalNanos":60000000000,"enableBusyWait":false,"listenerName":null,"useKeyStoreTls":false,"sslProvider":null,"tlsTrustStoreType":"JKS","tlsTrustStorePath":null,"tlsTrustStorePassword":null,"tlsCiphers":[],"tlsProtocols":[],"memoryLimitBytes":0,"proxyServiceUrl":null,"proxyProtocol":null,"enableTransaction":false,"dnsLookupBindAddress":null,"dnsLookupBindPort":0,"socks5ProxyAddress":null,"socks5ProxyUsername":null,"socks5ProxyPassword":null}
2023-10-20T07:19:42,007+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/test-java-input-partition-0][public/default/exclamation-2598d0c3] Subscribing to topic on cnx [id: 0xda231b5b, L:/10.44.3.121:58592 - R:func-on-cloud-test-broker-0.func-on-cloud-test-broker-headless.NAMESPACE.svc.cluster.local/240.240.0.86:6650], consumerId 0
2023-10-20T07:19:42,974+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/test-java-input-partition-0][public/default/exclamation-2598d0c3] Subscribed to topic on func-on-cloud-test-broker-0.func-on-cloud-test-broker-headless.NAMESPACE.svc.cluster.local/240.240.0.86:6650 -- consumer: 0
2023-10-20T07:19:42,977+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.MultiTopicsConsumerImpl - [persistent://public/default/test-java-input] [public/default/exclamation-2598d0c3] Success subscribe new topic persistent://public/default/test-java-input in topics consumer, partitions: 1, allTopicPartitionsNumber: 1
2023-10-20T07:21:47,466+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ClientCnx - [id: 0xda231b5b, L:/10.44.3.121:58592 ! R:func-on-cloud-test-broker-0.func-on-cloud-test-broker-headless.NAMESPACE.svc.cluster.local/240.240.0.86:6650] Disconnected
2023-10-20T07:21:47,470+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/test-java-output-partition-0] [func-on-cloud-test-5-0] Closed connection [id: 0xda231b5b, L:/10.44.3.121:58592 ! R:func-on-cloud-test-broker-0.func-on-cloud-test-broker-headless.NAMESPACE.svc.cluster.local/240.240.0.86:6650] -- Will try again in 0.1 s
2023-10-20T07:21:47,471+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/test-java-input-partition-0] [public/default/exclamation-2598d0c3] Closed connection [id: 0xda231b5b, L:/10.44.3.121:58592 ! R:func-on-cloud-test-broker-0.func-on-cloud-test-broker-headless.NAMESPACE.svc.cluster.local/240.240.0.86:6650] -- Will try again in 0.1 s
2023-10-20T07:21:47,571+0000 [pulsar-timer-8-1] INFO  org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/test-java-output-partition-0] [func-on-cloud-test-5-0] Reconnecting after timeout
2023-10-20T07:21:47,572+0000 [pulsar-timer-8-1] INFO  org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/test-java-input-partition-0] [public/default/exclamation-2598d0c3] Reconnecting after timeout
2023-10-20T07:21:47,576+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x218f058a, L:/10.44.3.121:39488 - R:func-on-cloud-test-broker-0.func-on-cloud-test-broker-headless.NAMESPACE.svc.cluster.local/240.240.0.85:6650]] Connected to server
2023-10-20T07:21:47,580+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/test-java-input-partition-0][public/default/exclamation-2598d0c3] Subscribing to topic on cnx [id: 0x218f058a, L:/10.44.3.121:39488 - R:func-on-cloud-test-broker-0.func-on-cloud-test-broker-headless.NAMESPACE.svc.cluster.local/240.240.0.85:6650], consumerId 0
2023-10-20T07:21:47,580+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/test-java-output-partition-0] [func-on-cloud-test-5-0] Creating producer on cnx [id: 0x218f058a, L:/10.44.3.121:39488 - R:func-on-cloud-test-broker-0.func-on-cloud-test-broker-headless.NAMESPACE.svc.cluster.local/240.240.0.85:6650]
2023-10-20T07:21:47,751+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/test-java-output-partition-0] [func-on-cloud-test-5-0] Created producer on cnx [id: 0x218f058a, L:/10.44.3.121:39488 - R:func-on-cloud-test-broker-0.func-on-cloud-test-broker-headless.NAMESPACE.svc.cluster.local/240.240.0.85:6650]
2023-10-20T07:21:47,756+0000 [pulsar-client-io-1-1] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/test-java-input-partition-0][public/default/exclamation-2598d0c3] Subscribed to topic on func-on-cloud-test-broker-0.func-on-cloud-test-broker-headless.NAMESPACE.svc.cluster.local/240.240.0.85:6650 -- consumer: 0
```

### View the function logs on StreamNative Cloud Console

After you have successfully deployed a function, you can check its status, logs, and any exceptions through StreamNative Cloud Console.

<Note title="Note">
  The **Functions** option in the **Resources** area is only available if you have configured your environment and deployed at least one function.
</Note>

1. [Log in to StreamNative Cloud Console](https://console.streamnative.cloud/?defaultMethod=signup).

2. On the left navigation pane, in the **Resources** section, click **Functions**.

3. Click the status of the function you just deployed to view the details page. In the figure below, the status is **Running**.

   <img alt="screenshot of functions detail page" />

4. On the functions detail page, click **Logs** to view a real-time log of the running function you selected. Click **Exceptions** to view any exceptions.

   <img alt="screenshot of functions exception logs" />

### View the function logs from log topics

You can use Pulsar Functions log topic feature to view the function logs. For more information, see [Pulsar Functions log topic](https://pulsar.apache.org/docs/en/functions-debug-log-topic).

Functions on StreamNative provide an alternative implementation of the Pulsar Functions log topic feature called `sidecar` mode, so you can use the feature in production environments with less performance impact.

To enable the `sidecar` mode, user should provide the following configurations when deploying the function:

```yaml theme={null}
pulsarctl functions create --custom-runtime-options '{"logTopicAgent":"sidecar"}' --log-topic log-topic ...
```

For more details about the `custom-runtime-options`, see [Custom runtime options](/cloud/process/pulsar-functions/function-config#stream-native-cloud-custom-runtime-options).

Once the function is deployed with the `sidecar` mode, you can configure consumers to consume messages from the log topic.


# Set up Your Environment
Source: https://docs.streamnative.io/cloud/process/pulsar-functions/function-setup



This section introduces how to set up a new service account with the minimum permissions to run functions. To perform the following operations, you need to be the cluster administrator beforehand.

## Prerequisites

* [Install](/tools/cli/snctl/snctl-overview#install-snctl) and [configure](/tools/cli/snctl/snctl-overview#configure-snctl) the `snctl` CLI tool.
* [Install](https://github.com/streamnative/pulsarctl#install-pulsarctl) the `pulsarctl` CLI tool.
* [Log in to StreamNative Cloud Console](https://console.streamnative.cloud/?defaultMethod=signup).
* Create a [Pulsar cluster](/cloud/clusters/manage-clusters/cluster#create-a-cluster) and [connect to](/tools/cli/pulsarctl/pulsarctl-overview) your Pulsar cluster using the `pulsarctl` CLI tool.
* Create a [tenant](/cloud/manage-data-streams/tenant#create-a-tenant) and a [namespace](/cloud/manage-data-streams/namespace#create-a-namespace).

## Create a service account for Pulsar users

1. On the left navigation pane of StreamNative Cloud Console, click **Service Accounts**.

2. Click **Create Service Account**.

3. Enter a name for the service account, and then click **Confirm**.

<Note title="Note">
  Do **NOT** check the **Super Admin** option when creating this service account.
</Note>

## Authorize the service account

To make the service account work, you need to make the service account granted with proper permissions (`functions`, `packages`, `produce`, and `consume`).

To grant the service account permissions on the namespace level, follow these steps:

<Tabs>
  <Tab title="Cloud Console">
    1. Navigate to the **Namespace Dashboard** page by [switching to the namespace workspace](/cloud/get-started/cloud-console#switch-a-namespace).

    2. On the **Namespace Dashboard** page, click **Configuration** on the left navigation pane.

    3. On the **Namespace configuration** page, click **ADD ROLE**, and select the service account that you want to authorize.

    4. On the drop-down menu below the service account, select the proper permissions to assign to the service account. There are six permissions in total:

    * `consume`: allow the service account to consume messages.
    * `produce`: allow the service account to publish messages.
    * `functions`: allow the service account to submit and manage functions.
    * `sinks`: allow the service account to create and manage sink connectors.
    * `sources`: allow the service account to create and manage source connectors.
    * `packages`: allow the service account to upload and manage pulsar packages. If you want to submit a customized function/connector, then you will need to upload the function/connector’s JAR/NAR/Python file first, which requires the `packages` permission.

      <img alt="Authorize Service Account" />
  </Tab>
</Tabs>

## Grant access to the service account

To grant the underlying infrastructure with access to the newly created service account's OAuth2 key file, you need to create a service account binding via UI.

Go to the `Service Accounts` tab and choose the service account you want to use for running the connector. Clicking on the right button and there will be a `Edit service account bindings` option.

<img alt="Binding Service Account step-1" />

Click the `Edit service account bindings`, choose the desired pool member and confirm.

<img alt="Binding Service Account step-2" />

You can also enable the `Enable IAM Role Creation` option to create a separate IAM role for the service account.

Now your connector is ready to use the service account in StreamNative environments.

## (Optional) Create a separate IAM role for the service account

StreamNative's I/O components (Pulsar Functions, Pulsar Connectors, and Kafka Connectors) run as cloud-native workloads on AWS, GCP, and Azure infrastructures.

Use the cloud providers' native IAM (Identity and Access Management) services to control access to infrastructure resources such as S3, GCS, and Azure Blob Storage. This removes the need for password-based authentication for the service accounts that run connectors in StreamNative Cloud.

By default, StreamNative uses a single service account per `PulsarCluster` for all I/O components (Pulsar Functions, Pulsar Connectors, and Kafka Connectors) to access underlying infrastructure resources. This means all I/O Components in the same cluster share one service account and the same permissions.

<Warning>
  Using a single service account for all I/O components means all functions and connectors share the same permissions. If one component is compromised, it could access resources intended for other components.

  So it's better to leave the default service account with no permissions and use it for running IO components that do not require access to external resources.

  And create separate service accounts with the minimum required permissions for each IO component that need to access external resources.
</Warning>

To enhance security and improve isolation, you can create a separate IAM role for each service account used to run connectors. This let you grant only the permissions each service account needs.

Create a separate IAM role for your service account:

<Tabs>
  <Tab title="Snctl">
    <Note title="Note">
      This feature is available in [snctl](/tools/cli/snctl/snctl-overview) v1.3.0 or later.
    </Note>

    1. Get the `PoolMember` name and namespace from the `PulsarCluster`

       ```shell theme={null}
       snctl get pulsarcluster <cluster-id> -o yaml
       ```

       In the output, find the `poolMemberRef` block, which looks like:

       ```yaml theme={null}
       poolMemberRef:
         name: <pool-member-name>
         namespace: <namespace>
       ```

       <Note title="Note">
         Multiple clusters may be located in the same `PoolMember`. You do not need to create separate IAM roles for each cluster within the same `PoolMember`.
       </Note>

    2. Create a new `ServiceAccountBinding` that binds the service account to the `PoolMember`

       ```shell theme={null}
       snctl create serviceaccountbinding <binding-name> \
         --pool-member <pool-member-namespace>/<pool-member-name> \
         --enable-iam-account-creation \
         --service-account-name <service-account-name>
       ```

       **AWS only:** You can specify one or more `AWS Assume Role ARNs` that can be assumed by the IAM role created for the service account (repeat the flag for each ARN):

       ```shell theme={null}
       snctl create serviceaccountbinding <binding-name> \
         --pool-member <pool-member-namespace>/<pool-member-name> \
         --enable-iam-account-creation \
         --service-account-name <service-account-name> \
         --aws-assume-role-arns <arn1> \
         --aws-assume-role-arns <arn2>
       ```

       <Note title="Note">
         The IAM role created by StreamNative will include permissions to call `sts:AssumeRole` on `arn1` and `arn2`. You must still add a trust policy on `arn1` and `arn2` to allow the newly created role to assume them, an example likes below:

         ```json theme={null}
         {
             "Version": "2012-10-17",
             "Statement": [
                 {
                     "Effect": "Allow",
                     "Principal": {
                         "AWS": "arn:aws:iam::[aws-account-id]:role/StreamNative/sncloud-role/authorization.streamnative.io/iamaccounts/IamAccount-[org-id]-sab-[binding-name]"
                     },
                     "Action": "sts:AssumeRole"
                 }
             ]
         }
         ```
       </Note>

    3. (Optional) Update an existing `ServiceAccountBinding` to create the IAM role

       ```shell theme={null}
       snctl edit serviceaccountbinding <binding-name>
       ```

    4. Verify the IAM role was created successfully

       ```shell theme={null}
       snctl get serviceaccountbinding <binding-name> -o yaml
       ```

       Expected output (example):

       ```yaml theme={null}
       apiVersion: cloud.streamnative.io/v1alpha1
       kind: ServiceAccountBinding
       metadata:
         creationTimestamp: "2025-06-25T08:45:35Z"
         finalizers:
         - serviceaccountbinding.finalizers.cloud.streamnative.io
         generation: 1
         name: test-admin
         namespace: o-lftqu
         ownerReferences:
         - apiVersion: cloud.streamnative.io/v1alpha1
           kind: ServiceAccount
           name: admin
           uid: 4ef639aa-6278-4863-9a23-f1da50cea448
         resourceVersion: "54707713"
         uid: 918d40ad-551d-416b-a2ae-d41548d6608e
       spec:
         enableIamRoleCreation: true
         poolMemberRef:
           name: azure-eastus-zephyr
           namespace: streamnative
         serviceAccountName: admin
       status:
         conditions:
         - lastTransitionTime: "2025-06-25T08:45:35Z"
           status: "True"
           type: IAMAccountReady
         - lastTransitionTime: "2025-06-25T08:45:35Z"
           status: "True"
           type: ServiceAccountReady
         - lastTransitionTime: "2025-07-16T13:56:17Z"
           status: "True"
           type: ResourceExists
         - lastTransitionTime: "2025-07-16T13:56:17Z"
           status: "True"
           type: PoolMemberReady
         - lastTransitionTime: "2025-07-16T13:56:17Z"
           reason: AllConditionStatusTrue
           status: "True"
           type: Ready
       ```

       In the output, the `status.conditions` array should include a condition with `type: IAMAccountReady` and `status: "True"`, indicating the IAM role was created successfully.<br /><br />

    5. Use the service account when creating I/O components

       <br /><br />You can now select this service account in Console (or use its API key with the CLI) when creating I/O components (Pulsar Functions, Pulsar Connectors, and Kafka Connectors). The components will inherit the permissions granted to the IAM role created in the previous step.
  </Tab>

  <Tab title="Console">
    You can also create a separate IAM role for your service account via StreamNative Cloud Console. Just enable the `Enable IAM Role Creation` option when creating or editing a service account binding.

    <img alt="Binding Service Account step-2" />

    **AWS only:** You can specify one or more `AWS Assume Role ARNs` that can be assumed by the IAM role created for the service account (use one line for each ARN)

    <img alt="Binding Service Account step-3" />

    <Note title="Note">
      The IAM role created by StreamNative will include permissions to call `sts:AssumeRole` on `arn1` and `arn2`. You must still add a trust policy on `arn1` and `arn2` to allow the newly created role to assume them, an example likes below:

      ```json theme={null}
      {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Principal": {
                      "AWS": "arn:aws:iam::[aws-account-id]:role/StreamNative/sncloud-role/authorization.streamnative.io/iamaccounts/IamAccount-[org-id]-sab-[binding-name]"
                  },
                  "Action": "sts:AssumeRole"
              }
          ]
      }
      ```
    </Note>
  </Tab>
</Tabs>

<Note title="Note">
  In Azure, a Managed Identity `sab-[binding-name]-[org-id]` is created;

  In AWS, an IAM role `role/StreamNative/sncloud-role/authorization.streamnative.io/iamaccounts/IamAccount-[org-id]-sab-[binding-name]` is created;

  In GCP, a service account with display name: `IamAccount/[org-id]/sab-[binding-name]` is created.
</Note>

## Set up client tools

### Use `snctl` to manage Pulsar Functions with a service account

The StreamNative CLI `snctl` includes Pulsar admin commands that connect directly to your StreamNative Cloud cluster. You can use a service account for the admin request, the function runtime identity, or both.

1. Use `snctl config set --organization $ORG` to set your StreamNative Cloud organization.
2. Use `snctl context use` to select your target StreamNative Cloud cluster interactively.
3. To send the admin request as a service account, use `snctl pulsar admin --as-service-account $SERVICE_ACCOUNT_NAME ...` or `snctl pulsar admin --use-service-account ...`.
4. To run a function as a service account, use `--sn-service-account $SERVICE_ACCOUNT_NAME` on `snctl pulsar admin functions create` or `snctl pulsar admin functions update`. To select the runtime service account interactively, use `--use-sn-service-account`.

<Note title="Tip">
  `--as-service-account` and `--sn-service-account` set different identities.

  Use `--as-service-account` to send the request with the specified service account's credentials. That same service account also becomes the runtime identity of the function. Therefore, the service account must have permissions to create or update functions, download packages, and produce or consume messages.

  Use `--sn-service-account` to keep the request authenticated as the current caller, but run the function with the specified service account as its runtime identity. In this case, the caller must have permission to create or update the function and use the selected service account (see [account-admin](/cloud/security/access/rbac/manage-rbac-roles#account-admin)). The runtime service account only needs the permissions required by the function itself, such as producing or consuming messages and downloading packages.

  You can use `--as-service-account` on Pulsar clusters of any supported version. The `--sn-service-account` and `--use-sn-service-account` flags require Pulsar 4.0.x version 4.0.10.6 or later, or Pulsar 4.2.x version 4.2.1.4 or later.
</Note>

### Use `pulsarctl`, `pulsar-admin`, or the REST API to manage Pulsar Functions with a service account

StreamNative Cloud Console provides a step-by-step wizard to walk you through the basic client setup process. You can connect your Pulsar client that uses the previously created service account to interact with your Pulsar cluster.

1. On the left navigation pane of StreamNative Cloud Console, in the **Admin** section, click **Pulsar Clients**.

<img alt="Set up client tools" />

2. Select the **CLI Tools** tab and follow the wizard to generate the sample code you need for connecting to your Pulsar cluster. The steps may vary depending on the tool you use.

   a. Select either `pulsarctl` or `pulsar-admin`.

   b. Download the selected CLI tool.

   c. Select the service account you created.

   d. Select **OAuth2** as the authentication type and download the key file to your local machine.

   e. Set up your CLI tool with that key file, and the steps vary depending on the CLI tool you use.

<Tabs>
  <Tab title="pulsarctl">
    f. Copy the command for setting client configurations to your terminal, update the path of the OAuth2 key file, and run it.

    g. Select the target tenant, namespace and topic, and copy the sample command to run.
  </Tab>

  <Tab title="pulsar-admin">
    f. Select the target tenant, namespace and topic, and copy the sample command to your terminal and update the path of the OAuth2 key file before running.
  </Tab>
</Tabs>

## What’s next?

* Learn how to [develop functions](/cloud/process/pulsar-functions/develop-functions/function-develop-overview).
* Learn how to [manage functions](/cloud/process/pulsar-functions/function-manage).
* Learn how to [configure stateful functions](/cloud/process/pulsar-functions/function-state).
* Learn how to [monitor functions](/cloud/process/pulsar-functions/function-monitoring).
* Reference [common configurations](/cloud/process/pulsar-functions/function-config).


# Configure state storage (Private Preview)
Source: https://docs.streamnative.io/cloud/process/pulsar-functions/function-state



<Note title="Note">
  This feature is currently in private preview. If you want to try it out or have any questions, [submit a ticket](https://support.streamnative.io/hc/en-us/requests/new) to the support team.
</Note>

# Configure state storage for Pulsar Functions

StreamNative Pulsar Functions support **stateful functions** that can maintain state across function invocations. This allows you to build more complex and powerful stream processing applications.

It uses [Oxia](https://github.com/oxia-db/oxia) as a state storage interface.

States are key-value pairs, where a key is a string and its value is arbitrary binary data - counters are stored as 64-bit big-endian binary values. Keys are scoped to an individual function and shared between instances of that function.

To enable state storage for Pulsar Functions, you need to enable it explicitly when creating or updating a function by setting below arguments:

```
--custom-runtime-options '{"enableStateStore": true}'
```

<Note title="Note">
  State storage is only available for **Java** functions for now.
</Note>

## Call state APIs

Pulsar Functions expose below APIs for mutating and accessing `state`.

The following table outlines the states that can be accessed within Java functions.

| State-related API                       | Java                                   |
| --------------------------------------- | -------------------------------------- |
| [Increment counter](#increment-counter) | `incrCounter` <br />`incrCounterAsync` |
| [Retrieve counter](#retrieve-counter)   | `getCounter` <br />`getCounterAsync`   |
| [Update state](#update-state)           | `putState` <br />`putStateAsync`       |
| [Retrieve state](#retrieve-state)       | `getState` <br />`getStateAsync`       |
| [Delete state](#delete-state)           | `deleteState`                          |

## Increment counter

Use `incrCounter` to increment the counter of a given `key` by the given `amount`.
If the `key` does not exist, a new key is created.

```java theme={null}
    /**
     * Increment the built-in distributed counter referred by key
     * @param key The name of the key
     * @param amount The amount to be incremented
     */
    void incrCounter(String key, long amount);
```

To asynchronously increment the counter, you can use `incrCounterAsync`.

```java theme={null}
     /**
     * Increment the built-in distributed counter referred by key
     * but dont wait for the completion of the increment operation
     *
     * @param key The name of the key
     * @param amount The amount to be incremented
     */
    CompletableFuture<Void> incrCounterAsync(String key, long amount);
```

### Retrieve counter

Use `getCounter` to retrieve the counter of a given `key` mutated by `incrCounter`.

```java theme={null}
    /**
     * Retrieve the counter value for the key.
     *
     * @param key name of the key
     * @return the amount of the counter value for this key
     */
    long getCounter(String key);
```

To asynchronously retrieve the counter mutated by `incrCounterAsync`, you can use `getCounterAsync`.

```java theme={null}
     /**
     * Retrieve the counter value for the key, but don't wait
     * for the operation to be completed
     *
     * @param key name of the key
     * @return the amount of the counter value for this key
     */
    CompletableFuture<Long> getCounterAsync(String key);
```

### Update state

Besides the `counter` API, Pulsar also exposes a general key/value API for functions to store and update the state of a given `key`.

```java theme={null}
    /**
     * Update the state value for the key.
     *
     * @param key name of the key
     * @param value state value of the key
     */
    void putState(String key, ByteBuffer value);
```

To asynchronously update the state of a given `key`, you can use `putStateAsync`.

```java theme={null}
    /**
     * Update the state value for the key, but don't wait for the operation to be completed
     *
     * @param key name of the key
     * @param value state value of the key
     */
    CompletableFuture<Void> putStateAsync(String key, ByteBuffer value);
```

### Retrieve state

Use `getState` to retrieve the state of a given `key`.

```java theme={null}
    /**
     * Retrieve the state value for the key.
     *
     * @param key name of the key
     * @return the state value for the key.
     */
    ByteBuffer getState(String key);
```

To asynchronously retrieve the state of a given `key`, you can use `getStateAsync`.

```java theme={null}
    /**
     * Retrieve the state value for the key, but don't wait for the operation to be completed
     *
     * @param key name of the key
     * @return the state value for the key.
     */
    CompletableFuture<ByteBuffer> getStateAsync(String key);
```

### Delete state

```java theme={null}
    /**
     * Delete the state value for the key.
     *
     * @param key   name of the key
     */
    void deleteState(String key);
```

## Query state via CLI

You can also query function state using CLI commands. This is useful for debugging and monitoring stateful functions.

```bash theme={null}
bin/pulsar-admin functions querystate \
    --tenant <tenant> \
    --namespace <namespace> \
    --name <function-name> \
    --key <state-key> \
    [---watch]
```

If `--watch` is specified, the CLI tool keeps running to get the latest value of the provided `state-key`.

## Example

The example of `WordCountFunction` demonstrates how `state` is stored within Pulsar Functions.

1. The function splits the received `String` into multiple words using regex `\\.`.
2. For each `word`, the function increments `counter` by 1 via `incrCounter(key, amount)`.

   ```java theme={null}
   import org.apache.pulsar.functions.api.Context;
   import org.apache.pulsar.functions.api.Function;

   import java.util.Arrays;

   public class WordCountFunction implements Function<String, Void> {
       @Override
       public Void process(String input, Context context) throws Exception {
           Arrays.asList(input.split("\\.")).forEach(word -> context.incrCounter(word, 1));
           return null;
       }
   }
   ```


# Pulsar Functions Overview
Source: https://docs.streamnative.io/cloud/process/pulsar-functions/functions-overview



[Apache Pulsar Functions](https://pulsar.apache.org/docs/functions-overview/) are lightweight functions that consume messages from Pulsar topics, apply custom processing logic, and publish the results of the computation to other topics. With the initial release of this feature, you deploy your functions through the command line to StreamNative Cloud. You can also monitor your functions on the StreamNative Cloud Console, which visually displays all of your functions and their current state.

To learn more, watch the [Pulsar Functions on StreamNative Cloud](https://www.youtube.com/playlist?list=PL7-BmxsE3q4V8cMgsTtDA64OJtC25blxn) playlist.

Pulsar Functions are designed to perform lightweight stream processing. They are best for basic use cases that do not require the complexity of a full-stream processing engine.

* Simple per-message transformations for normalization, cleanup, or enriching with metadata.
* Simple aggregations like sums, counts, or averages for a single stream (over a short time period) and where duplicates are tolerated.
* Chained sequences of transformations on data in a single topic.

Pulsar Functions are computing infrastructure of Pulsar messaging system. With Pulsar Functions, you can create complex processing logic without deploying a separate neighboring system, such as [Apache Storm](http://storm.apache.org/), [Apache Heron](https://heron.incubator.apache.org/), or [Apache Flink](https://flink.apache.org/).

Pulsar Functions can be described as [Lambda](https://aws.amazon.com/lambda/)-style functions that are specifically designed to use Pulsar as a message bus.

For information about Pulsar Functions in general, see [Pulsar Functions Overview](https://pulsar.apache.org/docs/functions-overview/).

## What’s next?

* Learn how to [set up your enviroment](/cloud/process/pulsar-functions/function-setup).
* Learn how to [develop function](/cloud/process/pulsar-functions/develop-functions/function-develop-overview).
* Learn how to [manage functions](/cloud/process/pulsar-functions/function-manage).
* Learn how to [monitor functions](/cloud/process/pulsar-functions/function-monitoring).
* Learn how to [configure stateful functions](/cloud/process/pulsar-functions/function-state).
* Reference [common configurations](/cloud/process/pulsar-functions/function-config).
* Watch the [playlist for Pulsar Functions on StreamNative Cloud](https://www.youtube.com/playlist?list=PL7-BmxsE3q4V8cMgsTtDA64OJtC25blxn).
* Learn how to use [pfSQL (Alpha)](/cloud/process/pfsql/pfsql-overview)


# Understanding StreamNative Cloud objects
Source: https://docs.streamnative.io/cloud/references/cloud-object



This document explains how StreamNative Cloud objects are represented in the StreamNative Cloud API, and how you can express them in `.yaml` format.

StreamNative Cloud objects are persistent entities in the StreamNative Cloud system. StreamNative Cloud uses these entities to represent the state of your organization. Specifically, they can describe:

* The containerized applications that are running (and on which nodes)
* The resources available to those applications
* The policies about the operation way for those applications

Once you create a StreamNative Cloud object, the StreamNative Cloud system constantly works to ensure that the object exists. To work with StreamNative Cloud objects, you need to use the [StreamNative Cloud API](/api-references/rest-messaging-api/rest-messaging-api). When you use the `snctl` CLI tool, the CLI tool makes the necessary StreamNative Cloud API calls for you.

### Object names and IDs

Each object in your cluster has a [name](#names) that is unique for that type of resource. Every StreamNative Cloud object also has a [UID](#uids) that is unique across your whole cluster. For example, you can only have one pod named `myapp-1234` within the same organization.

#### Names

Below are three types of commonly used name constraints for resources.

* DNS subdomain names: most resources require a name that can be used as a DNS subdomain name as defined in [RFC 1123](https://tools.ietf.org/html/rfc1123).
  * Contain no more than 253 characters.
  * Contain only lowercase alphanumeric characters, '-' or '.'.
  * Start with an alphanumeric character.
  * End with an alphanumeric character.
* DNS label names: some resource types require their names to follow the DNS label standard as defined in [RFC 1123](https://tools.ietf.org/html/rfc1123).
  * Contain no more than 63 characters.
  * Contain only lowercase alphanumeric characters, '-'.
  * Start with an alphanumeric character.
  * End with an alphanumeric character.
* Path segment names: some resource types require their names to be able to be safely encoded as a path segment. In other words, the name might not be "." or ".." and the name might not contain "/" or "%".

#### UIDs

StreamNative Cloud UIDs are universally unique identifiers (also known as UUIDs). UUIDs are standardized as ISO/IEC 9834-8 and as ITU-T X.667.

### Object spec and status

Almost every StreamNative Cloud object includes the `spec` and the `status` fields to govern the object's configuration. The `spec` field describes the desired characteristics of the object. The `status` describes the current state of the object, supplied and updated by the StreamNative Cloud system and its components. The StreamNative Cloud continually and actively manages every object's actual state.

## Describe StreamNative Cloud objects

When you create an object in StreamNative Cloud, you must provide the `spec` field that describes its desired state, and some basic information about the object (such as a name). When you use the StreamNative Cloud API to create an object, that API request must include that information as JSON in the request body. **Most often, you provide the information to `snctl` in a `.yaml` file.** `snctl` converts the information to JSON when making the API request.

Here is an example of `.yaml` file that shows the required fields and the `spec`object for a cluster in StreamNative Cloud.

```yaml theme={null}
apiVersion: cloud.streamnative.io/v1alpha1
kind: PulsarCluster
metadata:
  namespace: matrix
  name: neo-1
spec:
  instanceName: neo
  location: us-east4
  broker:
    replicas: 1
  bookkeeper:
    replicas: 3
```

You can create a cluster by using the `snctl apply` command in the `snctl` CLI tool, passing the `.yaml` file as an argument. Here is an example.

```shell theme={null}
snctl apply -f /path/to/clusterneo1.yaml
```

The output is similar to this:

```sh theme={null}
cluster.cloud.streamnative.io/neo created
```

### Required fields

In the `.yaml` file for the StreamNative Cloud object that you want to create, you need to set values for the following fields:

* `apiVersion` : specify the version of the StreamNative Cloud API used to create this object.
* `kind`: specify the object to be created.
* `metadata`: specify the data that helps uniquely identify the object, including a `name` string and a `namespace` string.
* `spec` - specify the state you desire for the object.

The precise format of the `spec` field is different for every StreamNative Cloud object, and contains nested fields specific to that object.

## Manage StreamNative Cloud objects

The `snctl` CLI tool supports imperative commands to create and manage StreamNative Cloud objects. Imperative commands are simple, easy to learn and easy to remember. The imperative commands operate directly on live objects in an organization. You provide operations to the `snctl` command as arguments or flags. This is the simplest way to get started or to run a one-off task in an organization. Because this technique operates directly on live objects, it provides no history of previous configurations.

The following example shows how to create a cluster object `neo` using the imperative command.

```sh theme={null}
snctl create pulsarinstances neo
```


# Glossary of Terms & Concepts
Source: https://docs.streamnative.io/cloud/references/glossary



Are you new to StreamNative? Trying to learn and understand? Listed below are terms and concepts relevant to understanding StreamNative products.

<Note title="Note">
  If you have feedback about terms and definitions you'd like to see included in this glossary, please [email](mailto:training@streamnative.io) us.
</Note>

## Apache BookKeeper

BookKeeper is a distributed write-ahead log (WAL) system or distributed journal. Pulsar uses Apache BookKeeper for persistent message storage. By default, Pulsar persistently stores all unacknowledged messages on multiple BookKeeper bookies (storage nodes).

BookKeeper is a scalable, fault-tolerant, and low-latency storage service optimized for real-time workloads.

## Broker

A Broker receives messages from producers, stores them and then delivers them to subscribed consumers.

A broker is the message dispatcher responsible for sending and receiving messages from a client. You typically have multiple brokers in a cluster, so if messages get backed up or a broker goes down, another broker can take on the extra load. This transfer can happen quickly due to the stateless nature of brokers.

## Cluster

A cluster is a secure messaging environment within Pulsar. Each Pulsar cluster consists a set of 3 components in a geographical location.

* **Pulsar brokers** - set of brokers handling all the data going in and out of Pulsar (or client requests)
* **Metadata storage** - providing coordination and service discovery between services
* **Bookie ensemble** - set of bookies that retain copies of the messages

A cluster has two layers: a stateless serving layer (made up of brokers) and a stateful storage layer (made up of bookies). See also [Pulsar Architecture and Design](https://pulsar.apache.org/docs/4.0.x/concepts-architecture-overview/).

<Note title="Note">
  In StreamNative Console, you can create one and only one cluster for an instance.
</Note>

## Consumer

A consumer processes incoming messages and takes action based on the content of the message. In Pulsar, messages are sent to a specific topic, which is a logical name for a stream of data. A consumer subscribes to a topic and receives all messages published to that topic.

To receive a message, the consumer needs to send a request to the broker that handles this message. The message is dispatched when the client permits the broker to push it. Typically, the consumer uses a queue to accumulate the messages to consume (you can configure the receiverQueueSize).

Pulsar manages the subscription cursor which determines the starting position to read data for consumers. Specifically, consumers read from the earliest unacknowledged message. If you need to manually manage the cursor and customize the starting position for consumers, go to readers.

## Cursor

Subscriptions use cursors to manage messages. Pulsar can go back to a specific message using a cursor. A cursor is a "restart" point. Each subscription for a topic has a cursor. The cursor contains information about message acknowledgements. When a consumer reads and processes a message, it sends an acknowledgment to the Pulsar broker and the cursor is updated. Updating the cursor ensures that the consumer will not receive that message again — even when the consumer crashes, recovers, and reattaches to the subscription.

## Geo-Replication

Geo-replication is the replication of persistently stored message data across multiple clusters of a Pulsar instance.

You can produce and consume messages in different geo-locations. For example, your application may be publishing data in one region or market and you would like to process it for consumption in other regions or markets. Geo-replication in Pulsar enables you to do that.

## Instance

A Pulsar **instance** is a group of Pulsar clusters that act together as a single unit. Clusters can be distributed across geographical locations and can replicate amongst themselves using geo-replication. For details about how to work with instances, such as creating, editing, checking, and deleting instances, see [work with instances](/cloud/clusters/manage-instances/instance).

With StreamNative Cloud, you can create a regional Pulsar cluster. A regional cluster has replicas running on numerous Availability Zones (AZs) within a given region. This arrangement maximizes availability but involves more inter-zone network traffic. Currently, three AZs are used per regional cluster and StreamNative Cloud supports multi AZ only.

## Multi-Tenancy

Multi-tenancy allows you to support multiple organizations (or sub-organizations) within your company on a single platform. A single Pulsar cluster can support many tenants and allows you to map Pulsar topics to different teams, applications, or use cases. This hierarchical structure serves as the foundation of security and allows for unified, global management of multiple clusters.

The underlying components for multi-tenancy include:

* **Instance** - group of Pulsar clusters that act together as a single unit.
* **Cluster** - a group of brokers and bookies that create a secure messaging environment within Pulsar.
* **Tenant** - the administrative unit within a shared environment.
* **Namespace** - a grouping mechanism for related topics.
* **Topic** - a unit of storage that structures data in Pulsar and organizes messages into a stream.

See [Multi-tenancy Get Started Guide](id:get-started-multi-tenancy) or watch a [short video](https://youtu.be/QJJaT5GgbJY) to learn more about multi-tenancy in Pulsar.

## Namespace

A namespace is a grouping mechanism for related topics. It allows teams to keep their data and teams separate. Each namespace has its own policies.

You create a separate namespace for each application. A namespace allows the application to create and manage a hierarchy of topics. You can create any number of topics under the namespace. For example, the topic **my-tenant/app1** is a namespace for the application "app1" for "my-tenant".

The configuration policies you set on a namespace apply to all the topics created in that namespace. You can create multiple namespaces for a tenant using the StreamNative Console, REST API or the pulsar-admin CLI tool.

For details about how to create namespaces, see [work with namespace](/cloud/manage-data-streams/namespace).

For more information about namespace details, including permissions, backlog quotas, retention policies, bundles, and dispatch rates, see the [Concepts](/cloud/overview/concepts-overview#namespaces) section.

## Organization

In StreamNative Cloud, organizations are intended for use in environments with many users spread across multiple teams. They are used to divide cluster resources between multiple users (through resource quotas). Names of resources must be unique within an organization. Organizations cannot be nested and each snctl resource can only be in one organization.

When you sign up for service with StreamNative Cloud, you provide a descriptive name for your first organization. A system-generated, random string is also assigned to your organization upon creation. You can see the random string next to the descriptive name on the Dashboard, as shown in the figure below.

The Pulsar clusters and other resources are owned by your organization. Organizations are team-based. As an organization administrator, you control access to the organization by adding and removing members and by granting permissions to them.
Currently, you can't delete an organization through either the StreamNative Console or the CLI. If you need to delete an organization, please [submit a ticket](https://support.streamnative.io/hc/en-us/requests/new).

For details about how to create an organization, see [work with organizations](/cloud/security/access/resource-hierarchy/organizations#create-an-organization).

## Producer

A producer sends messages to a Pulsar topic.

Pulsar producers play a crucial role in the Pulsar messaging system by generating and publishing messages to topics, which can be consumed by one or more consumers for various use cases such as real-time event streaming, messaging, and data processing.

You create producers using various programming languages such as Java, Python, Go, and C++.

Producers can operate in different modes such as synchornous or asynchornous publishing.

* In synchronous mode, the producer blocks until the broker acknowledges receipt of the message.
* In asynchronous mode, the producer sends messages in the background, and the application can continue executing without waiting for acknowledgments.

Pulsar producers also support various features such as message batching, compression, and encryption, which allow for efficient and secure message transmission.

## Publish-Subscribe Model

The publish-subscribe (pub-sub) software design pattern provides a framework for exchanging messages between the sender of messages (publishers) and receivers of messages (subscribers).

In Pulsar, a publisher is called a **producer** and a subscriber is called a **consumer**.

View the [Pub-Sub Animation](/media/pubsub.gif) to step through how the pub-sub model works.

## Schema

Pulsar has a built-in schema registry that enables clients to upload data schemas on a per-topic basis. Those schemas dictate which data types are recognized as valid for that topic. Pulsar schema enables you to use language-specific types of data when constructing and handling messages from simple types like `string` to more complex application-specific types.

## Server Pool

A **server pool** is an abstract definition of the compute, storage, and networking needed to host Pulsar instances. Currently, only `shared` and `shared-aws` server pools are available for StreamNative Cloud.

The following table lists the relationship between the server pool where the instance is located and the location of clusters available for the instance.

| Server pool  | Description                           | Cluster location                                                                      |
| ------------ | ------------------------------------- | ------------------------------------------------------------------------------------- |
| `shared`     | Instances are hosted on Google Cloud. | `asia-south1`, `europe-west1`, `europe-west3`, `us-central1`, `us-east4`, `us-west1`  |
| `shared-aws` | Instances are hosted on AWS.          | `ap-south-1`, `ap-southeast-2`, `eu-central-1`, `eu-west-1`, `us-east-1`, `us-east-2` |

## Service Account

You can create **service accounts** to automate actions such as to authenticate bots that operate on your organization. For example, a GitHub action or Jenkins job can use a service account to automatically provision a Pulsar cluster.

When you create a service account, you receive a JSON document called a key file that contains the secret credentials for the service account. It is your responsibility to protect the key file. (You can use the key file to authenticate both the Cloud API and to managed Pulsar clusters.)

The StreamNative Console uses role-based access control. As an organization administrator, you grant permission to access resources by assigning roles to users and to service accounts. Role assignments control access to the StreamNative Cloud API and to the Pulsar clusters that you provision.

Each organization has a built-in `admin` role, allowing full control of organization resources, including "Super Admin" access to the organization's Pulsar clusters. Currently, all logged-in users have the same "admin" level access.

For details about how to create a service account, see [work with service accounts](/private-cloud/v1/streamnative-console/service-account).

## StreamNative Cloud

StreamNative Cloud is the industry's only fully-managed, cloud-native messaging and event streaming platform powered by Apache Pulsar. Apache Pulsar is an open-source, distributed pub/sub messaging and event streaming platform that enables industry leaders globally to build pub/sub messaging and event-driven applications at scale.

Built and operated by the original developers of Apache Pulsar and Apache BookKeeper, StreamNative Cloud provides a scalable, resilient, and secure messaging and event streaming platform for enterprises. You can sign up for StreamNative Cloud through the StreamNative website to create and manage StreamNative Cloud resources and Pulsar components.

## Subscription

A subscription is the binding between a topic (or a partition) and a consumer. It is a named configuration rule that determines how messages are delivered to consumers.

Consumers register their interest in a topic by creating a subscription. A topic can have multiple attached subscriptions.

In Pulsar, you have flexibility to use four different subscription modes:

* Exclusive - allows only a single consumer to be connected at a time.
* Failover - allows multiple consumers to subscribe to the same topic (in the case of failover)
* Shared - allows multiple consumers to attach to the same subscription.
* Key Shared - distributes messages using an ordering key.

For more information about subscriptions and subscription modes, see [Pulsar messaging model](https://pulsar.apache.org/docs/4.0.x/concepts-messaging/).

## Tenant

Pulsar was created from the ground up as a multi-tenant system. To support multi-tenancy, Pulsar has a concept of tenants. Tenants can be spread across clusters and can each have their own authentication and authorization scheme applied to them. A tenant is the administrative unit within a shared environment. You manage storage quotas, message TTL, and set isolation policies with tenants.

A tenant also provides a security boundary. You can spread tenants across clusters and apply an authentication and authorization scheme to each one. You can also isolate tenants to different clusters.

For details about how to create tenants, see [work with tenant](/cloud/manage-data-streams/tenant).

## Tiered Storage

Tiered storage is a storage architecture that uses multiple levels of storage media to optimize data management. Pulsar provides access to tiered storage for infinite message retention (without the need for external tools).

Instead of using your fast disks for historical data, you can leverage the use of third party cloud storage systems and move the data from BookKeeper into a more cost effective storage tier. Pulsar clients can still access the data, making the storage of huge volumes of data in Pulsar manageable by reducing operational burden and cost.

For more information, see [supported object storage solutions](https://pulsar.apache.org/docs/tiered-storage-overview/).

## Topic

A topic is a unit of storage that structures data in Pulsar and organizes messages into a stream.

You must provide a unique name for a topic. Topics are named using a URI structure to fully qualify the name in the form of:

`persistent://[tenant]/[namespace]/[topic]`

where `tenant` and `namespace` are the organizational units in the multi-tenancy model and the `topic` is an arbitrary string (usually named with alphanumeric and characters such as "\_" or "-").

For example: `persistent://[compliance]/[risk]/[risk-detection]`

Pulsar creates a topic under the namespace provided in the topic name automatically. You do not need to explicitly create topics. If no tenant or namespace is specified when a client creates a topic, the topic is created in the default tenant and namespace.

### Non-persistent topics

Pulsar also supports non-persistent topics, which are topics on which messages are never persisted to disk and live only in memory. When using non-persistent delivery, stopping a Pulsar broker or disconnecting a subscriber to a topic means that all in-transit messages are lost on that non-persistent topic. In non-persistent topics, brokers immediately deliver messages to all connected subscribers without persisting them in BookKeeper.

### Partitioned topics

Normal topics are served only by a single broker that limits the maximum throughput of the topic. Partitioned topics are a special type of topic handled by multiple brokers, allowing for higher throughput.

A partitioned topic is actually implemented as N internal topics, where N is the number of partitions. When publishing messages to a partitioned topic, each message is routed to one of several brokers. The distribution of partitions across brokers is handled automatically by Pulsar.

It's recommended to have at least one partition per topic so that you can add more partitions in the future. If there are zero partitions (a non-partitioned topic), you will not be able to add more partitions to the topic after it is created.

### Topic Bundle

Pulsar uses a distributed architecture where topics are partitioned across multiple brokers for scalability and fault-tolerance.

A topic bundle is a group of topics that are assigned to a single broker for handling. To achieve this, Pulsar divides the topics into bundles, where each bundle is assigned to a specific broker. This ensures that the load is evenly distributed across brokers and that each broker is responsible for a subset of topics.

When a new topic is created, Pulsar assigns it to a specific bundle based on the topic name and the number of bundles configured for the cluster. If the number of bundles changes, Pulsar will rebalance the topics across the new set of bundles.

Topic bundles are an important concept in Pulsar as they play a crucial role in ensuring that topics are evenly distributed across brokers and that the cluster is scalable and fault-tolerant.

## User

In StreamNative Console, users are identified by their email address and authenticated through social login, by a username/password combination, or through SSO. As an organization administrator, you invite users to an organization, and they receive an email to complete the registration.
For details about how to manage or invite a user, see [work with users](/cloud/security/authentication/user-accounts).

## Uniform Resource Name (URN)

An instance is identified by a Uniform Resource Name (URN). The format of instance URN is "urn:sn:pulsar:pulsar-instance-namespace:pulsar-instance-name". You can get the organization name and the instance name through the `snctl get organizations` and `snctl get pulsarinstance` commands. When a Pulsar client connects to a Pulsar cluster through the OAuth2 authentication method, the URN is a required field for OAuth2 authentication.


# Start building with StreamNative
Source: https://docs.streamnative.io/home

Stream with Apache Pulsar and Apache Kafka, build agents on the Agent Engine. Run it all on StreamNative Cloud.

<div>
  <div>
    <div>
      <p>
        StreamNative Platform
      </p>

      <h1>
        Start building<br />with StreamNative
      </h1>

      <p>
        Everything you need to stream with Apache Pulsar and Apache Kafka, and build agents on the Agent Engine. From first message to production.
      </p>

      <div>
        <a href="/cloud/get-started/quickstart-console">
          ▶ Quickstart
        </a>

        <a href="https://console.streamnative.cloud">
          <svg>
            <polyline />

            <line />
          </svg>

          Open Console
        </a>

        <a href="/api-references/cloudapi/cloud-api">
          <svg>
            <polyline />

            <polyline />
          </svg>

          API reference
        </a>
      </div>
    </div>

    <div>
      <CodeGroup>
        ```java Pulsar (Java) theme={null}
        import org.apache.pulsar.client.api.*;

        PulsarClient client = PulsarClient.builder()
          .serviceUrl("pulsar+ssl://your-cluster.streamnative.cloud:6651")
          .authentication(AuthenticationFactory.token("your-token"))
          .build();

        Producer<String> producer = client.newProducer(Schema.STRING)
          .topic("persistent://public/default/orders")
          .create();

        producer.send("Hello StreamNative!");
        ```

        ```python Pulsar (Python) theme={null}
        import pulsar

        client = pulsar.Client(
            'pulsar+ssl://your-cluster.streamnative.cloud:6651',
            authentication=pulsar.AuthenticationToken('your-token'),
        )

        producer = client.create_producer('persistent://public/default/orders')
        producer.send('Hello StreamNative!'.encode('utf-8'))

        client.close()
        ```

        ```go Pulsar (Go) theme={null}
        client, _ := pulsar.NewClient(pulsar.ClientOptions{
            URL: "pulsar+ssl://your-cluster.streamnative.cloud:6651",
            Authentication: pulsar.NewAuthenticationToken("your-token"),
        })

        producer, _ := client.CreateProducer(pulsar.ProducerOptions{
            Topic: "persistent://public/default/orders",
        })

        producer.Send(context.Background(), &pulsar.ProducerMessage{
            Payload: []byte("Hello StreamNative!"),
        })
        ```

        ```java Kafka (Java) theme={null}
        import org.apache.kafka.clients.producer.*;
        import java.util.Properties;

        Properties props = new Properties();
        props.put("bootstrap.servers", "your-cluster.streamnative.cloud:9093");
        props.put("security.protocol", "SASL_SSL");
        props.put("sasl.mechanism", "PLAIN");
        props.put("sasl.jaas.config",
          "org.apache.kafka.common.security.plain.PlainLoginModule required " +
          "username=\"$Token\" password=\"your-token\";");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

        try (Producer<String, String> producer = new KafkaProducer<>(props)) {
          producer.send(new ProducerRecord<>("orders", "Hello StreamNative!"));
        }
        ```

        ```bash Kafka (kcat) theme={null}
        kcat -P -b your-cluster.streamnative.cloud:9093 \
          -X security.protocol=SASL_SSL \
          -X sasl.mechanisms=PLAIN \
          -X sasl.username='$Token' \
          -X sasl.password='your-token' \
          -t orders <<< "Hello StreamNative!"
        ```
      </CodeGroup>
    </div>
  </div>

  <p>
    Platform
  </p>

  <h2>
    Choose how you stream
  </h2>

  <p>
    Pick the streaming protocol that matches your stack. StreamNative Cloud runs both natively on the same engine.
  </p>

  <CardGroup>
    <Card title="Pulsar" icon={<img src="https://mintcdn.com/streamnative/CDw7JxphxyMzStpZ/images/pulsar-icon.svg?fit=max&auto=format&n=CDw7JxphxyMzStpZ&q=85&s=045a675e6dd1cfdd79e726ede3d5177c" alt="Pulsar" style={{ width: '24px', height: '24px' }} />} href="/cloud/overview/cloud-overview">
      Native Apache Pulsar messaging with multi-tenancy, geo-replication, and tiered storage.

      * [Quickstart](/cloud/get-started/quickstart-console)
      * [Pulsar Clients](/clients/pulsar-clients/pulsar-clients-overview)
      * [API reference](/api-references/cloudapi/cloud-api)
    </Card>

    <Card title="Kafka" icon={<img src="https://mintcdn.com/streamnative/CDw7JxphxyMzStpZ/images/kafka-icon.svg?fit=max&auto=format&n=CDw7JxphxyMzStpZ&q=85&s=de5f67563aeadfe9b1741ddb7f6fde4f" alt="Kafka" style={{ width: '24px', height: '24px' }} />} href="/kafka/overview">
      Native Kafka API on the StreamNative Ursa engine. Bring your existing Kafka clients.

      * [Quickstart](/cloud/get-started/quickstart-kafka)
      * [Kafka Clients](/clients/kafka-clients/kafka-clients-overview)
      * [API reference](/api-references/kafka-rest-api/kafka-rest-api)
    </Card>
  </CardGroup>

  <p>
    Cluster types
  </p>

  <h2>
    Choose the right cluster for your workload
  </h2>

  <p>
    Pay-per-use, predictable, or in your own cloud account.
  </p>

  <CardGroup>
    <Card title="Serverless" icon="bolt" href="/cloud/clusters/cluster-types">
      **Burstable** — pay as you go. Best for event-driven workloads with variable throughput.
    </Card>

    <Card title="Dedicated" icon="layer-group" href="/cloud/clusters/cluster-types">
      **Predictable** — reserved capacity. Best for steady production workloads at scale.
    </Card>

    <Card title="BYOC" icon="cloud-arrow-up" href="/cloud/clusters/byoc/byoc-overview">
      **Your account** — runs in your AWS/Azure/GCP/Alibaba VPC. Data never leaves your perimeter.
    </Card>
  </CardGroup>

  <p>
    Developer journey
  </p>

  <h2>
    From idea to production
  </h2>

  <p>
    Follow the lifecycle or jump to what you need.
  </p>

  <Tabs>
    <Tab title="Pulsar">
      <Steps>
        <Step title="Get started">
          <CardGroup>
            <Card title="Quickstart" icon="play" href="/cloud/get-started/quickstart-console" />

            <Card title="Choose a cluster" icon="layer-group" href="/cloud/clusters/cluster-types" />

            <Card title="Pulsar overview" icon="circle-info" href="/cloud/overview/cloud-overview" />
          </CardGroup>
        </Step>

        <Step title="Build">
          <CardGroup>
            <Card title="Pulsar Clients" icon="code" href="/clients/pulsar-clients/pulsar-clients-overview" />

            <Card title="Topics & namespaces" icon="folder-tree" href="/cloud/manage-data-streams/data-streams-overview" />

            <Card title="Pulsar Functions" icon="bolt" href="/cloud/process/pulsar-functions/functions-overview" />
          </CardGroup>
        </Step>

        <Step title="Connect">
          <CardGroup>
            <Card title="Pulsar IO" icon="plug" href="/cloud/connect/pulsar-io/connector-overview" />

            <Card title="Connector Hub" icon="puzzle-piece" href="/connect/overview" />
          </CardGroup>
        </Step>

        <Step title="Operate">
          <CardGroup>
            <Card title="Service accounts" icon="key" href="/cloud/security/authentication/service-accounts/service-accounts" />

            <Card title="Networking" icon="network-wired" href="/cloud/networking/networking" />

            <Card title="Monitoring" icon="chart-line" href="/cloud/log-and-monitor/cloud-metrics-api" />
          </CardGroup>
        </Step>

        <Step title="Scale">
          <CardGroup>
            <Card title="Autoscaling" icon="up-right-and-down-left-from-center" href="/cloud/clusters/scale-clusters/cloud-autoscaling" />

            <Card title="Resize cluster" icon="arrow-up-right-dots" href="/cloud/clusters/scale-clusters/resize-a-cluster" />

            <Card title="Geo-replication" icon="globe" href="/cloud/clusters/cloud-geo-replication" />
          </CardGroup>
        </Step>
      </Steps>
    </Tab>

    <Tab title="Kafka">
      <Steps>
        <Step title="Get started">
          <CardGroup>
            <Card title="Quickstart" icon="play" href="/cloud/get-started/quickstart-kafka" />

            <Card title="Choose a cluster" icon="layer-group" href="/cloud/clusters/cluster-types" />

            <Card title="Kafka overview" icon="circle-info" href="/kafka/overview" />
          </CardGroup>
        </Step>

        <Step title="Build">
          <CardGroup>
            <Card title="Kafka Clients" icon="code" href="/clients/kafka-clients/kafka-clients-overview" />

            <Card title="Schemas" icon="diagram-project" href="/cloud/governance/kafka-schemas/kafka-schema-registry" />

            <Card title="Kafka Streams" icon="bolt" href="/cloud/process/kafka-streams-and-ksql/kafka-streams-and-ksql" />
          </CardGroup>
        </Step>

        <Step title="Migrate">
          <CardGroup>
            <Card title="Migration guide" icon="arrow-right-arrow-left" href="/kafka/kafka-migration-guide" />
          </CardGroup>
        </Step>

        <Step title="Operate">
          <CardGroup>
            <Card title="Service accounts" icon="key" href="/cloud/security/authentication/service-accounts/service-accounts" />

            <Card title="Networking" icon="network-wired" href="/cloud/networking/networking" />

            <Card title="Monitoring" icon="chart-line" href="/cloud/log-and-monitor/cloud-metrics-api" />
          </CardGroup>
        </Step>

        <Step title="Scale">
          <CardGroup>
            <Card title="Autoscaling" icon="up-right-and-down-left-from-center" href="/cloud/clusters/scale-clusters/cloud-autoscaling" />

            <Card title="Resize cluster" icon="arrow-up-right-dots" href="/cloud/clusters/scale-clusters/resize-a-cluster" />
          </CardGroup>
        </Step>
      </Steps>
    </Tab>
  </Tabs>

  <p>
    Resources
  </p>

  <h2>
    Keep learning
  </h2>

  <CardGroup>
    <Card title="Connector Hub" icon="puzzle-piece" href="/connect/overview">
      Browse the marketplace of pre-built connectors.
    </Card>

    <Card title="Release notes" icon="list-check" href="/release-notes/cloud/release-notes">
      What's new across StreamNative releases.
    </Card>
  </CardGroup>
</div>


# Build Consumer
Source: https://docs.streamnative.io/clients/pulsar-clients/go/tutorial/pulsar-go-build-consumer



Next, create the consumer application by pasting the following Go code into a file named `consumer.go`.

```go theme={null}
package main

import (
	"context"
	"fmt"
	"os"
	"os/signal"
	"syscall"
	"time"

	"github.com/apache/pulsar-client-go/pulsar"
)

func main() {

	client, err := pulsar.NewClient(pulsar.ClientOptions{
		URL:            "<Broker Service URL>",
		Authentication: pulsar.NewAuthenticationToken("<API KEY>"),
	})

	if err != nil {
		fmt.Printf("Failed to create Pulsar client: %s", err)
		os.Exit(1)
	}

	if err != nil {
		fmt.Printf("Failed to create consumer: %s", err)
		os.Exit(1)
	}

	defer client.Close()

	topic := "purchases"

	consumer, err := client.Subscribe(pulsar.ConsumerOptions{
		Topic:                       topic,
		SubscriptionName:            "pulsar-go-getting-started",
		SubscriptionInitialPosition: pulsar.SubscriptionPositionEarliest,
	})

	if err != nil {
		fmt.Printf("Failed to create Pulsar consumer: %s", err)
		os.Exit(1)
	}

	defer consumer.Close()

	// Set up a channel for handling Ctrl-C, etc
	sigchan := make(chan os.Signal, 1)
	signal.Notify(sigchan, syscall.SIGINT, syscall.SIGTERM)

	// Process messages
	run := true
	for run {
		select {
		case sig := <-sigchan:
			fmt.Printf("Caught signal %v: terminating\n", sig)
			run = false
		default:
			ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
			msg, err := consumer.Receive(ctx)
			cancel() // Clean up the context
			if err != nil {
				// Errors are informational and automatically handled by the consumer
				continue
			}
			fmt.Printf("Consumed event from topic %s: ID = %s, key = %-10s value = %s\n",
				topic, msg.ID(), msg.Key(), msg.Payload())

			consumer.Ack(msg)
		}
	}
}

```

Fill in the appropriate `<Broker Service URL>` and `<API KEY>` in the `URL` and `Authentication` properties where the consumer is instantiated via the `pulsar.NewClient` method. Compile the consumer as follows:

```bash theme={null}
go build -o out/consumer consumer.go
```


# Build Producer
Source: https://docs.streamnative.io/clients/pulsar-clients/go/tutorial/pulsar-go-build-producer



Let's create the producer application by pasting the following Go code into a file named `producer.go`.

```go theme={null}
package main

import (
	"context"
	"fmt"
	"math/rand"
	"os"

	"github.com/apache/pulsar-client-go/pulsar"
)

func main() {

	client, err := pulsar.NewClient(pulsar.ClientOptions{
		URL:            "<Broker Service URL>",
		Authentication: pulsar.NewAuthenticationToken("<API KEY>"),
	})

	if err != nil {
		fmt.Printf("Failed to create Pulsar client: %s", err)
		os.Exit(1)
	}

	users := [...]string{"eabara", "jsmith", "sgarcia", "jbernard", "htanaka", "awalther"}
	items := [...]string{"book", "alarm clock", "t-shirts", "gift card", "batteries"}
	topic := "purchases"

	producer, err := client.CreateProducer(pulsar.ProducerOptions{
		Topic: topic,
	})

	if err != nil {
		fmt.Printf("Failed to create Pulsar producer: %s", err)
		os.Exit(1)
	}

	defer producer.Close()

	for n := 0; n < 10; n++ {
		key := users[rand.Intn(len(users))]
		data := items[rand.Intn(len(items))]
		if msgId, err := producer.Send(context.Background(), &pulsar.ProducerMessage{
			Key:     key,
			Payload: []byte(data),
		}); err != nil {
			fmt.Printf("Failed to deliver message: %s", err)
		} else {
			fmt.Printf("Produced event to topic: ID = %s, key = %-10s value = %s\n", msgId, key, data)
		}
	}
}

```

Fill in the appropriate `<Broker Service URL>` endpoint and `<API KEY>` in the `URL` and `Authentication` properties where the producer is instantiated using the `pulsar.NewClient` method.

Compile the producer with the following:

```bash theme={null}
go build -o out/producer producer.go
```

If you get any errors during the build make sure that you initialized the module correctly per the instructions in the [previous step](/clients/pulsar-clients/go/tutorial/pulsar-go-create-project).


# Cluster Setup
Source: https://docs.streamnative.io/clients/pulsar-clients/go/tutorial/pulsar-go-cluster-setup



We are going to need a StreamNative Cloud cluster for our client application to operate with. This guide can help you create your first StreamNative Cloud cluster.

## Create a StreamNative Cloud cluster

From within the StreamNative Cloud console, creating a new cluster is just a few clicks.

<Tabs>
  <Tab title="Cloud Console">
    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Organizations** to list your created organizations.

    2. Click the name of your first organization. In the figure below, the organization name is `Demos`.

       <img alt="screenshot of organization section" />

    3. On the **Select an Instance** card of the **Organization Dashboard** page, click the `+` button.

    4. On the **Choose the deployment type for your instance**, click **Deploy Serverless** to start the instance creation process.

    5. On the **Instance Configuration** page, enter a name for your instance, and select a Cloud Provider. The instance name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

    6. Click **Cluster Location** to start the cluster creation process.

    7. On the **Cluster Location** page, enter a name for your cluster, select the cluster location, and then click **Finish**. The cluster name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

    The cluster page appears, displaying the cluster creation process. Depending on the chosen cloud provider and other settings, it may take a few seconds to several minutes to provision the cluster. Once the cluster has been provisioned, the **Cluster Dashboard** page will be displayed.

    Now you can get started configuring apps and data on your new cluster.
  </Tab>
</Tabs>

## Get the cluster service URLs

Next, you'll need to get your StreamNative Cloud cluster's Service URLs to configure your client applications. These URLs allow your applications to connect to and interact with your cluster.

Your cluster provides different Service URLs depending on which protocol you plan to use:

For Pulsar Protocol:

* **Broker Service URL**: Used to configure Pulsar clients to connect to the brokers for producing and consuming messages
* **HTTP Service URL**: Used for administrative operations via the Pulsar admin API

For Kafka Protocol:

* **Kafka Service URL**: Used as bootstrap servers in Kafka client configurations
* **Schema Registry URL**: Used to configure Kafka serializers and deserializers that work with schemas

You'll need these URLs in the upcoming steps when setting up your producer and consumer clients.

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

## Create a service account and API key

Next, to connect your client application to your cluster to produce and consume messages, you need to create a [Service Account](/cloud/security/authentication/service-accounts/manage-service-accounts) and choose an authentication mechanism: either [API Key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview) or [OAuth](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview).

API Key authentication is quicker to implement since you only need to create an API key in StreamNative Cloud. This guide uses API Key authentication.

<Note title="Note">
  Currently, you can't edit a service account. If you need a service account to have Super Admin access, make sure to enable it when creating the service account. By default, service accounts do not have Super Admin enabled.
</Note>

<Tabs>
  <Tab title="Cloud Console">
    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Accounts & Accesses**.

    2. On the left navigation pane, click **Service Accounts**.

    3. On the **Service Account** page, click **+ New Service Account**.

    4. On the **Create Service Account** dialog, enter a name for the service account, and then click **Confirm**.

    5. On the **Service Account** page, in the row of the service account you just created, click the **...** icon, and select **Create API Key** in the dropdown menu.

    6. On the **New API Key** dialog:

       * Enter a name for the API key
       * Set the expiration date
       * Select the instance you created in Step 2
       * Write a description for the API key
       * Click **Confirm**

    <Note title="Note">
      An API key and associated secret apply to the active StreamNative instance. If you add a new instance, you must create a new API key for producers and consumers on the new Pulsar instance. For more information, see [Use API Keys to Authenticate to StreamNative Cloud](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview).
    </Note>

    7. After the API key is created, you can see the API key shown in the **New API Key** dialog. Click the **Copy and close** button to copy the API key to your clipboard. **Please note that you cannot retrieve the API key later after closing the dialog.**
  </Tab>
</Tabs>

Note the API key as we will use them when configuring the producer and consumer clients in upcoming steps.


# Consume Messages
Source: https://docs.streamnative.io/clients/pulsar-clients/go/tutorial/pulsar-go-consume-messages



From another terminal, run the following command to run the consumer application which will read the events from the `purchases` topic and write the information to the terminal.

```bash theme={null}
./out/consumer
```

The consumer application will start and print any events it has not yet consumed and then wait for more events to arrive. On startup of the consumer, you should see output resembling this:

```bash theme={null}
Consumed event from topic purchases: ID = 890243:0:2, key = awalther   value = batteries
Consumed event from topic purchases: ID = 890243:1:2, key = htanaka    value = book
Consumed event from topic purchases: ID = 890243:2:2, key = jbernard   value = t-shirts
Consumed event from topic purchases: ID = 890243:3:2, key = eabara     value = gift card
Consumed event from topic purchases: ID = 890243:4:2, key = htanaka    value = batteries
Consumed event from topic purchases: ID = 890243:5:2, key = htanaka    value = book
Consumed event from topic purchases: ID = 890243:6:2, key = awalther   value = t-shirts
Consumed event from topic purchases: ID = 890243:7:2, key = awalther   value = t-shirts
Consumed event from topic purchases: ID = 890243:8:2, key = sgarcia    value = book
Consumed event from topic purchases: ID = 890243:9:2, key = awalther   value = gift card
```

Rerun the producer to see more events, or feel free to modify the code as necessary to create more or different events.

Once you are done with the consumer, enter `Ctrl-C` to terminate the consumer application.


# Create Project
Source: https://docs.streamnative.io/clients/pulsar-clients/go/tutorial/pulsar-go-create-project



Create a new directory anywhere you’d like for this project:

```bash theme={null}
mkdir pulsar-go-getting-started && cd pulsar-go-getting-started
```

Initialize the Go module and download the Pulsar Go client dependency:

```bash theme={null}
go mod init pulsar-go-getting-started
go get github.com/apache/pulsar-client-go
go get github.com/apache/pulsar-client-go/pulsar/auth
go get github.com/apache/pulsar-client-go/pulsar
```


# Create Topic
Source: https://docs.streamnative.io/clients/pulsar-clients/go/tutorial/pulsar-go-create-topic



In a StreamNative Cluster, topics are grouped into [tenants](/cloud/manage-data-streams/tenant) and [namespaces](/cloud/manage-data-streams/namespace). In this guide, we will use the existing `public/default` namespace.

## Authorize the Service Account

<Note title="Note">
  If you create a super-user service account, you can skip this step.
</Note>

Before configuring the producer and consumer clients, you need to authorize the service account to grant the necessary permissions for the service account to interact with your StreamNative Cloud cluster.

To grant the service account permissions on the namespace level, follow these steps:

<Tabs>
  <Tab title="Cloud Console">
    1. Navigate to the **Namespace Dashboard** page by [switching to the namespace workspace](/cloud/get-started/cloud-console#switch-a-namespace).

    2. On the **Namespace Dashboard** page, click **Configuration** on the left navigation pane.

    3. On the **Namespace configuration** page, click **ADD ROLE**, and select the service account that you want to authorize.

    4. On the drop-down menu below the service account, select the proper permissions to assign to the service account. There are six permissions in total:

    * `consume`: allow the service account to consume messages.
    * `produce`: allow the service account to publish messages.
    * `functions`: allow the service account to submit and manage functions.
    * `sinks`: allow the service account to create and manage sink connectors.
    * `sources`: allow the service account to create and manage source connectors.
    * `packages`: allow the service account to upload and manage pulsar packages. If you want to submit a customized function/connector, then you will need to upload the function/connector’s JAR/NAR/Python file first, which requires the `packages` permission.

      <img alt="Authorize Service Account" />
  </Tab>
</Tabs>

In this guide, we will authorize the service account with the `produce` and `consume` permissions for the `public/default` namespace.

## Create a Topic

Navigate to the **Namespace Dashboard** page of the `public/default` namespace and follow the instructions below to create a topic, `purchases`, which you will use to produce and consume events.

<Tabs>
  <Tab title="StreamNative Console">
    To create a topic, follow these steps.

    1. Navigate to the desired **Namespace Dashboard** page by [switching to the namespace workspace](/cloud/get-started/cloud-console#switch-a-namespace).

    2. On the left navigation pane, under **Resources**, click **Topics**.

    3. Click **New Topic**.

       <img alt="screenshot of new topic dialog box" />

    4. Configure the topic, as outlined in the following table.

       | Item       | Description                                                                                                                                                                                                                                                                    |
       | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
       | Persistent | Configure the type of the topic. <br />- Persistent: messages in a persistent topic are durably persisted on the storage disk. <br />- Non-persistent: messages in a non-persistent topic are not persisted on the storage disk. <br /> By default, it is set to *Persistent*. |
       | Topic Name | Enter a name for the topic. It is a string of up to 40 characters, supporting lowercase letters (a-z), numeric characters (0-9), and the special character hyphen (-).                                                                                                         |
       | Partitions | (Optional) Configure the number of partitions for a partitioned topic. You can have from 1 up to 100 partitions per topic. It's recommended to have at least 1 partition.                                                                                                      |

    5. Click **Confirm**.
  </Tab>
</Tabs>


# Introduction
Source: https://docs.streamnative.io/clients/pulsar-clients/go/tutorial/pulsar-go-introduction



In this tutorial, you will build Go client applications which produce and consume messages to and from a StreamNative Cloud cluster.

As you learn how to run your first Pulsar application, we recommend using [StreamNative Cloud](https://streamnative.io/deployment) so you don't have to run your own Pulsar cluster and can focus on client development.

If you do not already have an account, you can follow the steps below to sign up for StreamNative Cloud. New signups receive \$200 to spend within StreamNative Cloud. No credit card is required for the first 30 days or until your credits run out.

## Sign up for StreamNative Cloud

<Note title="Note">
  If you have an email account configured for using Single Sign-On (SSO) with StreamNative Cloud, use that email address and password when signing up.
</Note>

To sign up, navigate to the [StreamNative Cloud Console login page](https://console.streamnative.cloud/?defaultMethod=signup). Follow the prompts to create an account.

After you click **Finish**, you might have to wait briefly for your first organization to be created.


# Prerequisites
Source: https://docs.streamnative.io/clients/pulsar-clients/go/tutorial/pulsar-go-prerequisites



Using Windows? You'll need to download [Windows Subsystem for Linux](https://learn.microsoft.com/en-us/windows/wsl/install).

This guide assumes that you already have the [Go language tools (version 1.18 or later)](https://go.dev/doc/install) installed.


# Produce Messages
Source: https://docs.streamnative.io/clients/pulsar-clients/go/tutorial/pulsar-go-produce-messages



Execute the compiled producer binary in order to produce messages to the `purchases` topic.

```bash theme={null}
./out/producer
```

You should see output resembling this:

```bash theme={null}
Produced event to topic: ID = 890243:0:2, key = awalther   value = batteries
Produced event to topic: ID = 890243:1:2, key = htanaka    value = book
Produced event to topic: ID = 890243:2:2, key = jbernard   value = t-shirts
Produced event to topic: ID = 890243:3:2, key = eabara     value = gift card
Produced event to topic: ID = 890243:4:2, key = htanaka    value = batteries
Produced event to topic: ID = 890243:5:2, key = htanaka    value = book
Produced event to topic: ID = 890243:6:2, key = awalther   value = t-shirts
Produced event to topic: ID = 890243:7:2, key = awalther   value = t-shirts
Produced event to topic: ID = 890243:8:2, key = sgarcia    value = book
Produced event to topic: ID = 890243:9:2, key = awalther   value = gift card
```


# What's Next
Source: https://docs.streamnative.io/clients/pulsar-clients/go/tutorial/pulsar-go-whats-next



* For the Pulsar Go client API, checkout the [Pulsar Go Client documentation](https://pulsar.apache.org/docs/client-libraries-python/)
* For details on the Go API, checkout the [Go documentation](https://pkg.go.dev/github.com/apache/pulsar-client-go/pulsar)


# Connect to StreamNative Cloud Using Pulsar Clients
Source: https://docs.streamnative.io/clients/pulsar-clients/pulsar-clients-overview



StreamNative Cloud supports Pulsar clients, allowing you to develop applications using your preferred programming language, IDE, and test framework through the Pulsar Protocol. The following sections provide working examples that demonstrate how to read from, process, and write data to StreamNative Cloud using Pulsar clients.

## Pulsar Clients

* [Go](/clients/pulsar-clients/go/tutorial/pulsar-go-introduction)


# Build applications using MQTT
Source: https://docs.streamnative.io/cloud/build/mqtt-clients/mqtt-on-cloud-overview



<Note title="Note">
  This feature is currently in Public Preview.
</Note>

## Overview

Migrating foundational message queues can be a highly challenging task, requiring coordination among multiple teams and carrying significant risks. The migration process often involves inevitable modifications to code and data migration when transitioning to a new product. As Pulsar gains popularity, more and more users are drawn to its outstanding features. But the cost of migration presents a barrier to trying it out.

The [MQTT protocol handler aka MoP](https://github.com/streamnative/mop) solves this problem, providing MQTT protocol support for Pulsar and facilitating migrating to Pulsar from MQTT without code changes.

## Compatibility

MoP is available on all StreamNative Clusters with the following specifications:

* Pulsar version >= 3.0

You can determine if MoP is enabled on your cluster by checking the MQTT Service URL (TCP) on the Cluster Details page. Additionally, the MQTT clients will also be enabled if MoP is enabled.

<img alt="mop-enabled" />

## Feature Overview

MoP supports the vast majority of the MQTT protocol. For a more detailed breakdown, see the following:

| Version | MQTT | MoP |
| ------- | ---- | --- |
| 3.1     | YES  | YES |
| 3.11    | YES  | YES |
| 5.0     | YES  | YES |

## MQTT Client Page Wizard

To help you get started with setting up MQTT client libraries and tools after provisioning your cluster, StreamNative Console provides a step-by-step wizard to walk you through the basic setup and configuration process, such as selecting or creating service accounts, downloading key files or tokens, installing client libraries, generating sample codes to run, and so on.

To get started with the MQTT client setup wizard, follow these steps.

1. On the left navigation pane of StreamNative Console, in the **Admin** section, click **MQTT Clients**.

2. Follow the wizard to generate the sample code you need for connecting to your Pulsar cluster.

With a copy-and-paste, you can run the given sample code to produce and consume messages.

## How MQTT Topics map to Pulsar Topics

For Apache Pulsar, The topic name consists of 4 parts:

```
<domain>://<tenant>/<namespace>/<local-name>
```

And / is not allowed in the local topic name. But for the MQTT topic name can have multiple levels such as:

```
/a/b/c/d/e/f
```

MoP mapping the MQTT topic name to Pulsar topic name as follows:

1. If the MQTT topic name does not start with the topic domain, MoP treats the URL encoded MQTT topic name as the Pulsar local topic name, and the default tenant and default namespace will be used to map the Pulsar topic name.
2. If the MQTT topic name starts with the topic domain, MoP will treat the first level topic name as the tenant and the second level topic name as the namespace and the remaining topic name levels will be covert as the local topic name with URL encoded.

## Related Links

* [MQTT-on-Pulsar GitHub Repo](https://github.com/streamnative/mop)


# Connect to your cluster using the Pulsar C++ client
Source: https://docs.streamnative.io/cloud/build/pulsar-clients/cloud-connect-cpp



This document describes how to connect to a cluster using a C++ client, and use the C++ producer and consumer to produce and consume messages to and from a topic. The C++ client supports connecting to a StreamNative cluster using either [OAuth2](#use-oauth2) or [API Keys](#use-apikeys) authentication.

<Note title="Note">
  This document assumes that you have created a StreamNative cluster and a service account, and have granted the service account `produce` and `consume` permissions to the namespace for the target topic.
</Note>

## Connect to your cluster using API keys

<span />

To connect a StreamNative cluster using [API keys](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview), follow these steps.

### Step 1: Get the broker service URL of your cluster

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

### Step 2: Create an API key of your service account

<Note title="Note">
  Before using an API key, verify that the service account is authorized to access the resources, such as tenants, namespaces, and topics.
</Note>

You can follow the instructions to [create an API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#using-api-keys-to-connect-to-your-cluster) for the service account you choose to use.

### Step 3: Connect to your cluster

For a complete example of how to connect to a cluster using the Pulsar C++ client, see [C++ client examples](https://github.com/streamnative/cloud-manager/tree/master/ui/src/data/code/clients/cpp).

#### Create a C++ consumer to consume messages

You can create and configure a C++ consumer to consume messages using Token authentication as follows. For more information about the placeholders in the code sample, see [parameters for Token authentication](/cloud/build/pulsar-clients/cloud-connect-cpp#parameters-for-token-authentication).

```cpp theme={null}
#include <pulsar/Client.h>
#include <thread>

using namespace pulsar;

int main() {
    ClientConfiguration clientConfig;
    clientConfig.setAuth(AuthToken::createWithToken("${apikey}"));
    Client client("${brokerServiceURL}", clientConfig);

    Consumer consumer;
    ConsumerConfiguration consumerConfig;
    consumerConfig.setSubscriptionInitialPosition(InitialPositionEarliest);
    Result result = client.subscribe("persistent://${tenant}/${namespace}/${topic}", "${subscription}", consumerConfig, consumer);
    if (result != ResultOk) {
        std::cout << "Failed to subscribe: " << result << std::endl;
        return -1;
    }

    Message msg;
    int ctr = 0;

    while (ctr < 10) {
        consumer.receive(msg);
        std::cout << "Received: " << msg
            << "  with payload '" << msg.getDataAsString() << "'" << std::endl;

        consumer.acknowledge(msg);
        ctr++;
    }

    std::cout << "Finished consuming synchronously!" << std::endl;

    client.close();
    return 0;
}
```

#### Create a C++ producer to produce messages

You can create and configure a C++ producer to produce messages using Token authentication as follows. For more information about the placeholders in the code sample, see [parameters for Token authentication](/cloud/build/pulsar-clients/cloud-connect-cpp#parameters-for-token-authentication).

```cpp theme={null}
#include <pulsar/Client.h>
#include <thread>

using namespace pulsar;

int main() {
    ClientConfiguration config;
    config.setAuth(AuthToken::createWithToken("${apikey}"));
    Client client("${brokerServiceURL}", config);

    Producer producer;
    Result result = client.createProducer("persistent://${tenant}/${namespace}/${topic}", producer);
    if (result != ResultOk) {
        std::cout << "Error creating producer: " << result << std::endl;
        return -1;
    }

    int ctr = 0;
    while (ctr < 10) {
        std::string content = "msg" + std::to_string(ctr);
        Message msg = MessageBuilder().setContent(content).setProperty("x", "1").build();
        Result result = producer.send(msg);
        if (result != ResultOk) {
            std::cout << "The message " << content << " could not be sent, received code: " << result << std::endl;
        } else {
            std::cout << "The message " << content << " sent successfully" << std::endl;
        }

        std::this_thread::sleep_for(std::chrono::milliseconds(100));
        ctr++;
    }

    std::cout << "Finished producing synchronously!" << std::endl;

    client.close();
    return 0;
}
```

#### Parameters for Token authentication

* `${brokerServiceURL}`: the broker service URL of your StreamNative cluster.
* `${apikey}`: an API key of your service account.
* `${tenant}/${namespace}/${topic}`: the full name of the topic for message production & consumption. It is a combination of the tenant name, the namespace name and the topic name.
* `${subscription}`: the name of the subscription that will determine how messages are delivered.

## Connect to your cluster using OAuth2 authentication

<span />

To connect a StreamNative cluster using OAuth2 authentication, follow these steps.

### Step 1: Get the broker service URL of your cluster

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

### Step 2: Get the OAuth2 credential file of your service account

To get an OAuth2 credential file of a service account through the StreamNative Console, follow these steps.

1. On the left navigation pane, click **Service Accounts**.

2. In the row of the service account you want to use, in the **Key File** column, click the **Download** icon to download the OAuth2 credential file to your local directory.

   The OAuth2 credential file should be something like this:

   ```json theme={null}
   {
     "type": "SN_SERVICE_ACCOUNT",
     "client_id": "CLIENT_ID",
     "client_secret": "CLIENT_SECRET",
     "client_email": "test@auth.streamnative.cloud",
     "issuer_url": "https://auth.streamnative.cloud"
   }
   ```

### Step 3: Connect to your cluster

For a complete example of how to connect to a cluster through the Pulsar C++ client, see [C++ client examples](https://github.com/streamnative/cloud-manager/tree/master/ui/src/data/code/clients/cpp).

#### Create a C++ consumer to consume messages

You can create and configure a C++ consumer to consume messages using the OAuth2 credential file as follows. For more information about the placeholders in the code sample, see [parameters for OAuth2 authentication](/cloud/build/pulsar-clients/cloud-connect-cpp#parameters-for-oauth2-authentication).

```cpp theme={null}
#include <pulsar/Client.h>
#include <thread>

using namespace pulsar;

int main() {
    ClientConfiguration clientConfig;
     // Replace YOUR-KEY-FILE-PATH with the absolute path or your downloaded JSON key file:
   std::string params = R"({
    "issuer_url": "https://auth.streamnative.cloud/",
    "private_key": "{{ file://YOUR-KEY-FILE-PATH }}",
    "audience": "urn:sn:pulsar:${orgName}:${instanceName}"})";

    clientConfig.setAuth(pulsar::AuthOauth2::create(params));
    Client client("${brokerServiceURL}", clientConfig);

    Consumer consumer;
    ConsumerConfiguration consumerConfig;
    consumerConfig.setSubscriptionInitialPosition(InitialPositionEarliest);
    Result result = client.subscribe("persistent://${tenant}/${namespace}/${topic}", "${subscription}", consumerConfig, consumer);
    if (result != ResultOk) {
        std::cout << "Failed to subscribe: " << result << std::endl;
        return -1;
    }

    Message msg;
    int ctr = 0;

    while (ctr < 10) {
        consumer.receive(msg);
        std::cout << "Received: " << msg
            << "  with payload '" << msg.getDataAsString() << "'" << std::endl;

        consumer.acknowledge(msg);
        ctr++;
    }

    std::cout << "Finished consuming synchronously!" << std::endl;

    client.close();
    return 0;
}
```

#### Create a C++ producer to produce messages

You can create and configure a C++ producer to produce messages using the OAuth2 credential file as follows. For more information about the placeholders in the code sample, see [parameters for OAuth2 authentication](/cloud/build/pulsar-clients/cloud-connect-cpp#parameters-for-oauth2-authentication).

```cpp theme={null}
#include <pulsar/Client.h>
#include <thread>

using namespace pulsar;

int main() {
    ClientConfiguration config;
    // Replace YOUR-KEY-FILE-PATH with the absolute path or your downloaded JSON key file:
    std::string params = R"({
    "issuer_url": "https://auth.streamnative.cloud/",
    "private_key": "/YOUR-KEY-FILE-PATH",
    "audience": "urn:sn:pulsar:${orgName}:${instanceName}"})";

    config.setAuth(pulsar::AuthOauth2::create(params));
    Client client("${brokerServiceURL}", config);

    Producer producer;
    Result result = client.createProducer("persistent://${tenant}/${namespace}/${topic}", producer);
    if (result != ResultOk) {
        std::cout << "Error creating producer: " << result << std::endl;
        return -1;
    }

    int ctr = 0;
    while (ctr < 10) {
        std::string content = "msg" + std::to_string(ctr);
        Message msg = MessageBuilder().setContent(content).setProperty("x", "1").build();
        Result result = producer.send(msg);
        if (result != ResultOk) {
            std::cout << "The message " << content << " could not be sent, received code: " << result << std::endl;
        } else {
            std::cout << "The message " << content << " sent successfully" << std::endl;
        }

        std::this_thread::sleep_for(std::chrono::milliseconds(100));
        ctr++;
    }

    std::cout << "Finished producing synchronously!" << std::endl;

    client.close();
    return 0;
}
```

#### Parameters for OAuth2 authentication

* `private_key`: your downloaded OAuth2 credential. This parameter supports the following two pattern formats:
  * `file:///path/to/file`: the path to your downloaded OAuth2 credential file.
  * `data:application/json;base64,<base64-encoded value>`: the credential file content encoded into Base64 format.
* `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
  * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
  * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
* `${brokerServiceURL}`: the broker service URL of your StreamNative cluster.
* `${tenant}/${namespace}/${topic}`: the full name of the topic for message production & consumption. It is a combination of the tenant name, the namespace name and the topic name.
* `${subscription}`: the name of the subscription that will determine how messages are delivered.


# Connect to your cluster using the Pulsar C# client
Source: https://docs.streamnative.io/cloud/build/pulsar-clients/cloud-connect-dotnet



This document describes how to connect to a cluster using a C# client, and use the C# producer and consumer to produce and consume messages to and from a topic. The C# client supports connecting to a StreamNative cluster using either [OAuth2](#use-oauth2) or [API Keys](#use-apikeys) authentication.

<Note title="Note">
  This document assumes that you have created a StreamNative cluster and a service account, and have granted the service account `produce` and `consume` permissions to the namespace for the target topic.

  This guide uses the open source Pulsar .NET client from the `fsprojects/pulsar-client-dotnet` repository: [fsprojects/pulsar-client-dotnet](https://github.com/fsprojects/pulsar-client-dotnet).
</Note>

## Prerequisites

Install the Pulsar .NET client package from NuGet. For package details and versions, see [Pulsar.Client on NuGet](https://www.nuget.org/packages/Pulsar.Client).

<Note>
  Choose the `Pulsar.Client` major version based on your target framework:

  * If you target **.NET 8.0 or later**, install **`Pulsar.Client` 3.x**.
  * If you don't target .NET 8.0+ and your project supports **.NET Standard 2.0**, install **`Pulsar.Client` 2.x**.
</Note>

```bash theme={null}
dotnet add package Pulsar.Client
```

## Connect to your cluster using API keys

<span />

To connect a StreamNative cluster using [API keys](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview), follow these steps.

### Step 1: Get the broker service URL of your cluster

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

### Step 2: Create an API key of your service account

<Note title="Note">
  Before using an API key, verify that the service account is authorized to access the resources, such as tenants, namespaces, and topics.
</Note>

You can follow the instructions to [create an API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#using-api-keys-to-connect-to-your-cluster) for the service account you choose to use.

### Step 3: Connect to your cluster

For a complete example of how to connect to a cluster using the C# client, see [C# client examples](https://github.com/streamnative/cloud-manager/tree/master/ui/src/data/code/clients/csharp).

#### Create a C# consumer to consume messages

You can create and configure a C# consumer to consume messages using Token authentication as follows. For more information about the placeholders in the code sample, see [parameters for Token authentication](/cloud/build/pulsar-clients/cloud-connect-dotnet#parameters-for-token-authentication).

```csharp theme={null}
using System;
using System.Text;
using System.Threading.Tasks;
using Pulsar.Client.Api;

namespace CsharpExamples
{
    internal class JWT
    {
        internal static async Task Consumer()
        {
            const string serviceUrl = "${brokerServiceURL}";
            const string subscriptionName = "${subscription}";
            var topicName = "persistent://${tenant}/${namespace}/${topic}";
            var token = "${apikey}";

            var client = await new PulsarClientBuilder()
                .ServiceUrl(serviceUrl)
                .Authentication(AuthenticationFactory.Token(token))
                .BuildAsync();

            var consumer = await client.NewConsumer()
                .Topic(topicName)
                .SubscriptionName(subscriptionName)
                .SubscribeAsync();

            for(int i=0; i< 10; i++){
                var message = await consumer.ReceiveAsync();
                Console.WriteLine($"Received: {Encoding.UTF8.GetString(message.Data)}");

                await consumer.AcknowledgeAsync(message.MessageId);
            }
        }
    }
}
```

#### Create a C# producer to produce messages

You can create and configure a C# producer to produce messages using Token authentication as follows. For more information about the placeholders in the code sample, see [parameters for Token authentication](/cloud/build/pulsar-clients/cloud-connect-dotnet#parameters-for-token-authentication).

```csharp theme={null}
using System;
using System.Text;
using System.Threading.Tasks;
using Pulsar.Client.Api;

namespace CsharpExamples
{
    internal class JWT
    {
        internal static async Task Producer()
        {
            const string serviceUrl = "${brokerServiceURL}";
            var topicName = "persistent://${tenant}/${namespace}/${topic}";
            var token = "${apikey}";

            var client = await new PulsarClientBuilder()
                .ServiceUrl(serviceUrl)
                .Authentication(AuthenticationFactory.Token(token))
                .BuildAsync();

            var producer = await client.NewProducer()
                .Topic(topicName)
                .CreateAsync();

            for(int i=0; i< 10; i++){
                var messageId = await producer.SendAsync(Encoding.UTF8.GetBytes($"Sent from C# at '{DateTime.Now}'"));
                Console.WriteLine($"MessageId is: '{messageId}'");
            }
        }
    }
}
```

#### Parameters for Token authentication

* `${brokerServiceURL}`: the broker service URL of your StreamNative cluster.
* `${subscription}`: the name of the subscription that will determine how messages are delivered.
* `${tenant}/${namespace}/${topic}`: the full name of the topic for message production & consumption. It is a combination of the tenant name, the namespace name and the topic name.
* `${apikey}`: an API key of your service account.

## Connect to your cluster using OAuth2 authentication

<span />

To connect a StreamNative cluster using OAuth2 authentication, follow these steps.

### Step 1: Get the broker service URL of your cluster

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

### Step 2: Get the OAuth2 credential file of your service account

To get an OAuth2 credential file of a service account through the StreamNative Console, follow these steps.

1. On the left navigation pane, click **Service Accounts**.

2. In the row of the service account you want to use, in the **Key File** column, click the **Download** icon to download the OAuth2 credential file to your local directory.

   The OAuth2 credential file should be something like this:

   ```json theme={null}
   {
     "type": "SN_SERVICE_ACCOUNT",
     "client_id": "CLIENT_ID",
     "client_secret": "CLIENT_SECRET",
     "client_email": "test@auth.streamnative.cloud",
     "issuer_url": "https://auth.streamnative.cloud"
   }
   ```

### Step 3: Connect to your cluster

For a complete example of how to connect to a cluster using the C# client, see [C# client examples](https://github.com/streamnative/cloud-manager/tree/master/ui/src/data/code/clients/csharp).

#### Create a C# consumer to consume messages

You can create and configure a C# consumer to consume messages using the OAuth2 credential file as follows. For more information about the placeholders in the code sample, see [parameters for OAuth2 authentication](/cloud/build/pulsar-clients/cloud-connect-dotnet#parameters-for-oauth2-authentication).

```csharp theme={null}
using System;
using System.IO;
using System.Reflection;
using System.Text;
using System.Threading.Tasks;
using Pulsar.Client.Api;
using Pulsar.Client.Common;

namespace CsharpExamples
{
    internal class Oauth2
    {
        internal static async Task RunOauth()
        {
            var fileUri = new Uri("{{ file://YOUR-KEY-FILE-PATH }}");
            var issuerUrl = new Uri("https://auth.streamnative.cloud/");
            var audience = "urn:sn:pulsar:${orgName}:${instanceName}";

            const string serviceUrl = "${brokerServiceURL}";
            var topicName = "persistent://${tenant}/${namespace}/${topic}";
            const string subscriptionName = "${subscription}";

            var client = await new PulsarClientBuilder()
                .ServiceUrl(serviceUrl)
                .Authentication(AuthenticationFactoryOAuth2.ClientCredentials(issuerUrl, audience, fileUri))
                .BuildAsync();

            var consumer = await client.NewConsumer()
                .Topic(topicName)
                .SubscriptionName(subscriptionName)
                .SubscribeAsync();

            for(int i=0; i< 10; i++){
                var message = await consumer.ReceiveAsync();
                Console.WriteLine($"Received: {Encoding.UTF8.GetString(message.Data)}");

                await consumer.AcknowledgeAsync(message.MessageId);
            }
        }
    }
}
```

#### Create a C# producer to produce messages

You can create and configure a C# producer to produce messages using the OAuth2 credential file as follows. For more information about the placeholders in the code sample, see [parameters for OAuth2 authentication](/cloud/build/pulsar-clients/cloud-connect-dotnet#parameters-for-oauth2-authentication).

```csharp theme={null}
using System;
using System.IO;
using System.Reflection;
using System.Text;
using System.Threading.Tasks;
using Pulsar.Client.Api;
using Pulsar.Client.Common;

namespace CsharpExamples
{
    internal class Oauth2
    {
        internal static async Task RunOauth()
        {
            var fileUri = new Uri("{{ file://YOUR-KEY-FILE-PATH }}");
            var issuerUrl = new Uri("https://auth.streamnative.cloud/");
            var audience = "urn:sn:pulsar:${orgName}:${instanceName}";

            const string serviceUrl = "${brokerServiceURL}";
            var topicName = "persistent://${tenant}/${namespace}/${topic}";

            var client = await new PulsarClientBuilder()
                .ServiceUrl(serviceUrl)
                .Authentication(AuthenticationFactoryOAuth2.ClientCredentials(issuerUrl, audience, fileUri))
                .BuildAsync();

            var producer = await client.NewProducer()
                .Topic(topicName)
                .CreateAsync();

            for(int i=0; i< 10; i++){
                var messageId = await producer.SendAsync(Encoding.UTF8.GetBytes($"Sent from C# at '{DateTime.Now}'"));
                Console.WriteLine($"MessageId is: '{messageId}'");
            }
        }
    }
}
```

#### Parameters for OAuth2 authentication

* `fileUri`: your downloaded OAuth2 credential. This parameter supports the following two pattern formats:
  * `file:///path/to/file`: the path to your downloaded OAuth2 credential file.
  * `data:application/json;base64,<base64-encoded value>`: the credential file content encoded into Base64 format.
* `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
  * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
  * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
* `${brokerServiceURL}`: the broker service URL of your StreamNative cluster.
* `${tenant}/${namespace}/${topic}`: the full name of the topic for message production & consumption. It is a combination of the tenant name, the namespace name and the topic name.
* `${subscription}`: the name of the subscription that will determine how messages are delivered.


# Connect to your cluster using the Pulsar Go client
Source: https://docs.streamnative.io/cloud/build/pulsar-clients/cloud-connect-go



This example shows how to connect to a cluster using a Go client and use the Go producer and consumer to produce and consume messages to and from a topic. The Go client supports connecting to a StreamNative cluster using either [OAuth2](#use-oauth2) or [API Keys](#use-apikeys) authentication.

<Note title="Note">
  This document assumes that you have created a StreamNative cluster and a service account, and have granted the service account `produce` and `consume` permissions to the namespace for the target topic.
</Note>

## Prerequisites

* Go 1.11 or higher version
* Go client 0.1.1+ (without 0.1.1)

For more information, see the [installation instructions](http://golang.org/doc/install).

## Connect to your cluster using API keys

<span />

To connect to a StreamNative cluster using [API keys](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview), follow these steps.

### Step 1: Get the broker service URL of your cluster

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

### Step 2: Create an API key of your service account

<Note title="Note">
  Before using an API key, verify that the service account is authorized to access the resources, such as tenants, namespaces, and topics.
</Note>

You can follow the instructions to [create an API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#using-api-keys-to-connect-to-your-cluster) for the service account you choose to use.

### Step 3: Connect to your cluster

For a complete example of how to connect to a cluster using the Pulsar Go client, see [Go client examples](https://github.com/streamnative/cloud-manager/tree/master/ui/src/data/code/clients/golang).

#### Create a Go consumer to consume messages

You can create and configure a Go consumer to consume messages using Token authentication as follows. For more information about the placeholders in the code sample, see [parameters for Token authentication](/cloud/build/pulsar-clients/cloud-connect-go#parameters-for-token-authentication).

```go theme={null}
package main

import (
	"context"
	"fmt"
	"github.com/apache/pulsar-client-go/pulsar"
	"log"
)

func main() {
    client, err := pulsar.NewClient(pulsar.ClientOptions{
        URL:               "${brokerServiceURL}",
        Authentication:    pulsar.NewAuthenticationToken("${apikey}"),
    })

    if err != nil {
        log.Fatalf("Could not instantiate Pulsar client: %v", err)
    }

    defer client.Close()

    consumer, err := client.Subscribe(pulsar.ConsumerOptions{
        Topic:            "persistent://${tenant}/${namespace}/${topic}",
        SubscriptionName: "${subscription}",
        SubscriptionInitialPosition: pulsar.SubscriptionPositionEarliest,
    })

    if err != nil {
        log.Fatal(err)
    }

    defer consumer.Close()

    for i := 0; i < 10; i++ {
        msg, err := consumer.Receive(context.Background())
        if err != nil {
            log.Fatal(err)
        }

        fmt.Printf("Received message msgId: %v -- content: '%s'\n",
            msg.ID(), string(msg.Payload()))

        consumer.Ack(msg)
    }

    if err := consumer.Unsubscribe(); err != nil {
        log.Fatal(err)
    }
}
```

#### Create a Go producer to produce messages

You can create and configure a Go producer to produce messages using Token authentication as follows. For more information about the placeholders in the code sample, see [parameters for Token authentication](/cloud/build/pulsar-clients/cloud-connect-go#parameters-for-token-authentication).

```go theme={null}
package main

import (
	"context"
	"fmt"
	"github.com/apache/pulsar-client-go/pulsar"
	"log"
)

func main() {
    client, err := pulsar.NewClient(pulsar.ClientOptions{
        URL:               "${brokerServiceURL}",
        Authentication:    pulsar.NewAuthenticationToken("${apikey}"),
    })

    if err != nil {
        log.Fatalf("Could not instantiate Pulsar client: %v", err)
    }

    defer client.Close()

    producer, err := client.CreateProducer(pulsar.ProducerOptions{
        Topic: "persistent://${tenant}/${namespace}/${topic}",
    })

    if err != nil {
        log.Fatal(err)
    }

    defer producer.Close()

    for i := 0; i < 10; i++ {
        if msgId, err := producer.Send(context.Background(), &pulsar.ProducerMessage{
            Payload: []byte(fmt.Sprintf("hello-%d", i)),
        }); err != nil {
            log.Fatal(err)
        } else {
            fmt.Printf("Published message: %v \n", msgId)
        }
    }
}
```

#### Parameters for Token authentication

* `${brokerServiceURL}`: the broker service URL of your StreamNative cluster.
* `${apikey}`: an API key of your service account.
* `${tenant}/${namespace}/${topic}`: the full name of the topic for message production & consumption. It is a combination of the tenant name, the namespace name and the topic name.
* `${subscription}`: the name of the subscription that will determine how messages are delivered.

## Connect to your cluster using OAuth2 authentication

<span />

To connect to a StreamNative cluster through OAuth2 authentication, follow these steps.

### Step 1: Get the broker service URL of your cluster

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

### Step 2: Get the OAuth2 credential file of your service account

To get an OAuth2 credential file of a service account through the StreamNative Console, follow these steps.

1. On the left navigation pane, click **Service Accounts**.

2. In the row of the service account you want to use, in the **Key File** column, click the **Download** icon to download the OAuth2 credential file to your local directory.

   The OAuth2 credential file should be something like this:

   ```json theme={null}
   {
     "type": "SN_SERVICE_ACCOUNT",
     "client_id": "CLIENT_ID",
     "client_secret": "CLIENT_SECRET",
     "client_email": "test@auth.streamnative.cloud",
     "issuer_url": "https://auth.streamnative.cloud"
   }
   ```

### Step 3: Connect to your cluster

For a complete example of how to connect to a cluster using the Pulsar Go client, see [Go client examples](https://github.com/streamnative/cloud-manager/tree/master/ui/src/data/code/clients/golang).

#### Create a Go consumer to consume messages

You can create and configure a Go consumer to consume messages using the OAuth2 credential file as follows. For more information about the placeholders in the code sample, see [parameters for OAuth2 authentication](/cloud/build/pulsar-clients/cloud-connect-go#parameters-for-oauth2-authentication).

```go theme={null}
package main

import (
	"context"
	"fmt"
	"github.com/apache/pulsar-client-go/pulsar"
	"log"
)

func main() {
    oauth := pulsar.NewAuthenticationOAuth2(map[string]string{
        "type":       "client_credentials",
        "issuerUrl":  "https://auth.streamnative.cloud/",
        "audience":   "urn:sn:pulsar:${orgName}:${instanceName}",
        "privateKey": "file:///YOUR-KEY-FILE-PATH",  // Absolute path of your downloaded key file
    })

    client, err := pulsar.NewClient(pulsar.ClientOptions{
        URL:               "${brokerServiceURL}",
        Authentication:    oauth,
    })

    if err != nil {
        log.Fatalf("Could not instantiate Pulsar client: %v", err)
    }

    defer client.Close()

    consumer, err := client.Subscribe(pulsar.ConsumerOptions{
        Topic:            "persistent://${tenant}/${namespace}/${topic}",
        SubscriptionName: "${subscription}",
        SubscriptionInitialPosition: pulsar.SubscriptionPositionEarliest,
    })

    if err != nil {
        log.Fatal(err)
    }

    defer consumer.Close()

    for i := 0; i < 10; i++ {
        msg, err := consumer.Receive(context.Background())
        if err != nil {
            log.Fatal(err)
        }

        fmt.Printf("Received message msgId: %v -- content: '%s'\n",
            msg.ID(), string(msg.Payload()))

        consumer.Ack(msg)
    }

    if err := consumer.Unsubscribe(); err != nil {
        log.Fatal(err)
    }
}
```

#### Create a Go producer to produce messages

You can create and configure a Go producer to produce messages using the OAuth2 credential file as follows. For more information about the placeholders in the code sample, see [parameters for OAuth2 authentication](/cloud/build/pulsar-clients/cloud-connect-go#parameters-for-oauth2-authentication).

```go theme={null}
package main

import (
	"context"
	"fmt"
	"github.com/apache/pulsar-client-go/pulsar"
	"log"
)

func main() {
    oauth := pulsar.NewAuthenticationOAuth2(map[string]string{
        "type":       "client_credentials",
        "issuerUrl":  "https://auth.streamnative.cloud/",
        "audience":   "urn:sn:pulsar:${orgName}:${instanceName}",
        "privateKey": "file:///YOUR-KEY-FILE-PATH",  // Absolute path of your downloaded key file
    })

    client, err := pulsar.NewClient(pulsar.ClientOptions{
        URL:               "${brokerServiceURL}",
        Authentication:    oauth,
    })

    if err != nil {
        log.Fatalf("Could not instantiate Pulsar client: %v", err)
    }

    defer client.Close()

    producer, err := client.CreateProducer(pulsar.ProducerOptions{
        Topic: "persistent://${tenant}/${namespace}/${topic}",
    })

    if err != nil {
        log.Fatal(err)
    }

    defer producer.Close()

    for i := 0; i < 10; i++ {
        if msgId, err := producer.Send(context.Background(), &pulsar.ProducerMessage{
            Payload: []byte(fmt.Sprintf("hello-%d", i)),
        }); err != nil {
            log.Fatal(err)
        } else {
            fmt.Printf("Published message: %v \n", msgId)
        }
    }
}
```

#### Parameters for OAuth2 authentication

* `privateKey`: your downloaded OAuth2 credential. This parameter supports the following two pattern formats:
  * `file:///path/to/file`: the path to your downloaded OAuth2 credential file.
  * `data:application/json;base64,<base64-encoded value>`: the credential file content encoded into Base64 format.
* `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
  * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
  * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
* `${brokerServiceURL}`: the broker service URL of your StreamNative cluster.
* `${tenant}/${namespace}/${topic}`: the full name of the topic for message production & consumption. It is a combination of the tenant name, the namespace name and the topic name.
* `${subscription}`: the name of the subscription that will determine how messages are delivered.


# Connect to your cluster using the Pulsar Java client
Source: https://docs.streamnative.io/cloud/build/pulsar-clients/cloud-connect-java



This document describes how to connect to a cluster using a Java client, and use the Java producer and consumer to produce and consume messages to and from a topic. The Java client supports connecting to a StreamNative cluster using either [OAuth2](#use-oauth2) or [API Keys](#use-apikeys) authentication.

<Note title="Note">
  This document assumes that you have created a StreamNative cluster and a service account, and have granted the service account `produce` and `consume` permissions to the namespace for the target topic.
</Note>

## Prerequisites

* Java 1.8 or higher version
* Pulsar client 2.6.1 or higher version

## Connect to your cluster using API keys

<span />

To connect a StreamNative cluster using [API keys](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview), follow these steps.

### Step 1: Get the broker service URL of your cluster

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

### Step 2: Create an API key of your service account

<Note title="Note">
  Before using an API key, verify that the service account is authorized to access the resources, such as tenants, namespaces, and topics.
</Note>

You can follow the instructions to [create an API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#using-api-keys-to-connect-to-your-cluster) for the service account you choose to use.

### Step 3: Connect to your cluster

For a complete example of how to connect to a cluster using the Pulsar Java client, see [Java client examples](https://github.com/streamnative/cloud-manager/tree/master/ui/src/data/code/clients/java).

#### Create a Java consumer to consume messages

You can create and configure a Java consumer to consume messages using API keys as follows. For more information about the placeholders in the code sample, see [parameters for Token authentication](/cloud/build/pulsar-clients/cloud-connect-java#parameters-for-token-authentication).

```java theme={null}
import java.net.URL;
import org.apache.pulsar.client.api.*;

public class SNConsumer {

    public static void main(String[] args) throws Exception
    {

        PulsarClient client = PulsarClient.builder()
            .serviceUrl("${brokerServiceURL}")
            .authentication(
                AuthenticationFactory.token("${apikey}")
            )
            .build();

        Consumer consumer = client.newConsumer()
                .topic("persistent://${tenant}/${namespace}/${topic}")
                .subscriptionName("${subscription}")
                .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
                .subscribe();

        for (int i = 0; i < 10; i++) {
            Message<byte[]> msg = consumer.receive();
            consumer.acknowledge(msg);
            System.out.println("Receive message " + new String(msg.getData()));
        }

        consumer.close();
        client.close();

    }

}
```

#### Create a Java producer to produce messages

You can create and configure a Java consumer to consume messages using API keys as follows. For more information about the placeholders in the code sample, see [parameters for Token authentication](/cloud/build/pulsar-clients/cloud-connect-java#parameters-for-token-authentication).

```java theme={null}
import java.net.URL;
import org.apache.pulsar.client.api.*;

public class SNProducer {

    public static void main(String[] args) throws Exception
    {

        PulsarClient client = PulsarClient.builder()
            .serviceUrl("${brokerServiceURL}")
            .authentication(
                AuthenticationFactory.token("${apikey}")
            )
            .build();

        Producer<byte[]> producer = client.newProducer()
            .topic("persistent://${tenant}/${namespace}/${topic}")
            .create();

        for (int i = 0; i < 10; i++) {
            String message = "my-message-" + i;
            MessageId msgID = producer.send(message.getBytes());
            System.out.println("Publish " + "my-message-" + i
                               + " and message ID " + msgID);
        }

        producer.close();
        client.close();

    }

}
```

#### Parameters for Token authentication

* `${brokerServiceURL}`: the broker service URL of your StreamNative cluster.
* `${apikey}`: an API key of your service account.
* `${tenant}/${namespace}/${topic}`: the full name of the topic for message production & consumption. It is a combination of the tenant name, the namespace name and the topic name.
* `${subscription}`: the name of the subscription that will determine how messages are delivered.

## Connect to your cluster using OAuth2 authentication

<span />

To connect a StreamNative cluster using OAuth2 authentication, follow these steps.

### Step 1: Get the broker service URL of your cluster

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

### Step 2: Get the OAuth2 credential file of your service account

To get an OAuth2 credential file of a service account through the StreamNative Console, follow these steps.

1. On the left navigation pane, click **Service Accounts**.

2. In the row of the service account you want to use, in the **Key File** column, click the **Download** icon to download the OAuth2 credential file to your local directory.

   The OAuth2 credential file should be something like this:

   ```json theme={null}
   {
     "type": "SN_SERVICE_ACCOUNT",
     "client_id": "CLIENT_ID",
     "client_secret": "CLIENT_SECRET",
     "client_email": "test@auth.streamnative.cloud",
     "issuer_url": "https://auth.streamnative.cloud"
   }
   ```

### Step 3: Connect to your cluster

For a complete example of how to connect to a cluster using the Pulsar Java client, see [Java client examples](https://github.com/streamnative/cloud-manager/tree/master/ui/src/data/code/clients/java).

#### Create a Java consumer to consume messages

You can create and configure a Java consumer to consume messages using the OAuth2 credential file as follows. For more information about the placeholders in the code sample, see [parameters for OAuth2 authentication](/cloud/build/pulsar-clients/cloud-connect-java#parameters-for-oauth2-authentication).

```java theme={null}
import java.net.URL;
import org.apache.pulsar.client.api.*;
import org.apache.pulsar.client.impl.auth.oauth2.AuthenticationFactoryOAuth2;

public class SNConsumer {

  public static void main(String[] args) throws Exception
  {
      String issuerUrl = "https://auth.streamnative.cloud/";
      String credentialsUrl = "{{ file://YOUR-KEY-FILE-PATH }}";
      String audience = "urn:sn:pulsar:${orgName}:${instanceName}";

      PulsarClient client = PulsarClient.builder()
          .serviceUrl("${brokerServiceURL}")
          .authentication(
                AuthenticationFactoryOAuth2.clientCredentials(new URL(issuerUrl),
                                                              new URL(credentialsUrl),
                                                              audience))
          .build();

      Consumer consumer = client.newConsumer()
              .topic("persistent://${tenant}/${namespace}/${topic}")
              .subscriptionName("${subscription}")
              .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
              .subscribe();

      for (int i = 0; i < 10; i++) {
          Message<byte[]> msg = consumer.receive();
          consumer.acknowledge(msg);
          System.out.println("Receive message " + new String(msg.getData()));
      }

      consumer.close();
      client.close();

  }
}
```

#### Create a Java producer to produce messages

You can create and configure a Java producer to produce messages using the OAuth2 credential file as follows. For more information about the placeholders in the code sample, see [parameters for OAuth2 authentication](/cloud/build/pulsar-clients/cloud-connect-java#parameters-for-oauth2-authentication).

```java theme={null}
import java.net.URL;
import org.apache.pulsar.client.api.*;
import org.apache.pulsar.client.impl.auth.oauth2.AuthenticationFactoryOAuth2;

public class SNProducer {

  public static void main(String[] args) throws Exception
  {

      String issuerUrl = "https://auth.streamnative.cloud/";
      String credentialsUrl = "file:///YOUR-KEY-FILE-PATH"; // Absolute path of your downloaded key file
      String audience = "urn:sn:pulsar:${orgName}:${instanceName}";

      PulsarClient client = PulsarClient.builder()
          .serviceUrl("${brokerServiceURL}")
          .authentication(
                AuthenticationFactoryOAuth2.clientCredentials(new URL(issuerUrl),
                                                              new URL(credentialsUrl),
                                                              audience))
          .build();

      Producer<byte[]> producer = client.newProducer()
          .topic("persistent://${tenant}/${namespace}/${topic}")
          .create();

      for (int i = 0; i < 10; i++) {
          String message = "my-message-" + i;
          MessageId msgID = producer.send(message.getBytes());
          System.out.println("Publish " + "my-message-" + i
                             + " and message ID " + msgID);
      }

      producer.close();
      client.close();
  }
}
```

#### Parameters for OAuth2 authentication

* `credentialsUrl`: your downloaded OAuth2 credential. This parameter supports the following two pattern formats:
  * `file:///path/to/file`: the path to your downloaded OAuth2 credential file.
  * `data:application/json;base64,<base64-encoded value>`: the credential file content encoded into Base64 format.
* `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
  * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
  * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
* `${brokerServiceUrl}`: the broker service URL of your StreamNative cluster.
* `${tenant}/${namespace}/${topic}`: the full name of the topic for message production & consumption. It is a combination of the tenant name, the namespace name and the topic name.
* `${subscription}`: the name of the subscription that will determine how messages are delivered.


# Connect to your cluster using the Pulsar Node.js client
Source: https://docs.streamnative.io/cloud/build/pulsar-clients/cloud-connect-nodejs



This document describes how to connect to a cluster using a Node.js client, and use the Node.js producer and consumer to produce and consume messages to and from a topic. The Node.js client supports connecting to a StreamNative cluster using either [OAuth2](#use-oauth2) or [API Keys](#use-apikeys) authentication.

<Note title="Note">
  This document assumes that you have created a StreamNative cluster and a service account, and have granted the service account `produce` and `consume` permissions to the namespace for the target topic.
</Note>

## Connect to your cluster using API keys

<span />

To connect a StreamNative cluster using [API keys](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview), follow these steps.

### Step 1: Get the broker service URL of your cluster

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

### Step 2: Create an API key of your service account

<Note title="Note">
  Before using an API key, verify that the service account is authorized to access the resources, such as tenants, namespaces, and topics.
</Note>

You can follow the instructions to [create an API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#using-api-keys-to-connect-to-your-cluster) for the service account you choose to use.

### Step 3: Connect to your cluster

For a complete example of how to connect to a cluster using the Pulsar Node.js client, see [Node.js client examples](https://github.com/streamnative/cloud-manager/tree/master/ui/src/data/code/clients/nodejs).

#### Create a Node.js consumer to consume messages

You can create and configure a Node.js consumer to consume messages using Token authentication as follows. For more information about the placeholders in the code sample, see [parameters for Token authentication](/cloud/build/pulsar-clients/cloud-connect-nodejs#parameters-for-token-authentication).

```javascript theme={null}
const Pulsar = require('pulsar-client')

const service_url = '${brokerServiceURL}'
const auth_params = '${apikey}'

;(async () => {
  const auth = new Pulsar.AuthenticationToken({
    token: auth_params,
  })

  const client = new Pulsar.Client({
    serviceUrl: service_url,
    authentication: auth,
    operationTimeoutSeconds: 30,
  })

  const consumer = await client.subscribe({
    topic: 'persistent://${tenant}/${namespace}/${topic}',
    subscription: '${subscription}',
    subscriptionInitialPosition: 'Earliest',
  })

  for (let i = 0; i < 10; i += 1) {
    const msg = await consumer.receive()
    console.log(msg.getData().toString())
    await consumer.acknowledge(msg)
  }

  await consumer.close()
  await client.close()
})()
```

#### Create a Node.js producer to produce messages

You can create and configure a Node.js producer to produce messages using Token authentication as follows. For more information about the placeholders in the code sample, see [parameters for Token authentication](/cloud/build/pulsar-clients/cloud-connect-nodejs#parameters-for-token-authentication).

```javascript theme={null}
const Pulsar = require('pulsar-client')

const service_url = '${brokerServiceURL}'
const auth_params = '${apikey}'

;(async () => {
  const auth = new Pulsar.AuthenticationToken({
    token: auth_params,
  })

  const client = new Pulsar.Client({
    serviceUrl: service_url,
    authentication: auth,
    operationTimeoutSeconds: 30,
  })

  const producer = await client.createProducer({
    topic: 'persistent://${tenant}/${namespace}/${topic}',
    sendTimeoutMs: 30000,
    batchingEnabled: true,
  })

  for (let i = 0; i < 10; i += 1) {
    const msg = `my-message-${i}`
    await producer.send({
      data: Buffer.from(msg),
    })
    console.log(`Sent message: ${msg}`)
  }
  await producer.flush()

  await producer.close()
  await client.close()
})()
```

#### Parameters for Token authentication

* `${brokerServiceURL}`: the broker service URL of your StreamNative cluster.
* `${apikey}`: an API key of your service account.
* `${tenant}/${namespace}/${topic}`: the full name of the topic for message production & consumption. It is a combination of the tenant name, the namespace name and the topic name.
* `${subscription}`: the name of the subscription that will determine how messages are delivered.

## Connect to your cluster using OAuth2 authentication

<span />

To connect a StreamNative cluster using OAuth2 authentication, follow these steps.

### Step 1: Get the broker service URL of your cluster

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

### Step 2: Get the OAuth2 credential file of your service account

To get an OAuth2 credential file of a service account through the StreamNative Console, follow these steps.

1. On the left navigation pane, click **Service Accounts**.

2. In the row of the service account you want to use, in the **Key File** column, click the **Download** icon to download the OAuth2 credential file to your local directory.

   The OAuth2 credential file should be something like this:

   ```json theme={null}
   {
     "type": "SN_SERVICE_ACCOUNT",
     "client_id": "CLIENT_ID",
     "client_secret": "CLIENT_SECRET",
     "client_email": "test@auth.streamnative.cloud",
     "issuer_url": "https://auth.streamnative.cloud"
   }
   ```

### Step 3: Connect to your cluster

For a complete example of how to connect to a cluster using the Pulsar Node.js client, see [Node.js client examples](https://github.com/streamnative/cloud-manager/tree/master/ui/src/data/code/clients/nodejs).

#### Create a Node.js consumer to consume messages

You can create and configure a Node.js consumer to consume messages using the OAuth2 credential file as follows. For more information about the placeholders in the code sample, see [parameters for OAuth2 authentication](/cloud/build/pulsar-clients/cloud-connect-nodejs#parameters-for-oauth2-authentication).

```javascript theme={null}
const Pulsar = require('pulsar-client')
const issuer_url = 'https://auth.streamnative.cloud/'
const private_key = '/YOUR-KEY-FILE-PATH' // Absolute file path of your downloaded key file without file:// prefix
const audience = 'urn:sn:pulsar:${orgName}:${instanceName}'
const service_url = '${brokerServiceURL}'

;(async () => {
  const params = {
    issuer_url: issuer_url,
    private_key: private_key,
    audience: audience,
  }

  const auth = new Pulsar.AuthenticationOauth2(params)

  const client = new Pulsar.Client({
    serviceUrl: service_url,
    authentication: auth,
    operationTimeoutSeconds: 30,
  })

  const consumer = await client.subscribe({
    topic: 'persistent://${tenant}/${namespace}/${topic}',
    subscription: '${subscription}',
    subscriptionInitialPosition: 'Earliest',
  })

  for (let i = 0; i < 10; i += 1) {
    const msg = await consumer.receive()
    console.log(msg.getData().toString())
    await consumer.acknowledge(msg)
  }

  await consumer.close()
  await client.close()
})()
```

#### Create a Node.js producer to produce messages

You can create and configure a Node.js producer to produce messages using the OAuth2 credential file as follows. For more information about the placeholders in the code sample, see [parameters for OAuth2 authentication](/cloud/build/pulsar-clients/cloud-connect-nodejs#parameters-for-oauth2-authentication).

```javascript theme={null}
const Pulsar = require('pulsar-client')
const issuer_url = 'https://auth.streamnative.cloud/'
const private_key = '/YOUR-KEY-FILE-PATH' // Absolute file path of your downloaded key file without file:// prefix
const audience = 'urn:sn:pulsar:${orgName}:${instanceName}'
const service_url = '${brokerServiceURL}'

;(async () => {
  const params = {
    issuer_url: issuer_url,
    private_key: private_key,
    audience: audience,
  }

  const auth = new Pulsar.AuthenticationOauth2(params)

  const client = new Pulsar.Client({
    serviceUrl: service_url,
    authentication: auth,
    operationTimeoutSeconds: 30,
  })

  const producer = await client.createProducer({
    topic: 'persistent://${tenant}/${namespace}/${topic}',
    sendTimeoutMs: 30000,
    batchingEnabled: true,
  })

  for (let i = 0; i < 10; i += 1) {
    const msg = `my-message-${i}`
    await producer.send({
      data: Buffer.from(msg),
    })
    console.log(`Sent message: ${msg}`)
  }
  await producer.flush()

  await producer.close()
  await client.close()
})()
```

#### Parameters for OAuth2 authentication

* `private_key`: your downloaded OAuth2 credential. This parameter supports the following two pattern formats:
  * `file:///path/to/file`: the path to your downloaded OAuth2 credential file.
  * `data:application/json;base64,<base64-encoded value>`: the credential file content encoded into Base64 format.
* `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
  * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
  * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
* `${brokerServiceURL}`: the broker service URL of your StreamNative cluster.
* `${tenant}/${namespace}/${topic}`: the full name of the topic for message production & consumption. It is a combination of the tenant name, the namespace name and the topic name.
* `${subscription}`: the name of the subscription that will determine how messages are delivered.


# Connect to your cluster using the Pulsar Python client
Source: https://docs.streamnative.io/cloud/build/pulsar-clients/cloud-connect-python



This example describes how to connect to a cluster using a Python client, and use the Python producer and consumer to produce and consume messages to and from a topic. The Python client supports connecting to a StreamNative cluster using either [OAuth2](#use-oauth2) or [API Keys](#use-apikeys) authentication.

<Note title="Note">
  This document assumes that you have created a StreamNative cluster and a service account, and have granted the service account `produce` and `consume` permissions to the namespace for the target topic.
</Note>

## Prerequisites

Install both Python 3.0 or higher versions and the Pulsar Python client.

You can use the following command to install the Pulsar Python client:

```shell theme={null}
python -m pip install pulsar-client
```

## Connect to your cluster using API keys

<span />

To connect a StreamNative cluster using [API keys](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview), follow these steps.

### Step 1: Get the broker service URL of your cluster

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

### Step 2: Create an API key of your service account

<Note title="Note">
  Before using an API key, verify that the service account is authorized to access the resources, such as tenants, namespaces, and topics.
</Note>

You can follow the instructions to [create an API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#using-api-keys-to-connect-to-your-cluster) for the service account you choose to use.

### Step 3: Connect to your cluster

For a complete example of how to connect to a cluster using the Pulsar Python client, see [Python client examples](https://github.com/streamnative/cloud-manager/tree/master/ui/src/data/code/clients/python).

#### Create a Python consumer to consume messages

You can create and configure a Python consumer to consume messages using Token authentication as follows. For more information about the placeholders in the code sample, see [parameters for Token authentication](/cloud/build/pulsar-clients/cloud-connect-python#parameters-for-token-authentication).

```python theme={null}
import pulsar

client = pulsar.Client("${brokerServiceURL}", authentication=pulsar.AuthenticationToken("${apikey}"))

consumer = client.subscribe("persistent://${tenant}/${namespace}/${topic}", "${subscription}",initial_position=pulsar.InitialPosition.Earliest)

for i in range(10):
    msg = consumer.receive()
    try:
        print("Received message '{}' id='{}'".format(msg.data().decode('utf-8'), msg.message_id()))
        consumer.acknowledge(msg)
    except Exception:
        consumer.negative_acknowledge(msg)

client.close()
```

#### Create a Python producer to produce messages

You can create and configure a Python producer to produce messages using Token authentication as follows. For more information about the placeholders in the code sample, see [parameters for Token authentication](/cloud/build/pulsar-clients/cloud-connect-python#parameters-for-token-authentication).

```python theme={null}
import pulsar

client = pulsar.Client("${brokerServiceURL}",
                authentication=pulsar.AuthenticationToken("${apikey}"))

producer = client.create_producer("persistent://${tenant}/${namespace}/${topic}")

for i in range(10):
    producer.send(('Hello-%d' % i).encode('utf-8'))

client.close()
```

#### Parameters for Token authentication

* `${brokerServiceURL}`: the broker service URL of your StreamNative cluster.
* `${apikey}`: an API key of your service account.
* `${tenant}/${namespace}/${topic}`: the full name of the topic for message production & consumption. It is a combination of the tenant name, the namespace name and the topic name.
* `${subscription}`: the name of the subscription that will determine how messages are delivered.

## Connect to your cluster using OAuth2 authentication

To connect a StreamNative cluster using OAuth2 authentication, follow these steps.

### Step 1: Get the broker service URL of your cluster

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

### Step 2: Get the OAuth2 credential file of your service account

To get an OAuth2 credential file of a service account through the StreamNative Console, follow these steps.

1. On the left navigation pane, click **Service Accounts**.

2. In the row of the service account you want to use, in the **Key File** column, click the **Download** icon to download the OAuth2 credential file to your local directory.

   The OAuth2 credential file should be something like this:

   ```json theme={null}
   {
     "type": "SN_SERVICE_ACCOUNT",
     "client_id": "CLIENT_ID",
     "client_secret": "CLIENT_SECRET",
     "client_email": "test@auth.streamnative.cloud",
     "issuer_url": "https://auth.streamnative.cloud"
   }
   ```

### Step 3: Connect to your cluster

For a complete example of how to connect to a cluster through the Pulsar Python client, see [Python client examples](https://github.com/streamnative/cloud-manager/tree/master/ui/src/data/code/clients/python).

#### Create a Python consumer to consume messages

You can create and configure a Python consumer to consume messages using the OAuth2 credential file as follows. For more information about the placeholders in the code sample, see [parameters for OAuth2 authentication](/cloud/build/pulsar-clients/cloud-connect-python#parameters-for-oauth2-authentication).

```python theme={null}
import pulsar

params = '''
{
   "issuer_url": "https://auth.streamnative.cloud/",
   "private_key": "/YOUR-KEY-FILE-PATH", // Absolute path of your downloaded key file. Dont forget to remove this comment on the code
   "audience": "urn:sn:pulsar:${orgName}:${instanceName}"
}
'''

client = pulsar.Client("${brokerServiceURL}", authentication=pulsar.AuthenticationOauth2(params))

consumer = client.subscribe("persistent://${tenant}/${namespace}/${topic}", "${subscription}",initial_position=pulsar.InitialPosition.Earliest)

for i in range(10):
   msg = consumer.receive()
   try:
      print("Received message '{}' id='{}'".format(msg.data().decode('utf-8'), msg.message_id()))
      consumer.acknowledge(msg)
   except Exception:
      consumer.negative_acknowledge(msg)

client.close()
```

#### Create a Python producer to produce messages

You can create and configure a Python producer to produce messages using the OAuth2 credential file as follows. For more information about the placeholders in the code sample, see [parameters for OAuth2 authentication](/cloud/build/pulsar-clients/cloud-connect-python#parameters-for-oauth2-authentication).

```python theme={null}
import pulsar

params = '''
{
   "issuer_url": "https://auth.streamnative.cloud/",
   "private_key": "/YOUR-KEY-FILE-PATH", // Absolute path of your downloaded key file
   "audience": "urn:sn:pulsar:${orgName}:${instanceName}"
}
'''

client = pulsar.Client("${brokerServiceURL}", authentication=pulsar.AuthenticationOauth2(params))

producer = client.create_producer("persistent://${tenant}/${namespace}/${topic}")

for i in range(10):
   producer.send(('Hello-%d' % i).encode('utf-8'))

client.close()
```

#### Parameters for OAuth2 authentication

* `private_key`: your downloaded OAuth2 credential. This parameter supports the following two pattern formats:
  * `file:///path/to/file`: the path to your downloaded OAuth2 credential file.
  * `data:application/json;base64,<base64-encoded value>`: the credential file content encoded into Base64 format.
* `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
  * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
  * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
* `${brokerServiceURL}`: the broker service URL of your StreamNative cluster.
* `${tenant}/${namespace}/${topic}`: the full name of the topic for message production & consumption. It is a combination of the tenant name, the namespace name and the topic name.
* `${subscription}`: the name of the subscription that will determine how messages are delivered.


# Connect to your cluster using the Pulsar Rust client
Source: https://docs.streamnative.io/cloud/build/pulsar-clients/cloud-connect-rust



This document describes how to connect to a cluster using a Rust client, and use the Rust producer and consumer to produce and consume messages to and from a topic. The Rust client supports connecting to a StreamNative cluster using either [OAuth2](#use-oauth2) or [API Keys](#use-apikeys) authentication.

<Note title="Note">
  This document assumes that you have created a StreamNative cluster and a service account, and have granted the service account `produce` and `consume` permissions to the namespace for the target topic.
</Note>

## Prerequisites

See [the minimum supported versions required for the underlying libraries](https://github.com/streamnative/pulsar-rs#getting-started) for more details.

## Connect to your cluster using API keys

<span />

To connect a StreamNative cluster using [API keys](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview), follow these steps.

### Step 1: Get the broker service URL of your cluster

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

### Step 2: Create an API key of your service account

<Note title="Note">
  Before using an API key, verify that the service account is authorized to access the resources, such as tenants, namespaces, and topics.
</Note>

You can follow the instructions to [create an API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#using-api-keys-to-connect-to-your-cluster) for the service account you choose to use.

### Step 3: Connect to your cluster

For a complete example of how to connect to a cluster through the Rust client, see [Rust client examples](https://github.com/streamnative/cloud-manager/tree/master/ui/src/data/code/clients/rust).

#### Create a Rust consumer to consume messages

You can create and configure a Rust consumer to consume messages using Token authentication as follows. For more information about the placeholders in the code sample, see [parameters for Token authentication](/cloud/build/pulsar-clients/cloud-connect-rust#parameters-for-token-authentication).

```rust theme={null}
use futures::TryStreamExt;
use pulsar::{Authentication, Consumer, ConsumerOptions, Pulsar, SubType, TokioExecutor};
use pulsar::consumer::InitialPosition;

#[tokio::main]
async fn main() -> Result<(), pulsar::Error> {
    env_logger::init();

    let addr = "${brokerServiceURL}".to_string();
    let mut builder = Pulsar::builder(addr, TokioExecutor);
    let token = "${apikey}".to_string();
    builder = builder.with_auth(Authentication {
        name: "token".to_string(),
        data: token.into_bytes(),
    });

    let pulsar: Pulsar<_> = builder.build().await?;
    let mut consumer: Consumer<String, _> = pulsar
        .consumer()
        .with_topic("persistent://${tenant}/${namespace}/${topic}")
        .with_subscription_type(SubType::Exclusive)
        .with_subscription("${subscription}")
        .with_options(ConsumerOptions::default()
            .with_initial_position(InitialPosition::Earliest))
        .build()
        .await?;

    let mut counter = 0usize;
    while let Some(msg) = consumer.try_next().await? {
        consumer.ack(&msg).await?;
        let payload = match msg.deserialize() {
            Ok(payload) => payload,
            Err(e) => {
                println!("could not deserialize message: {:?}", e);
                break;
            }
        };

        counter += 1;
        println!("Received message '{:?}' id='{:?}'", payload, msg.message_id());

        if counter > 10 {
            consumer.close().await.expect("Unable to close consumer");
            break;
        }
    }

    Ok(())
}
```

#### Create a Rust producer to produce messages

You can create and configure a Rust producer to produce messages using Token authentication as follows. For more information about the placeholders in the code sample, see [parameters for Token authentication](/cloud/build/pulsar-clients/cloud-connect-rust#parameters-for-token-authentication).

```rust theme={null}
use pulsar::{Authentication, Pulsar, TokioExecutor};

#[tokio::main]
async fn main() -> Result<(), pulsar::Error> {
    env_logger::init();

    let addr = "${brokerServiceURL}".to_string();
    let mut builder = Pulsar::builder(addr, TokioExecutor);
    let token = "${apikey}".to_string();
    builder = builder.with_auth(Authentication {
        name: "token".to_string(),
        data: token.into_bytes(),
    });

    let pulsar: Pulsar<_> = builder.build().await?;
    let mut producer = pulsar
        .producer()
        .with_topic("persistent://${tenant}/${namespace}/${topic}")
        .build()
        .await?;

    let mut counter = 0usize;
    loop {
        producer
            .send(format!("Hello-{}", counter))
            .await?
            .await
            .unwrap();

        counter += 1;
        println!("{counter} messages");

        if counter > 10 {
            producer.close().await.expect("Unable to close connection");
            break;
        }
    }

    Ok(())
}
```

#### Parameters for Token authentication

* `${brokerServiceURL}`: the broker service URL of your StreamNative cluster.
* `${apikey}`: an API key of your service account.
* `${tenant}/${namespace}/${topic}`: the full name of the topic for message production & consumption. It is a combination of the tenant name, the namespace name and the topic name.
* `${subscription}`: the name of the subscription that will determine how messages are delivered.

## Connect to your cluster using OAuth2 authentication

<span />

To connect a StreamNative cluster using OAuth2 authentication, follow these steps.

### Step 1: Get the broker service URL of your cluster

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

### Step 2: Get the OAuth2 credential file of your service account

To get an OAuth2 credential file of a service account through the StreamNative Console, follow these steps.

1. On the left navigation pane, click **Service Accounts**.

2. In the row of the service account you want to use, in the **Key File** column, click the **Download** icon to download the OAuth2 credential file to your local directory.

   The OAuth2 credential file should be something like this:

   ```json theme={null}
   {
     "type": "SN_SERVICE_ACCOUNT",
     "client_id": "CLIENT_ID",
     "client_secret": "CLIENT_SECRET",
     "client_email": "test@auth.streamnative.cloud",
     "issuer_url": "https://auth.streamnative.cloud"
   }
   ```

### Step 3: Connect to your cluster

For a complete example of how to connect to a cluster using the Rust client, see [Rust client examples](https://github.com/streamnative/cloud-manager/tree/master/ui/src/data/code/clients/rust).

#### Create a Rust consumer to consume messages

You can create and configure a Rust consumer to consume messages using the OAuth2 credential file as follows. For more information about the placeholders in the code sample, see [parameters for OAuth2 authentication](/cloud/build/pulsar-clients/cloud-connect-rust#parameters-for-oauth2-authentication).

```rust theme={null}
use futures::TryStreamExt;
use pulsar::{Consumer, ConsumerOptions, Pulsar, SubType, TokioExecutor};
use pulsar::authentication::oauth2::{OAuth2Authentication, OAuth2Params};
use pulsar::consumer::InitialPosition;

#[tokio::main]
async fn main() -> Result<(), pulsar::Error> {
    env_logger::init();

    let addr = "${brokerServiceURL}".to_string();
    let mut builder = Pulsar::builder(addr, TokioExecutor);
    builder = builder.with_auth_provider(OAuth2Authentication::client_credentials(OAuth2Params {
        issuer_url: "https://auth.streamnative.cloud/".to_string(),
        credentials_url: "file:///YOUR-KEY-FILE-PATH".to_string(), // Absolute path of your downloaded key file
        audience: Some("urn:sn:pulsar:${orgName}:${instanceName}".to_string()),
        scope: None,
    }));

    let pulsar: Pulsar<_> = builder.build().await?;
    let mut consumer: Consumer<String, _> = pulsar
        .consumer()
        .with_topic("persistent://${tenant}/${namespace}/${topic}")
        .with_subscription_type(SubType::Exclusive)
        .with_subscription("${subscription}")
        .with_options(ConsumerOptions::default()
            .with_initial_position(InitialPosition::Earliest))
        .build()
        .await?;

    let mut counter = 0usize;
    while let Some(msg) = consumer.try_next().await? {
        consumer.ack(&msg).await?;
        let payload = match msg.deserialize() {
            Ok(payload) => payload,
            Err(e) => {
                println!("could not deserialize message: {:?}", e);
                break;
            }
        };

        counter += 1;
        println!("Received message '{:?}' id='{:?}'", payload, msg.message_id());

        if counter > 10 {
            consumer.close().await.expect("Unable to close consumer");
            break;
        }
    }

    Ok(())
}
```

#### Create a Rust producer to produce messages

You can create and configure a Rust producer to produce messages using the OAuth2 credential file as follows. For more information about the placeholders in the code sample, see [parameters for OAuth2 authentication](/cloud/build/pulsar-clients/cloud-connect-rust#parameters-for-oauth2-authentication).

```rust theme={null}
use pulsar::{Pulsar, TokioExecutor};
use pulsar::authentication::oauth2::{OAuth2Authentication, OAuth2Params};

#[tokio::main]
async fn main() -> Result<(), pulsar::Error> {
    env_logger::init();

    let addr = "${brokerServiceURL}".to_string();
    let mut builder = Pulsar::builder(addr, TokioExecutor);
    builder = builder.with_auth_provider(OAuth2Authentication::client_credentials(OAuth2Params {
        issuer_url: "https://auth.streamnative.cloud/".to_string(),
        credentials_url: "file:///YOUR-KEY-FILE-PATH".to_string(), // Absolute path of your downloaded key file
        audience: Some("urn:sn:pulsar:${orgName}:${instanceName}".to_string()),
        scope: None,
    }));

    let pulsar: Pulsar<_> = builder.build().await?;
    let mut producer = pulsar
        .producer()
        .with_topic("persistent://${tenant}/${namespace}/${topic}")
        .build()
        .await?;

    let mut counter = 0usize;
    loop {
        producer
            .send(format!("Hello-{}", counter))
            .await?
            .await
            .unwrap();

        counter += 1;
        println!("{counter} messages");

        if counter > 10 {
            producer.close().await.expect("Unable to close connection");
            break;
        }
    }

    Ok(())
}
```

#### Parameters for OAuth2 authentication

* `${brokerServiceURL}`: the broker service URL of your StreamNative cluster.
* `credentials_url`: your downloaded OAuth2 credential. This parameter supports the following two pattern formats:
  * `file:///path/to/file`: the path to your downloaded OAuth2 credential file.
  * `data:application/json;base64,<base64-encoded value>`: the credential file content encoded into Base64 format.
* `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
  * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
  * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).
* `${tenant}/${namespace}/${topic}`: the full name of the topic for message production & consumption. It is a combination of the tenant name, the namespace name and the topic name.
* `${subscription}`: the name of the subscription that will determine how messages are delivered.


# Connect to your cluster in Spring applications
Source: https://docs.streamnative.io/cloud/build/pulsar-clients/cloud-connect-spring



This document describes how to connect to a cluster in Spring applications, and use the producer and consumer to produce and consume messages to and from a topic. You can use either [OAuth2](#use-oauth2) or [API Keys](#use-apikeys) authentication in Spring applications.

<Note title="Note">
  This document assumes that you have created a StreamNative cluster and a service account, and have granted the service account `produce` and `consume` permissions to the namespace for the target topic.
</Note>

## Prerequisites

See the [minimum supported versions required for the underlying libraries](https://docs.spring.io/spring-pulsar/docs/current/reference/html/#_minimum_supported_versions) for more details.

## Connect to your cluster using API keys

<span />

To connect a StreamNative cluster using [API keys](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview), follow these steps.

### Step 1: Get the broker service URL of your cluster

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

### Step 2: Create an API key of your service account

<Note title="Note">
  Before using an API key, verify that the service account is authorized to access the resources, such as tenants, namespaces, and topics.
</Note>

You can follow the instructions to [create an API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#using-api-keys-to-connect-to-your-cluster) for the service account you choose to use.

### Step 3: Connect to your cluster

#### Configure a YAML file

Set the following configurations in the code of your Spring applications.

```yaml theme={null}
spring:
  pulsar:
    client:
      service-url: ${brokerServiceURL}
      auth-plugin-class-name: org.apache.pulsar.client.impl.auth.AuthenticationToken
      authentication:
        token: ${apikey}
```

* `${brokerServiceURL}l`: the broker service URL of your StreamNative cluster.
* `${apikey}`: an API key of your service account.

#### Consume messages

You can consume messages using Token authentication in your Spring application as follows.

```java theme={null}
@SpringBootApplication
public class PulsarBootHelloWorld {

    public static void main(String[] args) {
        SpringApplication.run(PulsarBootHelloWorld.class, args);
    }

    @PulsarListener(subscriptionName = "${subscription}", topics = "persistent://${tenant}/${namespace}/${topic}")
    void listen(String message) {
        System.out.println("Message Received: " + message);
    }

}
```

#### Produce messages

You can produce messages using Token authentication in your Spring application as follows.

```java theme={null}
@SpringBootApplication
public class PulsarBootHelloWorld {

    public static void main(String[] args) {
        SpringApplication.run(PulsarBootHelloWorld.class, args);
    }

    @Bean
    ApplicationRunner runner(PulsarTemplate<String> pulsarTemplate) {
        return (args) -> pulsarTemplate.send("persistent://${tenant}/${namespace}/${topic}", "Hello Pulsar World!");
    }

}
```

For a complete example of how to connect to a cluster in a Spring application, see [Spring client examples](https://github.com/streamnative/cloud-manager/tree/master/ui/src/data/code/clients/spring).

## Connect to your cluster using OAuth2 authentication

<span />

To connect a StreamNative cluster using OAuth2 authentication, follow these steps.

### Step 1: Get the broker service URL of your cluster

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

### Step 2: Get the OAuth2 credential file of your service account

To get an OAuth2 credential file of a service account through the StreamNative Console, follow these steps.

1. On the left navigation pane, click **Service Accounts**.

2. In the row of the service account you want to use, in the **Key File** column, click the **Download** icon to download the OAuth2 credential file to your local directory.

   The OAuth2 credential file should be something like this:

   ```json theme={null}
   {
     "type": "SN_SERVICE_ACCOUNT",
     "client_id": "CLIENT_ID",
     "client_secret": "CLIENT_SECRET",
     "client_email": "test@auth.streamnative.cloud",
     "issuer_url": "https://auth.streamnative.cloud"
   }
   ```

### Step 3: Connect to your cluster

#### Configure a YAML file

Set the following configurations in the code of your Spring applications.

```yaml theme={null}
spring:
  pulsar:
    client:
      service-url: ${brokerServiceURL}
      auth-plugin-class-name: org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2
      authentication:
        issuer-url: 'https://auth.streamnative.cloud/'
        private-key: '/YOUR-KEY-FILE-PATH' # TODO Absolute file path of your downloaded key file
        audience: 'urn:sn:pulsar:${orgName}:${instanceName}'
```

* `service-url`: the broker service URL of your StreamNative cluster.
* `private-key`: your downloaded OAuth2 credential. This parameter supports the following two pattern formats:
  * `file:///path/to/file`: the path to your downloaded OAuth2 credential file.
  * `data:application/json;base64,<base64-encoded value>`: the credential file content encoded into Base64 format.
* `audience`: the [Uniform Resource Name (URN)](/cloud/references/glossary#urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.
  * `${orgName}`: the name of your [organization](/cloud/references/glossary#organization).
  * `${instanceName}`: the name of your [instance](/cloud/references/glossary#instance).

#### Consume messages

You can consume messages using OAuth2 authentication in your Spring application as follows.

```java theme={null}
import org.springframework.pulsar.annotation.PulsarListener;

@SpringBootApplication
public class PulsarBootHelloWorld {

    public static void main(String[] args) {
        SpringApplication.run(PulsarBootHelloWorld.class, args);
    }

    @PulsarListener(subscriptionName = "${subscription}", topics = "persistent://${tenant}/${namespace}/${topic}")
    void listen(String message) {
        System.out.println("Message Received: " + message);
    }

}
```

#### Produce messages

You can produce messages using OAuth2 authentication in your Spring application as follows.

```java theme={null}
import org.springframework.pulsar.core.PulsarTemplate;

@SpringBootApplication
public class PulsarBootHelloWorld {

    public static void main(String[] args) {
        SpringApplication.run(PulsarBootHelloWorld.class, args);
    }

    @Bean
    ApplicationRunner runner(PulsarTemplate<String> pulsarTemplate) {
        return (args) -> pulsarTemplate.send("persistent://${tenant}/${namespace}/${topic}", "Hello Pulsar World!");
    }

}
```

For a complete example of how to connect to a cluster in your Spring application, see [Spring client examples](https://github.com/streamnative/cloud-manager/tree/master/ui/src/data/code/clients/spring).


# Connect to your cluster using the Pulsar WebSocket API
Source: https://docs.streamnative.io/cloud/build/pulsar-clients/cloud-connect-websocket



This document describes how to connect to a cluster through a WebSocket API, and use the WebSocket producer and consumer to produce and consume messages to and from a topic. The WebSocket API supports connecting to a StreamNative cluster using [API Keys](#use-apikeys) authentication.

To use the WebSocket API to connect to a StreamNative cluster, you need to enable the WebSocket service in advance. For details, see [enable WebSocket service](/cloud/clusters/manage-clusters/cluster#create-a-cluster).

<Note title="Note">
  This document assumes that you have created a StreamNative cluster and a service account, and have granted the service account `produce` and `consume` permissions to the namespace for the target topic.
</Note>

## Prerequisites

Use the `pip install websocket-client` command to install all dependencies. For details, see the [Pulsar WebSocket documentation](http://pulsar.apache.org/docs/client-libraries-websocket/#python).

## Connect to your cluster using API keys

<span />

To connect a StreamNative cluster using [API keys](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview), follow these steps.

### Step 1: Get the broker service URL of your cluster

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

### Step 2: Create an API key of your service account

<Note title="Note">
  Before using an API key, verify that the service account is authorized to access the resources, such as tenants, namespaces, and topics.
</Note>

You can follow the instructions to [create an API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#using-api-keys-to-connect-to-your-cluster) for the service account you choose to use.

### Step 3: Connect to your cluster

#### Create a consumer to consume messages

```python theme={null}
import websocket, base64, json

# TOPIC = 'ws://CLUSTER_HOST:9090/ws/v2/consumer/persistent/public/default/test/sub'
TOPIC = 'wss://CLUSTER_HOST:443/ws/v2/consumer/persistent/public/default/test/sub'
token = "${apikey}"
header = ["Authorization:Bearer " + token]
ws = websocket.create_connection(TOPIC, header=header)
while True:
  msg = json.loads(ws.recv())
  if not msg: break
  print("Received: {} - payload: {}".format(msg, base64.b64decode(msg['payload'])))
  # Acknowledge successful processing
  ws.send(json.dumps({'messageId' : msg['messageId']}))

ws.close()
```

* `${apikey}`: an API key of your service account.

#### Create a producer to produce messages

```python theme={null}
import websocket, base64, json

# TOPIC = 'ws://CLUSTER_HOST:9090/ws/v2/producer/persistent/public/default/test'
TOPIC = 'wss://CLUSTER_HOST:443/ws/v2/producer/persistent/public/default/test'

token = "${apikey}"

header = ["Authorization:Bearer " + token]

ws = websocket.create_connection(TOPIC, header=header)

# Send one message as JSON
ws.send(json.dumps({
'payload' : base64.b64encode('Hello World'),
'properties': {
  'key1' : 'VALUE1',
  'key2' : 'VALUE2'
},
'context' : 5
}))

response =  json.loads(ws.recv())
if response['result'] == 'ok':
  print('Message published successfully')
else:
  print('Failed to publish message:', response)
ws.close()
```

* `${apikey}`: an API key of your service account.

<Note title="Note">
  - Replace the `CLUSTER_HOST` with the domain name of the cluster. To get the domain name of the target cluster, click **Manage** > **Cluster** on the StreamNative Cloud Console.
  - In SN cloud you should use the port `443`.
</Note>


# Message Rest API QuickStart
Source: https://docs.streamnative.io/cloud/build/pulsar-clients/connect-restapi



## Set up Message Rest API

Before using Rest API, you need to complete the following setup steps.

### Step 1: Create a service account

<Note title="Note">
  Currently, you can't edit a service account. If you need a service account to have Super Admin access, make sure to enable it when creating the service account. By default, service accounts do not have Super Admin enabled.
</Note>

To create a service account, follow these steps.

1. On the left navigation pane, click **Service Accounts**.

2. Click **Create Service Account**.

3. (Optional) Select **Super Admin** to grant the service account with Super admin access to a namespace or tenant.

4. Enter a name for the service account, and then click **Confirm**.

### Step 2: Create an API key of your service account

<Note title="Note">
  Before using an API key, verify that the service account is authorized to access the resources, such as tenants, namespaces, and topics.
</Note>

You can follow the instructions to [create an API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#using-api-keys-to-connect-to-your-cluster) for the service account you choose to use.

### Step 3: Grant service account permissions

<Note title="Note">
  If you use a Super Admin service account, you can skip this step because a Super Admin service account has the required permissions already.
</Note>

1. On the left navigation pane, in the **Admin** section, click **Tenants/Namespaces**.

2. Select the **Public** tenant, then select the **Default** namespace under the tenant.

3. Select the **POLICY** tab.

4. In the **Authorization** area, click **ADD ROLE**, and select the name of the service account you just created in the previous section.

5. In the **Authorization** area, on the drop-down menu below the service name you just added, select the **consume** and **produce** roles. The roles are added to your service account.

### Step 4: Get the HTTP Service URL of your StreamNative cluster

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

## Work with Message Rest API

<Note title="Note">
  Rest API does not validate the data schema and directly stores the accepted binary data into a topic. You should ensure the correctness of the data schema to ensure compatibility with the Pulsar client in other languages.
</Note>

After you have completed the configuration steps above, you can use Rest API to produce and consume messages. Rest API sends binary data. You can use the cURL tool to encode a string (for example `Hi Pulsar`) into UTF-8 format bytes. To send bytes in another format, you need to specify an appropriate file. For more information, see the [cURL documentation](https://everything.curl.dev/http/post/binary).

1. Create a subscription on the topic.

   ```shell script theme={null}
   curl -X PUT https://<endpoint>:<port>/admin/v2/persistent/public/default/rest-topic/subscription/rest-sub \
     --header 'Authorization: Bearer <API Key>' \
     --header 'Accept: application/json' \
     --header 'Content-Type: application/json'
   ```

   You should see the following output:

   ```shell script theme={null}
   # No content response
   ```

2. Produce messages to the topic.

   ```shell script theme={null}
   curl -X POST https://<endpoint>:<port>/admin/rest/topics/v1/persistent/public/default/rest-topic/message \
     --header 'Authorization: Bearer <API Key>' \
     --header 'Accept: application/json' \
     --header 'Content-Type: application/octet-stream' \
     --data-binary 'Hi, Pulsar'
   ```

   You should see the following output:

   ```shell script theme={null}
   # Message id in string format
   10:0:-1:0
   ```

3. Consume messages from the topic.

   ```shell script theme={null}
   curl -X POST https://<endpoint>:<port>/admin/rest/topics/v1/persistent/public/default/rest-topic/rest-sub/message \
     --header 'Authorization: Bearer <API Key>' \
     --header 'Accept: application/octet-stream' \
     --header 'Content-Type: application/json' -v
   ```

   You should see the following output:

   ```shell script theme={null}
   # Headers
   X-Pulsar-Message-Id: CAoQACAAMAE=
   X-Pulsar-Message-String-Id: 10:0:-1:0
   X-Pulsar-Sequence-Id: 0

   # Body
   Hi, Pulsar
   ```

4. Acknowledge messages.

   ```shell script theme={null}
   curl -X PUT https://<endpoint>:<port>/admin/rest/topics/v1/persistent/public/default/rest-topic/rest-sub/message \
     --header 'Authorization: Bearer <API Key>' \
     --header 'Accept: application/json' \
     --header 'Content-Type: application/json' \
     --data-raw 'CAoQACAAMAE='
   ```

   You should see the following output:

   ```shell script theme={null}
   # No content response
   ```

5. Negative acknowledge messages.

   ```shell script theme={null}
   curl -X PUT https://<endpoint>:<port>/admin/rest/topics/v2/persistent/public/default/rest-topic/rest-sub/message \
     --header 'Authorization: Bearer <API Key>' \
     --header 'Accept: application/json' \
     --header 'Content-Type: application/json' \
     --data-raw '{"encodedMessageId":"CAoQACAAMAE=","negativeAck":"true"}'
   ```

   You should see the following output:

   ```shell script theme={null}
   # No content response
   ```

## Message Rest API reference

For further information, see [Message Rest API reference](/api-references/rest-messaging-api/rest-messaging-api).


# Build applications using Pulsar clients
Source: https://docs.streamnative.io/cloud/build/pulsar-clients/qs-connect



You can use the [Pulsar clients](#pulsar-clients) to connect to your StreamNative cluster.

If you are familiar with command lines and environments, you can also use the [CLI tools](/tools/cli/pulsarctl/pulsarctl-overview) or [Message Rest API](/cloud/build/pulsar-clients/connect-restapi).

## Jumpstart for beginners

To help you get started with setting up client libraries and tools after provisioning your cluster, StreamNative Cloud Console provides a step-by-step wizard to walk you through the basic setup and configuration process, such as installing client libraries, downloading key files, selecting properties, generating sample codes to run, and so on.

<img alt="gif of client setup process through wizard" />

To get started with the client setup wizard, follow these steps.

1. On the left navigation pane of StreamNative Cloud Console, in the **Admin** section, click **Pulsar Clients**.

2. Follow the wizard to generate the sample code you need for connecting to your StreamNative cluster.

With a copy-and-paste, you can run the given sample code to produce and consume messages. For a quick walkthrough, check out this [video](https://www.youtube.com/watch?v=zwlXFIWdhQo) (starting from 1'57'').

<Note title="Note">
  You can also use the [Kafka clients](/cloud/build/kafka-clients/kafka-on-cloud#kafka-clients) to connect to your StreamNative cluster.
</Note>

## Pulsar Clients

<span />

* [C# client](/cloud/build/pulsar-clients/cloud-connect-dotnet)
* [C++ client](/cloud/build/pulsar-clients/cloud-connect-cpp)
* [Go client](/cloud/build/pulsar-clients/cloud-connect-go)
* [Java client](/cloud/build/pulsar-clients/cloud-connect-java)
* [Node.js client](/cloud/build/pulsar-clients/cloud-connect-nodejs)
* [Python client](/cloud/build/pulsar-clients/cloud-connect-python)
* [Rust client](/cloud/build/pulsar-clients/cloud-connect-rust)
* [Spring client](/cloud/build/pulsar-clients/cloud-connect-spring)
* [WebSocket API](/cloud/build/pulsar-clients/cloud-connect-websocket)
* [Message REST API](/cloud/build/pulsar-clients/connect-restapi)
* [Transaction API](/cloud/build/pulsar-clients/transactions-overview)


# Pulsar Transactions
Source: https://docs.streamnative.io/cloud/build/pulsar-clients/transactions-overview



[Pulsar transactions](https://pulsar.apache.org/docs/txn-what/) enables event-streaming applications to consume, process, and produce messages in one atomic operation. That means:

* Atomic writes across multiple topic partitions
* Atomic acknowledgments across multiple topic partitions
* All the operations made within one transaction either all succeed or all fail
* Consumers are **ONLY** allowed to read committed messages

<Note title="Note">
  - Currently, Transactions is only available for Java clients.
  - This document assumes that you have created a [StreamNative cluster](/cloud/clusters/manage-clusters/cluster#create-a-cluster-through-streamnative-cloud-console) and a [service account](/cloud/security/authentication/service-accounts/service-accounts), and have [granted the service account the consume permission](/cloud/manage-data-streams/topic#manage-topics) to the `persistent://pulsar/system/transaction_coordinator_assign` topic.
</Note>

## Quick start

This section describes how to use the Transaction API to send and receive messages.

### Prerequisites

* Java 1.8 or higher version
* Pulsar cluster 2.9.1 or higher version

### Get the service URL of your StreamNative cluster

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

### Get the OAuth2 credential file of your service account

To get an OAuth2 credential file of a service account through the StreamNative Console, follow these steps.

1. On the left navigation pane, click **Service Accounts**.

2. In the row of the service account you want to use, in the **Key File** column, click the **Download** icon to download the OAuth2 credential file to your local directory.

   The OAuth2 credential file should be something like this:

   ```json theme={null}
   {
     "type": "SN_SERVICE_ACCOUNT",
     "client_id": "CLIENT_ID",
     "client_secret": "CLIENT_SECRET",
     "client_email": "test@auth.streamnative.cloud",
     "issuer_url": "https://auth.streamnative.cloud"
   }
   ```

### Enable Transactions on your StreamNative cluster

1. [Log in to StreamNative Cloud Console](/cloud/get-started/quickstart-kafka#step-1:-log-in-to-streamnative-cloud-console).
2. On the left navigation pane, in the **Admin** area, click **Pulsar Clusters**.
3. Click **Edit Cluster**.
4. Select the **Advanced** tab. In the **Features** area, enable the **Transaction** option.

   <img alt="Transactions on Cloud" />

### Use Transactions on your application

This example enables you to perform the following operations.

1. Create a Pulsar client and enable Transactions.
2. Create three producers to produce messages with a transaction to one input topic (`input-topic`) and two output topics (`output-topic-1` and `output-topic-2`).
3. Create three consumers to consume messages with a transaction from one input topic (input-topic) and two output topics (`output-topic-1` and `output-topic-2`).
4. Commit the transaction after the consumers consume messages successfully.

```java theme={null}
package io.streamnative.examples.transaction;

import com.beust.jcommander.JCommander;
import org.apache.pulsar.client.api.Consumer;
import org.apache.pulsar.client.api.Message;
import org.apache.pulsar.client.api.Producer;
import org.apache.pulsar.client.api.ProducerBuilder;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.PulsarClientException;
import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.client.api.transaction.Transaction;
import org.apache.pulsar.client.impl.auth.oauth2.AuthenticationFactoryOAuth2;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.net.URL;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;

public class TransactionSyncExample {
    private static final Logger log = LoggerFactory.getLogger(TransactionSyncExample.class);
    public static void main(String[] args) throws Exception {
        JCommanderPulsar jct = new JCommanderPulsar();
        JCommander jCommander = new JCommander(jct, args);
        if (jct.help) {
            jCommander.usage();
            return;
        }

        String inputTopic = "persistent://public/default/input-topic";
        String outputTopicOne = "persistent://public/default/output-topic-1";
        String outputTopicTwo = "persistent://public/default/output-topic-2";

        PulsarClient client = PulsarClient.builder()
                // Create a Pulsar client and enable Transactions.
                .enableTransaction(true)
                .serviceUrl(jct.serviceUrl)
                .authentication(
                        AuthenticationFactoryOAuth2.clientCredentials(new URL(jct.issuerUrl), new URL(jct.credentialsUrl), jct.audience))
                .build();

        // Create three producers to produce messages to input and output topics.
        ProducerBuilder<String> producerBuilder = client.newProducer(Schema.STRING);
        Producer<String> inputProducer = producerBuilder.topic(inputTopic)
                .sendTimeout(0, TimeUnit.SECONDS).create();
        Producer<String> outputProducerOne = producerBuilder.topic(outputTopicOne)
                .sendTimeout(0, TimeUnit.SECONDS).create();
        Producer<String> outputProducerTwo = producerBuilder.topic(outputTopicTwo)
                .sendTimeout(0, TimeUnit.SECONDS).create();
        // Create three consumers to consume messages from input and output topics.
        Consumer<String> inputConsumer = client.newConsumer(Schema.STRING)
                .subscriptionName("your-subscription-name").topic(inputTopic).subscribe();
        Consumer<String> outputConsumerOne = client.newConsumer(Schema.STRING)
                .subscriptionName("your-subscription-name").topic(outputTopicOne).subscribe();
        Consumer<String> outputConsumerTwo = client.newConsumer(Schema.STRING)
                .subscriptionName("your-subscription-name").topic(outputTopicTwo).subscribe();

        int count = 2;
        // Produce messages to topics.
        for (int i = 0; i < count; i++) {
            inputProducer.send("Hello Pulsar! count : " + i);
        }

        // consume messages and produce to output topics with transaction
        for (int i = 0; i < count; i++) {

            // The consumer successfully receives messages. Then, create a transaction.
            Message<String> message = inputConsumer.receive();
            Transaction txn = null;
            try {
                txn = client.newTransaction()
                        .withTransactionTimeout(10, TimeUnit.SECONDS).build().get();
                // process the message here...

                // The producers produce messages to output topics with the transaction
                outputProducerOne.newMessage(txn).value("Hello Pulsar! outputTopicOne count : " + i).send();
                outputProducerTwo.newMessage(txn).value("Hello Pulsar! outputTopicTwo count : " + i).send();

                // The consumers acknowledge the input message with the transaction
                inputConsumer.acknowledgeAsync(message.getMessageId(), txn).get();
                // commit the transaction
                txn.commit();
            } catch (ExecutionException e) {
                if (!(e.getCause() instanceof PulsarClientException.TransactionConflictException)) {
                    // if not TransactionConflictException,
                    // we should redeliver or negativeAcknowledge this message
                    // if you don't redeliver or negativeAcknowledge, the message will not receive again
                    inputConsumer.negativeAcknowledge(message);
                }

                // if a transaction has been created, should abort this transaction
                if (txn != null) {
                    txn.abort();
                }
            }
        }

        // consume messages from output topics and print them
        for (int i = 0; i < count; i++) {
            Message<String> message =  outputConsumerOne.receive();
            System.out.println("Receive transaction message: " + message.getValue());
        }

        for (int i = 0; i < count; i++) {
            Message<String> message =  outputConsumerTwo.receive();
            System.out.println("Receive transaction message: " + message.getValue());
        }
    }
}
```

* `serviceUrl`: the broker service URL of your StreamNative cluster.
* `issuerUrl`: the URL of your OAuth2 authentication provider. You can get the value from your downloaded OAuth2 credential file.
* `credentialsUrl`: the path to your downloaded OAuth2 credential file. The `privateKey` parameter supports the following pattern formats:
  * `file:///path/to/file`
  * `file:/path/to/file`
  * `data:application/json;base64,<base64-encoded value>`
* `audience`: the `audience` parameter is the [Uniform Resource Name (URN)](/cloud/references/glossary#urn), which is a combination of the `urn:sn:pulsar`, the organization name, and the Pulsar instance name, in this format `urn:sn:pulsar:<org_name>:<instance_name>`.

You should see the following output:

```bash theme={null}
Receive transaction message: Hello Pulsar! count : 1
Receive transaction message: Hello Pulsar! count : 2
Receive transaction message: Hello Pulsar! count : 1
Receive transaction message: Hello Pulsar! count : 2
```

For a complete example about how to connect to a StreamNative cluster through the Transaction API, see [Transaction API examples](https://github.com/streamnative/examples/tree/master/cloud/transaction/java/src/main/java/io/streamnative/examples/transaction).


# Data Governance on StreamNative Cloud
Source: https://docs.streamnative.io/cloud/governance/governance-overview



Data governance is a critical aspect of any data platform. StreamNative Cloud provides a comprehensive set of data governance capabilities to help you govern your data streams effectively. This section provides an overview of the data governance capabilities available on StreamNative Cloud.

## Schema Registry

Schema Registry allows teams to define and enforce universal data standards that enable scalable data compatibility while reducing operational complexity.

As a multi-protocol platform, StreamNative Cloud currently supports schema management for two different protocols: the built-in Pulsar schema registry and the Kafka Schema Registry, which is compatible with the open-source Confluent Schema Registry API. These two schema registries are not currently interoperable. When building your applications, ensure that producers and consumers use the same schema registry. Work is ongoing to make these two schema registries interoperable within StreamNative Cloud.

### Pulsar Schema Registry

The Pulsar schema registry is built into the brokers. You can use Pulsar CLI tools to manage your schemas. See [Pulsar Schema](https://pulsar.apache.org/docs/schema-overview/) for more information.

### Kafka Schema Registry

Kafka schema registry is introduced as part of the Kafka protocol support on StreamNative Cloud. Currently, it is compatible with the Confluent Schema Registry API.

See [Kafka Schema Registry](/cloud/governance/kafka-schemas/kafka-schema-registry) for more information.

## Related

To use Kafka schemas from Pulsar Java clients:

* [Use External JSON Schema with Pulsar clients](/cloud/governance/kafka-schemas/external-json-schema)
* [Use External Avro Schema with Pulsar clients](/cloud/governance/kafka-schemas/external-avro-schema)
* [Use External Protobuf Schema with Pulsar clients](/cloud/governance/kafka-schemas/external-protobuf-schema)


# Use External Avro Schema with Pulsar clients
Source: https://docs.streamnative.io/cloud/governance/kafka-schemas/external-avro-schema

Use Kafka Avro Schema and the Kafka Schema Registry from Pulsar Java clients with the kafka-schemas library.

External Avro Schema lets Pulsar Java clients produce and consume messages that use [Kafka Avro Schema](/cloud/governance/kafka-schemas/kafka-schema-registry) and the [Kafka Schema Registry](/cloud/governance/kafka-schemas/kafka-schema-registry) on StreamNative Cloud. Schemas are registered in and resolved from the Kafka Schema Registry, while your application uses the familiar Pulsar `Producer` and `Consumer` APIs.

Use External Avro Schema when you want to:

* Use Pulsar clients with Kafka Avro Schema and Schema Registry compatibility checks.
* Share Avro schemas between Kafka and Pulsar clients on the same topic.
* Work with Avro `SpecificRecord` classes generated from `.avsc` schema files.

The [`kafka-schemas`](https://github.com/streamnative/external-schemas) library provides a Pulsar `Schema` implementation backed by the Kafka Avro serializer. The same library also supports [External JSON Schema](/cloud/governance/kafka-schemas/external-json-schema) and [External Protobuf Schema](/cloud/governance/kafka-schemas/external-protobuf-schema).

## Prerequisites

* A StreamNative Pulsar cluster for message production and consumption.
* The [Kafka Schema Registry](/cloud/governance/kafka-schemas/kafka-schema-registry) service enabled on the cluster.
* A service account with `produce` and `consume` permissions on the target topic.
* RBAC permissions for the Kafka Schema Registry: assign the [`schema-writer`](/cloud/security/access/rbac/manage-rbac-roles#schema-writer) role to register schemas and the [`schema-reader`](/cloud/security/access/rbac/manage-rbac-roles#schema-reader) role to read schemas. See [Schema Registry RBAC roles](/cloud/security/access/rbac/manage-rbac-roles#schema-registry).
* Java 17 or higher.
* Pulsar Java client 4.1.0 or higher.

## Add the dependency

Add the following Maven dependencies to your project:

```xml theme={null}
<dependency>
  <groupId>org.apache.pulsar</groupId>
  <artifactId>pulsar-client</artifactId>
  <version>4.1.0</version>
  <exclusions>
      <exclusion>
          <groupId>javax.validation</groupId>
          <artifactId>validation-api</artifactId>
      </exclusion>
  </exclusions>
</dependency>
<dependency>
  <groupId>io.streamnative.schemas.external</groupId>
  <artifactId>kafka-schemas</artifactId>
  <version>1.0.0</version>
</dependency>
<dependency>
  <groupId>io.confluent</groupId>
  <artifactId>kafka-avro-serializer</artifactId>
  <version>8.0.0</version>
</dependency>
<dependency>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro</artifactId>
  <version>1.12.0</version>
</dependency>
```

The `pulsar-client` dependency provides the Pulsar `Producer`, `Consumer`, and `PulsarClient` APIs used in the examples below. The `kafka-avro-serializer` dependency is required at runtime because `kafka-schemas` declares it with `provided` scope. Declare `avro` explicitly so you control its version; `kafka-schemas` also pulls it in transitively.

### Add the Kafka Maven repository

`pulsar-client`, `kafka-schemas`, and `avro` are available from [Maven Central](https://repo1.maven.org/maven2/). You do not need to add a repository for those dependencies.

`kafka-avro-serializer` is not published to Maven Central. Add the following repository to your `pom.xml`:

```xml theme={null}
<repositories>
  <repository>
    <id>kafka</id>
    <url>https://packages.confluent.io/maven/</url>
  </repository>
</repositories>
```

<Note>
  If your organization already mirrors `kafka-avro-serializer` artifacts in an internal repository, configure that mirror instead of adding the public repository directly.
</Note>

## Define an Avro schema

External Avro Schema works with Avro `SpecificRecord` classes. Define a schema in an `.avsc` file and generate the Java class with the Avro Maven plugin.

### Step 1: Create a schema file

Create `src/main/avro/Player.avsc`:

```json theme={null}
{
  "namespace": "com.example.avro",
  "type": "record",
  "name": "Player",
  "doc": "A player's profile information.",
  "fields": [
    {
      "name": "name",
      "type": "string",
      "doc": "The player's full name."
    },
    {
      "name": "number",
      "type": ["null", "int"],
      "default": null,
      "doc": "The player's number (optional)."
    },
    {
      "name": "favorite_color",
      "type": ["null", "string"],
      "default": null,
      "doc": "The player's favorite color (optional)."
    }
  ]
}
```

### Step 2: Generate the SpecificRecord class

Add the Avro Maven plugin to your `pom.xml`:

```xml theme={null}
<plugin>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro-maven-plugin</artifactId>
  <version>1.12.0</version>
  <executions>
    <execution>
      <phase>generate-sources</phase>
      <goals>
        <goal>schema</goal>
      </goals>
      <configuration>
        <sourceDirectory>${project.basedir}/src/main/avro</sourceDirectory>
      </configuration>
    </execution>
  </executions>
</plugin>
```

Run `mvn generate-sources` to generate the `Player` class in the `com.example.avro` package.

## Configure Schema Registry authentication

`KafkaSchemaFactory` accepts the same Schema Registry configuration properties as `KafkaAvroSerializerConfig`. The [`external-schemas`](https://github.com/streamnative/external-schemas) examples authenticate to the Schema Registry with Basic authentication. Define the helper method as `private static` so you can call it from `main`.

Use your service account API key as the password. The username can be any non-empty string.

```java theme={null}
private static Map<String, Object> getSchemaRegistryConfigs(String schemaRegistryUrl, String apiKey) {
    Map<String, Object> configs = new HashMap<>();
    configs.put(KafkaAvroSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl);
    configs.put(KafkaAvroSerializerConfig.BASIC_AUTH_CREDENTIALS_SOURCE, "USER_INFO");
    configs.put(
            KafkaAvroSerializerConfig.USER_INFO_CONFIG,
            String.format("%s:%s", "public", apiKey));
    return configs;
}
```

For additional serializer options, see the `KafkaAvroSerializerConfig` class in the `kafka-avro-serializer` dependency.

## Produce and consume messages

Use `KafkaSchemaFactory` to create a Pulsar `Schema` backed by Kafka Avro Schema, then create a producer and consumer with the same schema instance.

```java theme={null}
import com.example.avro.Player;
import io.confluent.kafka.serializers.KafkaAvroSerializerConfig;
import io.streamnative.schemas.external.KafkaSchemaFactory;
import java.util.HashMap;
import java.util.Map;
import org.apache.pulsar.client.api.AuthenticationFactory;
import org.apache.pulsar.client.api.Consumer;
import org.apache.pulsar.client.api.Message;
import org.apache.pulsar.client.api.Producer;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.client.api.SubscriptionInitialPosition;

public class ExternalAvroSchemaExample {

    public static void main(String[] args) throws Exception {
        String serviceUrl = "<YOUR-PULSAR-SERVICE-URL>";
        String schemaRegistryUrl = "<YOUR-SCHEMA-REGISTRY-URL>";
        String apiKey = "<YOUR-API-KEY>";
        String topic = "persistent://public/default/players";

        KafkaSchemaFactory schemaFactory =
                new KafkaSchemaFactory(getSchemaRegistryConfigs(schemaRegistryUrl, apiKey));
        Schema<Player> schema = schemaFactory.avro(Player.class);

        PulsarClient client = PulsarClient.builder()
                .serviceUrl(serviceUrl)
                .authentication(AuthenticationFactory.token(apiKey))
                .build();

        Producer<Player> producer = client.newProducer(schema).topic(topic).create();
        Consumer<Player> consumer = client.newConsumer(schema)
                .topic(topic)
                .subscriptionName("my-subscription")
                .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
                .subscribe();

        for (int i = 0; i < 10; i++) {
            Player player = new Player();
            player.setName("name-" + i);
            player.setNumber(i);
            player.setFavoriteColor("color-" + i);
            producer.send(player);
        }

        for (int i = 0; i < 10; i++) {
            Message<Player> message = consumer.receive();
            consumer.acknowledge(message);
            Player player = message.getValue();
            System.out.println("name=>" + player.getName()
                    + ", number=>" + player.getNumber()
                    + ", favoriteColor=>" + player.getFavoriteColor());
        }

        consumer.close();
        producer.close();
        client.close();
    }

    private static Map<String, Object> getSchemaRegistryConfigs(String schemaRegistryUrl, String apiKey) {
        Map<String, Object> configs = new HashMap<>();
        configs.put(KafkaAvroSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl);
        configs.put(KafkaAvroSerializerConfig.BASIC_AUTH_CREDENTIALS_SOURCE, "USER_INFO");
        configs.put(
                KafkaAvroSerializerConfig.USER_INFO_CONFIG,
                String.format("%s:%s", "public", apiKey));
        return configs;
    }
}
```

When the producer sends the first message, the schema is automatically registered in the Kafka Schema Registry. The consumer resolves the schema from the registry when reading messages.

## Schema compatibility

External Avro Schema registers schemas with the Pulsar schema type `EXTERNAL`. A topic cannot mix `EXTERNAL` schemas with native Pulsar schemas such as `JSON`, `AVRO`, or `PROTOBUF` on the same topic.

For example, if a topic already uses Pulsar's built-in `Schema.AVRO(Player.class)`, creating a producer with External Avro Schema on the same topic fails with an incompatible schema error:

```
Incompatible schema: exists schema type AVRO, new schema type EXTERNAL
```

<Warning>
  Plan your schema strategy before publishing to a topic. Once a topic uses External Avro Schema, all producers and consumers on that topic must use the same External Avro Schema type.
</Warning>

Schema compatibility modes for Avro in the Kafka Schema Registry are described in [Configurable compatibility modes](/cloud/governance/kafka-schemas/kafka-schema-registry#configurable-compatibility-modes). StreamNative Cloud supports a subset of the Kafka Schema Registry REST API. See the [REST API](/cloud/governance/kafka-schemas/kafka-schema-registry#rest-api) section for supported operations.

## Related resources

<CardGroup>
  <Card title="External JSON Schema" icon="brackets-curly" href="/cloud/governance/kafka-schemas/external-json-schema">
    Use Kafka JSON Schema from Pulsar Java clients.
  </Card>

  <Card title="External Protobuf Schema" icon="code" href="/cloud/governance/kafka-schemas/external-protobuf-schema">
    Use Kafka Protobuf Schema from Pulsar Java clients.
  </Card>

  <Card title="Kafka Schema Registry" icon="database" href="/cloud/governance/kafka-schemas/kafka-schema-registry">
    Configure authentication, compatibility modes, and REST API access.
  </Card>

  <Card title="external-schemas repository" icon="github" href="https://github.com/streamnative/external-schemas">
    View source code, tests, and release notes for the kafka-schemas library.
  </Card>
</CardGroup>


# Use External JSON Schema with Pulsar clients
Source: https://docs.streamnative.io/cloud/governance/kafka-schemas/external-json-schema

Use Kafka JSON Schema and the Kafka Schema Registry from Pulsar Java clients with the kafka-schemas library.

External JSON Schema lets Pulsar Java clients produce and consume messages that use [Kafka JSON Schema](/cloud/governance/kafka-schemas/kafka-schema-registry) and the [Kafka Schema Registry](/cloud/governance/kafka-schemas/kafka-schema-registry) on StreamNative Cloud. Schemas are registered in and resolved from the Kafka Schema Registry, while your application uses the familiar Pulsar `Producer` and `Consumer` APIs.

Use External JSON Schema when you want to:

* Use Pulsar clients with Kafka JSON Schema and Schema Registry compatibility checks.
* Share JSON schemas between Kafka and Pulsar clients on the same topic.
* Build Key-Value messages where the key, value, or both use Kafka JSON Schema.

The [`kafka-schemas`](https://github.com/streamnative/external-schemas) library (previously published as `kafka-json-schema`) provides a Pulsar `Schema` implementation backed by the Kafka JSON Schema serializer. The same library also supports [External Avro Schema](/cloud/governance/kafka-schemas/external-avro-schema) and [External Protobuf Schema](/cloud/governance/kafka-schemas/external-protobuf-schema).

## Prerequisites

* A StreamNative Pulsar cluster for message production and consumption.
* The [Kafka Schema Registry](/cloud/governance/kafka-schemas/kafka-schema-registry) service enabled on the cluster.
* A service account with `produce` and `consume` permissions on the target topic.
* RBAC permissions for the Kafka Schema Registry: assign the [`schema-writer`](/cloud/security/access/rbac/manage-rbac-roles#schema-writer) role to register schemas and the [`schema-reader`](/cloud/security/access/rbac/manage-rbac-roles#schema-reader) role to read schemas. See [Schema Registry RBAC roles](/cloud/security/access/rbac/manage-rbac-roles#schema-registry).
* Java 17 or higher.
* Pulsar Java client 4.1.0 or higher.

## Add the dependency

Add the following Maven dependencies to your project:

```xml theme={null}
<dependency>
  <groupId>org.apache.pulsar</groupId>
  <artifactId>pulsar-client</artifactId>
  <version>4.1.0</version>
  <exclusions>
      <exclusion>
          <groupId>javax.validation</groupId>
          <artifactId>validation-api</artifactId>
      </exclusion>
  </exclusions>
</dependency>
<dependency>
  <groupId>io.streamnative.schemas.external</groupId>
  <artifactId>kafka-schemas</artifactId>
  <version>1.0.0</version>
</dependency>
<dependency>
  <groupId>io.confluent</groupId>
  <artifactId>kafka-json-schema-serializer</artifactId>
  <version>8.0.0</version>
</dependency>
```

The `pulsar-client` dependency provides the Pulsar `Producer`, `Consumer`, and `PulsarClient` APIs used in the examples below. The `kafka-json-schema-serializer` dependency is required at runtime because `kafka-schemas` declares it with `provided` scope.

### Add the Kafka Maven repository

`pulsar-client` and `kafka-schemas` are available from [Maven Central](https://repo1.maven.org/maven2/). You do not need to add a repository for those dependencies.

`kafka-json-schema-serializer` is not published to Maven Central. Add the following repository to your `pom.xml`:

```xml theme={null}
<repositories>
  <repository>
    <id>kafka</id>
    <url>https://packages.confluent.io/maven/</url>
  </repository>
</repositories>
```

<Note>
  If your organization already mirrors `kafka-json-schema-serializer` artifacts in an internal repository, configure that mirror instead of adding the public repository directly.
</Note>

## Configure Schema Registry authentication

`KafkaSchemaFactory` accepts the same Schema Registry configuration properties as `KafkaJsonSchemaSerializerConfig`. The [`external-schemas`](https://github.com/streamnative/external-schemas) examples authenticate to the Schema Registry with Basic authentication. Define the helper method as `private static` so you can call it from `main`.

Use your service account API key as the password. The username can be any non-empty string.

```java theme={null}
private static Map<String, Object> getSchemaRegistryConfigs(String schemaRegistryUrl, String apiKey) {
    Map<String, Object> configs = new HashMap<>();
    configs.put(KafkaJsonSchemaSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl);
    configs.put(KafkaJsonSchemaSerializerConfig.BASIC_AUTH_CREDENTIALS_SOURCE, "USER_INFO");
    configs.put(
            KafkaJsonSchemaSerializerConfig.USER_INFO_CONFIG,
            String.format("%s:%s", "public", apiKey));
    return configs;
}
```

For additional serializer options, see the `KafkaJsonSchemaSerializerConfig` class in the `kafka-json-schema-serializer` dependency.

## Produce and consume messages

The following example shows how to create a producer and consumer with External JSON Schema.

### Step 1: Define your message class

Define a POJO for your message payload. Lombok annotations are optional.

```java theme={null}
public class User {
    private String name;
    private Integer age;

    public User() {}

    public User(String name, Integer age) {
        this.name = name;
        this.age = age;
    }

    public String getName() { return name; }
    public void setName(String name) { this.name = name; }
    public Integer getAge() { return age; }
    public void setAge(Integer age) { this.age = age; }
}
```

### Step 2: Create a schema and connect to your cluster

Use `KafkaSchemaFactory` to create a Pulsar `Schema` backed by Kafka JSON Schema, then create a producer and consumer with the same schema instance.

```java theme={null}
import io.confluent.kafka.serializers.json.KafkaJsonSchemaSerializerConfig;
import io.streamnative.schemas.external.KafkaSchemaFactory;
import java.util.HashMap;
import java.util.Map;
import org.apache.pulsar.client.api.AuthenticationFactory;
import org.apache.pulsar.client.api.Consumer;
import org.apache.pulsar.client.api.Message;
import org.apache.pulsar.client.api.Producer;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.client.api.SubscriptionInitialPosition;

public class ExternalJsonSchemaExample {

    public static void main(String[] args) throws Exception {
        String serviceUrl = "<YOUR-PULSAR-SERVICE-URL>";
        String schemaRegistryUrl = "<YOUR-SCHEMA-REGISTRY-URL>";
        String apiKey = "<YOUR-API-KEY>";
        String topic = "persistent://public/default/users";

        KafkaSchemaFactory schemaFactory =
                new KafkaSchemaFactory(getSchemaRegistryConfigs(schemaRegistryUrl, apiKey));
        Schema<User> schema = schemaFactory.json(User.class);

        PulsarClient client = PulsarClient.builder()
                .serviceUrl(serviceUrl)
                .authentication(AuthenticationFactory.token(apiKey))
                .build();

        Producer<User> producer = client.newProducer(schema).topic(topic).create();
        Consumer<User> consumer = client.newConsumer(schema)
                .topic(topic)
                .subscriptionName("my-subscription")
                .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
                .subscribe();

        for (int i = 0; i < 10; i++) {
            producer.send(new User("name-" + i, 10 + i));
        }

        for (int i = 0; i < 10; i++) {
            Message<User> message = consumer.receive();
            consumer.acknowledge(message);
            System.out.println(message.getValue().getName());
        }

        consumer.close();
        producer.close();
        client.close();
    }

    private static Map<String, Object> getSchemaRegistryConfigs(String schemaRegistryUrl, String apiKey) {
        Map<String, Object> configs = new HashMap<>();
        configs.put(KafkaJsonSchemaSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl);
        configs.put(KafkaJsonSchemaSerializerConfig.BASIC_AUTH_CREDENTIALS_SOURCE, "USER_INFO");
        configs.put(
                KafkaJsonSchemaSerializerConfig.USER_INFO_CONFIG,
                String.format("%s:%s", "public", apiKey));
        return configs;
    }
}
```

When the producer sends the first message, the schema is automatically registered in the Kafka Schema Registry. The consumer resolves the schema from the registry when reading messages.

## Use Key-Value schemas

`KafkaSchemaFactory` also supports Key-Value messages. You can combine a native Pulsar schema for the key with External JSON Schema for the value, or use External JSON Schema for both key and value.

### Native Pulsar key with External JSON Schema value

Use a native Pulsar `Schema.STRING` key and an External JSON Schema value:

```java theme={null}
import org.apache.pulsar.common.schema.KeyValue;
import org.apache.pulsar.common.schema.KeyValueEncodingType;

Schema<KeyValue<String, User>> schema = schemaFactory.kv(
        Schema.STRING,
        schemaFactory.json(User.class),
        KeyValueEncodingType.INLINE);

Producer<KeyValue<String, User>> producer = client.newProducer(schema).topic(topic).create();
producer.send(new KeyValue<>("user-1", new User("Alice", 30)));
```

### External JSON Schema for both key and value

Use External JSON Schema for both the key and value:

```java theme={null}
public class UserKey {
    private Integer userId;
    private String name;
    // constructors and getters/setters omitted
}

Schema<KeyValue<UserKey, User>> schema = schemaFactory.kv(
        schemaFactory.json(UserKey.class),
        schemaFactory.json(User.class),
        KeyValueEncodingType.SEPARATED);
```

`KeyValueEncodingType` supports both `INLINE` and `SEPARATED` encoding, matching the behavior of Pulsar Key-Value schemas.

## Schema compatibility

External JSON Schema registers schemas with the Pulsar schema type `EXTERNAL`. A topic cannot mix `EXTERNAL` schemas with native Pulsar schemas such as `JSON`, `AVRO`, or `PROTOBUF` on the same topic.

For example, if a topic already uses Pulsar's built-in `Schema.JSON(User.class)`, creating a producer with External JSON Schema on the same topic fails with an incompatible schema error:

```
Incompatible schema: exists schema type JSON, new schema type EXTERNAL
```

<Warning>
  Plan your schema strategy before publishing to a topic. Once a topic uses External JSON Schema, all producers and consumers on that topic must use the same External JSON Schema type.
</Warning>

Schema compatibility modes for JSON Schema in the Kafka Schema Registry are described in [Configurable compatibility modes](/cloud/governance/kafka-schemas/kafka-schema-registry#configurable-compatibility-modes). StreamNative Cloud supports a subset of the Kafka Schema Registry REST API. See the [REST API](/cloud/governance/kafka-schemas/kafka-schema-registry#rest-api) section for supported operations.

## Related resources

<CardGroup>
  <Card title="External Avro Schema" icon="file-code" href="/cloud/governance/kafka-schemas/external-avro-schema">
    Use Kafka Avro Schema from Pulsar Java clients.
  </Card>

  <Card title="External Protobuf Schema" icon="code" href="/cloud/governance/kafka-schemas/external-protobuf-schema">
    Use Kafka Protobuf Schema from Pulsar Java clients.
  </Card>

  <Card title="Kafka Schema Registry" icon="database" href="/cloud/governance/kafka-schemas/kafka-schema-registry">
    Configure authentication, compatibility modes, and REST API access.
  </Card>

  <Card title="external-schemas repository" icon="github" href="https://github.com/streamnative/external-schemas">
    View source code, tests, and release notes for the kafka-schemas library.
  </Card>
</CardGroup>


# Use External Protobuf Schema with Pulsar clients
Source: https://docs.streamnative.io/cloud/governance/kafka-schemas/external-protobuf-schema

Use Kafka Protobuf Schema and the Kafka Schema Registry from Pulsar Java clients with the kafka-schemas library.

External Protobuf Schema lets Pulsar Java clients produce and consume messages that use [Kafka Protobuf Schema](/cloud/governance/kafka-schemas/kafka-schema-registry) and the [Kafka Schema Registry](/cloud/governance/kafka-schemas/kafka-schema-registry) on StreamNative Cloud. Schemas are registered in and resolved from the Kafka Schema Registry, while your application uses the familiar Pulsar `Producer` and `Consumer` APIs.

Use External Protobuf Schema when you want to:

* Use Pulsar clients with Kafka Protobuf Schema and Schema Registry compatibility checks.
* Share Protobuf schemas between Kafka and Pulsar clients on the same topic.
* Work with Protobuf message classes generated from `.proto` files.

The [`kafka-schemas`](https://github.com/streamnative/external-schemas) library provides a Pulsar `Schema` implementation backed by the Kafka Protobuf serializer. The same library also supports [External JSON Schema](/cloud/governance/kafka-schemas/external-json-schema) and [External Avro Schema](/cloud/governance/kafka-schemas/external-avro-schema).

## Prerequisites

* A StreamNative Pulsar cluster for message production and consumption.
* The [Kafka Schema Registry](/cloud/governance/kafka-schemas/kafka-schema-registry) service enabled on the cluster.
* A service account with `produce` and `consume` permissions on the target topic.
* RBAC permissions for the Kafka Schema Registry: assign the [`schema-writer`](/cloud/security/access/rbac/manage-rbac-roles#schema-writer) role to register schemas and the [`schema-reader`](/cloud/security/access/rbac/manage-rbac-roles#schema-reader) role to read schemas. See [Schema Registry RBAC roles](/cloud/security/access/rbac/manage-rbac-roles#schema-registry).
* Java 17 or higher.
* Pulsar Java client 4.1.0 or higher.

## Add the dependency

Add the following Maven dependencies to your project:

```xml theme={null}
<dependency>
  <groupId>org.apache.pulsar</groupId>
  <artifactId>pulsar-client</artifactId>
  <version>4.1.0</version>
  <exclusions>
      <exclusion>
          <groupId>javax.validation</groupId>
          <artifactId>validation-api</artifactId>
      </exclusion>
  </exclusions>
</dependency>
<dependency>
  <groupId>io.streamnative.schemas.external</groupId>
  <artifactId>kafka-schemas</artifactId>
  <version>1.0.0</version>
</dependency>
<dependency>
  <groupId>io.confluent</groupId>
  <artifactId>kafka-protobuf-serializer</artifactId>
  <version>8.0.0</version>
</dependency>
<dependency>
  <groupId>com.google.protobuf</groupId>
  <artifactId>protobuf-java</artifactId>
  <version>4.29.5</version>
</dependency>
```

The `pulsar-client` dependency provides the Pulsar `Producer`, `Consumer`, and `PulsarClient` APIs used in the examples below. Declare `kafka-protobuf-serializer` and `protobuf-java` explicitly so you control their versions. `kafka-schemas` also pulls them in transitively.

### Add the Kafka Maven repository

`pulsar-client`, `kafka-schemas`, and `protobuf-java` are available from [Maven Central](https://repo1.maven.org/maven2/). You do not need to add a repository for those dependencies.

`kafka-protobuf-serializer` is not published to Maven Central. Add the following repository to your `pom.xml`:

```xml theme={null}
<repositories>
  <repository>
    <id>kafka</id>
    <url>https://packages.confluent.io/maven/</url>
  </repository>
</repositories>
```

<Note>
  If your organization already mirrors `kafka-protobuf-serializer` artifacts in an internal repository, configure that mirror instead of adding the public repository directly.
</Note>

## Define a Protobuf schema

External Protobuf Schema works with Protobuf message classes generated from `.proto` files.

### Step 1: Create Protobuf definition files

Create `src/main/proto/other.proto`:

```protobuf theme={null}
syntax = "proto3";
package com.example.protobuf;

option java_multiple_files = true;
option java_package = "com.example.protobuf";

message OtherRecord {
  int32 other_id = 1;
}
```

Create `src/main/proto/myRecord.proto`:

```protobuf theme={null}
syntax = "proto3";
package com.example.protobuf;

option java_multiple_files = true;
option java_package = "com.example.protobuf";

import "other.proto";

message MyRecord {
  string f1 = 1;
  OtherRecord f2 = 2;
}
```

### Step 2: Generate the Protobuf classes

Add a Protobuf Maven plugin to your `pom.xml`. The following example uses the `protobuf-maven-plugin`:

```xml theme={null}
<plugin>
  <groupId>io.github.ascopes</groupId>
  <artifactId>protobuf-maven-plugin</artifactId>
  <version>3.10.1</version>
  <configuration>
    <protocVersion>4.29.5</protocVersion>
  </configuration>
  <executions>
    <execution>
      <goals>
        <goal>generate</goal>
      </goals>
    </execution>
  </executions>
</plugin>
```

Run `mvn generate-sources` to generate the `MyRecord` and `OtherRecord` classes in the `com.example.protobuf` package.

## Configure Schema Registry authentication

`KafkaSchemaFactory` accepts the same Schema Registry configuration properties as `KafkaProtobufSerializerConfig`. The [`external-schemas`](https://github.com/streamnative/external-schemas) examples authenticate to the Schema Registry with Basic authentication. Define the helper method as `private static` so you can call it from `main`.

Use your service account API key as the password. The username can be any non-empty string.

```java theme={null}
private static Map<String, Object> getSchemaRegistryConfigs(String schemaRegistryUrl, String apiKey) {
    Map<String, Object> configs = new HashMap<>();
    configs.put(KafkaProtobufSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl);
    configs.put(KafkaProtobufSerializerConfig.BASIC_AUTH_CREDENTIALS_SOURCE, "USER_INFO");
    configs.put(
            KafkaProtobufSerializerConfig.USER_INFO_CONFIG,
            String.format("%s:%s", "public", apiKey));
    return configs;
}
```

For additional serializer options, see the `KafkaProtobufSerializerConfig` class in the `kafka-protobuf-serializer` dependency.

## Produce and consume messages

Use `KafkaSchemaFactory` to create a Pulsar `Schema` backed by Kafka Protobuf Schema, then create a producer and consumer with the same schema instance.

```java theme={null}
import com.example.protobuf.MyRecord;
import com.example.protobuf.OtherRecord;
import io.confluent.kafka.serializers.protobuf.KafkaProtobufSerializerConfig;
import io.streamnative.schemas.external.KafkaSchemaFactory;
import java.util.HashMap;
import java.util.Map;
import org.apache.pulsar.client.api.AuthenticationFactory;
import org.apache.pulsar.client.api.Consumer;
import org.apache.pulsar.client.api.Message;
import org.apache.pulsar.client.api.Producer;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.client.api.SubscriptionInitialPosition;

public class ExternalProtobufSchemaExample {

    public static void main(String[] args) throws Exception {
        String serviceUrl = "<YOUR-PULSAR-SERVICE-URL>";
        String schemaRegistryUrl = "<YOUR-SCHEMA-REGISTRY-URL>";
        String apiKey = "<YOUR-API-KEY>";
        String topic = "persistent://public/default/protobuf-records";

        KafkaSchemaFactory schemaFactory =
                new KafkaSchemaFactory(getSchemaRegistryConfigs(schemaRegistryUrl, apiKey));
        Schema<MyRecord> schema = schemaFactory.protobuf(MyRecord.class);

        PulsarClient client = PulsarClient.builder()
                .serviceUrl(serviceUrl)
                .authentication(AuthenticationFactory.token(apiKey))
                .build();

        Producer<MyRecord> producer = client.newProducer(schema).topic(topic).create();
        Consumer<MyRecord> consumer = client.newConsumer(schema)
                .topic(topic)
                .subscriptionName("my-subscription")
                .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
                .subscribe();

        for (int i = 0; i < 10; i++) {
            MyRecord myRecord = MyRecord.newBuilder()
                    .setF1("name-" + i)
                    .setF2(OtherRecord.newBuilder().setOtherId(i).build())
                    .build();
            producer.send(myRecord);
        }

        for (int i = 0; i < 10; i++) {
            Message<MyRecord> message = consumer.receive();
            consumer.acknowledge(message);
            MyRecord myRecord = message.getValue();
            System.out.println("f1=>" + myRecord.getF1()
                    + ", f2.otherId=>" + myRecord.getF2().getOtherId());
        }

        consumer.close();
        producer.close();
        client.close();
    }

    private static Map<String, Object> getSchemaRegistryConfigs(String schemaRegistryUrl, String apiKey) {
        Map<String, Object> configs = new HashMap<>();
        configs.put(KafkaProtobufSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl);
        configs.put(KafkaProtobufSerializerConfig.BASIC_AUTH_CREDENTIALS_SOURCE, "USER_INFO");
        configs.put(
                KafkaProtobufSerializerConfig.USER_INFO_CONFIG,
                String.format("%s:%s", "public", apiKey));
        return configs;
    }
}
```

When the producer sends the first message, the schema is automatically registered in the Kafka Schema Registry. The consumer resolves the schema from the registry when reading messages.

## Schema compatibility

External Protobuf Schema registers schemas with the Pulsar schema type `EXTERNAL`. A topic cannot mix `EXTERNAL` schemas with native Pulsar schemas such as `JSON`, `AVRO`, or `PROTOBUF` on the same topic.

For example, if a topic already uses Pulsar's built-in Protobuf schema, creating a producer with External Protobuf Schema on the same topic fails with an incompatible schema error:

```
Incompatible schema: exists schema type PROTOBUF, new schema type EXTERNAL
```

<Warning>
  Plan your schema strategy before publishing to a topic. Once a topic uses External Protobuf Schema, all producers and consumers on that topic must use the same External Protobuf Schema type.
</Warning>

Schema compatibility modes for Protobuf in the Kafka Schema Registry are described in [Configurable compatibility modes](/cloud/governance/kafka-schemas/kafka-schema-registry#configurable-compatibility-modes). StreamNative Cloud supports a subset of the Kafka Schema Registry REST API. See the [REST API](/cloud/governance/kafka-schemas/kafka-schema-registry#rest-api) section for supported operations.

## Related resources

<CardGroup>
  <Card title="External JSON Schema" icon="brackets-curly" href="/cloud/governance/kafka-schemas/external-json-schema">
    Use Kafka JSON Schema from Pulsar Java clients.
  </Card>

  <Card title="External Avro Schema" icon="file-code" href="/cloud/governance/kafka-schemas/external-avro-schema">
    Use Kafka Avro Schema from Pulsar Java clients.
  </Card>

  <Card title="Kafka Schema Registry" icon="database" href="/cloud/governance/kafka-schemas/kafka-schema-registry">
    Configure authentication, compatibility modes, and REST API access.
  </Card>

  <Card title="external-schemas repository" icon="github" href="https://github.com/streamnative/external-schemas">
    View source code, tests, and release notes for the kafka-schemas library.
  </Card>
</CardGroup>


# Kafka Schema Registry
Source: https://docs.streamnative.io/cloud/governance/kafka-schemas/kafka-schema-registry



Kafka Schema Registry provides an interface for storing and managing schemas. Producers and consumers can register the schemas within the registry and retrieve them when necessary. Schemas are versioned, and the registry supports configurable compatibility modes between different schema versions. When a producer or consumer attempts to register a new schema version, the registry performs a compatibility check and returns an error if an incompatible change is detected. This mechanism ensures consistency and compatibility among all producers and consumers when schema changes occur.

## Access Schema Registry in Kafka clients

To access the Kafka Schema Registry, you must configure how to authenticate. There are two ways to configure authentication to the Schema Registry:

* OAuth2 authentication: only available for Kafka Java client
* Basic authentication: available for all Kafka clients

### OAuth2 authentication

First, import the following dependencies:

```xml theme={null}
    <dependency>
      <groupId>org.apache.kafka</groupId>
      <artifactId>kafka-clients</artifactId>
      <version>3.6.1</version>
    </dependency>
    <dependency>
      <groupId>io.streamnative.pulsar.handlers</groupId>
      <artifactId>oauth-client</artifactId>
      <version>3.2.2.6</version>
    </dependency>
    <dependency>
      <groupId>io.confluent</groupId>
      <artifactId>kafka-avro-serializer</artifactId>
      <version>7.5.0</version>
    </dependency>
```

Minimum required versions:

* `kafka-clients`: 3.4.0
* `oauth-client`: 3.1.0.4
* `kafka-avro-serializer`: 7.5.0

<Note title="Note">
  Before 3.2.2.6, `oauth-client` requires Java 17 or higher.
</Note>

Then, in addition to the existing properties, you need to configure more properties like:

```java theme={null}
// props is the Properties object that has already configures the OAuth2 authentication
// See https://docs.streamnative.io/docs/cloud-connect-kafka-java for the necessary configs
props.put(KafkaAvroSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl);
props.put(KafkaAvroSerializerConfig.BEARER_AUTH_CUSTOM_PROVIDER_CLASS, "io.streamnative.pulsar.handlers.kop.security.oauth.schema.OauthCredentialProvider");
props.put(KafkaAvroSerializerConfig.BEARER_AUTH_CREDENTIALS_SOURCE, "CUSTOM");
```

### Basic authentication

Unlike the OAuth2 authentication, Basic authentication does not require the `oauth-client` dependency or `kafka-clients` >= 3.4.0.

The username can be any non-empty string. The password should be the the token (the `jwtToken` variable in the code below) of your account.

```java theme={null}
// props is the Properties object that has already configures the Token authentication
// See https://docs.streamnative.io/docs/cloud-connect-kafka-java for the necessary configs
props.put(KafkaAvroSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl);
props.put(KafkaAvroSerializerConfig.BASIC_AUTH_CREDENTIALS_SOURCE, "USER_INFO");
props.put(KafkaAvroSerializerConfig.USER_INFO_CONFIG, String.format("%s:%s", "any-user", jwtToken));
```

## Configurable compatibility modes

When using serialization and deserialization formats such as Avro, JSON Schema, and Protobuf, we need to remember that there are different configurable compatibility modes. In the Schema Registry, schema compatibility is managed by versioning each individual schema. The compatibility type determines how the Schema Registry compares the new schema with previous versions of a schema, for a given subject. Upon its initial creation within a subject, a schema is assigned a unique identifier and a version number, starting at version 1. If the schema is updated and successfully passes the compatibility checks, it is given a new unique identifier and an incremented version number, i.e., version 2.

|                      |      |      |          |
| -------------------- | ---- | ---- | -------- |
| Compatibility modes  | AVRO | JSON | Protobuf |
| NONE                 | YES  | YES  | YES      |
| BACKWARD             | YES  | YES  | YES      |
| BACKWARD\_TRANSITIVE | YES  | YES  | YES      |
| FORWARD              | YES  | YES  | -        |
| FORWARD\_TRANSITIVE  | YES  | YES  | -        |
| FULL                 | YES  | YES  | -        |
| FULL\_TRANSITIVE     | YES  | YES  | -        |

## REST API

Kafka schema registry provides REST API for managing schemas. The following table lists the supported methods and parameters, more details about the API, please refer to [Schema Registry API](https://docs.confluent.io/platform/current/schema-registry/develop/api.html).

| API                                                                      | Method | Support Parameters                       |
| ------------------------------------------------------------------------ | ------ | ---------------------------------------- |
| `/schemas/ids/{int: id}`                                                 | GET    |                                          |
| `/schemas/ids/{int: id}/schema`                                          | GET    |                                          |
| `/schemas/types`                                                         | GET    |                                          |
| `/schemas/ids/{int: id}/versions`                                        | GET    |                                          |
| `/schemas/ids/{int: id}/subjects`                                        | GET    |                                          |
| `/subjects`                                                              | GET    | deleted (boolean), deletedOnly (boolean) |
| `/subjects/(string: subject)`                                            | POST   | normalize (boolean), deleted (boolean)   |
| `/subjects/(string: subject)`                                            | DELETE | permanent (boolean)                      |
| `/subjects/(string: subject)/versions`                                   | POST   | normalize (boolean)                      |
| `/subjects/(string: subject)/versions`                                   | GET    | deleted (boolean), deletedOnly (boolean) |
| `/subjects/(string: subject)/versions/(versionId: version)`              | GET    | deleted (boolean)                        |
| `/subjects/(string: subject)/versions/(versionId: version)`              | DELETE | permanent (boolean)                      |
| `/subjects/(string: subject)/versions/(versionId: version)/schema`       | GET    |                                          |
| `/subjects/(string: subject)/versions/(versionId: version)/referencedby` | GET    |                                          |
| `/compatibility/subjects/(string: subject)/versions/latest`              | GET    |                                          |
| `/config/(string: subject)`                                              | PUT    | only support set compatibility           |
| `/config/(string: subject)`                                              | GET    | only support get compatibility           |
| `/mode`                                                                  | GET    | only support the mode READWRITE          |

## Use Schema Registry on Console

1. On the left navigation pane of StreamNative Console, in the **Admin** section, click **Kafka Clients**, and choose the Java client, then enable the Kafka Schema Registry by following switch.
   <img alt="enable-kafka-schema-registry.png" />

2. Please make sure you granted permission(produce) for topic `public/__kafka_schemaregistry/__schema-registry` in the following page.
   <img alt="granted-permission-for-schema-registry-topic.png" />

We need to mention that Now the Kafka Schemas can’t work with Pulsar schemas. This is the mission of the unified schema registry.

## Enable Broker-side Schemas IDs Validation

Broker-side Schema ID Validation allows broker to validate the schema ID of the messages they send against the schema ID registered in the Schema Registry. This feature helps ensure that producers are sending messages with the correct schema, reducing the risk of data inconsistencies and errors. For more information, see [Validate Broker-side Schemas IDs](https://docs.confluent.io/platform/current/schema-registry/schema-validation.html).

### Limitations

Schema validation feature does not reject tombstone records (messages with null value) even if there is no schema ID associated with the record. This is to ensure that delete operations can still be performed on compacted topics without being blocked by schema validation.

### Enable Schema ID Validation on a Topic

Create a topic with Schema ID Validation enabled you can set the topic property `kop.kafka.key.schema.validation=true` and `kop.kafka.value.schema.validation=true` when creating the topic. For example, to create a topic named `my-topic-sv` with value schema validation, run the following command:

```bash theme={null}
snctl kafka admin topics create my-topic-sv --partitions 4 --config kop.kafka.value.schema.validation=true
```

Or

```bash theme={null}
bin/pulsar-admin \
    --admin-url <Pulsar Service Http Url> \
    --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
    --auth-params token:<API Key> \
  topics create-partitioned-topic persistent://public/default/my-topic-sv -p 4 -m kop.kafka.value.schema.validation=true
```

With this property set, if the message value does not have a schema ID or has a schema ID that does not match the schema registered in the Schema Registry, the broker will reject the message and return an error to the producer. And the message will be discarded.

### Change the subject name strategy

By default, the subject name strategy is set to `TopicNameStrategy`, which means that the subject name is derived from the topic name. If you want to change the subject name strategy, you can set the topic property `kop.kafka.schema.subject.name.strategy` to one of the following values:

* `TopicNameStrategy`: The subject name is derived from the topic name. For example, for a topic named `my-topic`, the subject name will be `my-topic-value` for value schema and `my-topic-key` for key schema.
* `RecordNameStrategy`: The subject name is derived from the fully qualified name of the record
* `TopicRecordNameStrategy`: The subject name is derived from the topic name and the fully qualified name of the record

For example, to create a topic named `my-topic-sv` with value schema validation and `RecordNameStrategy`, run the following command:

```bash theme={null}
snctl kafka admin topics create my-other-topic-sv --partitions 4 --config kop.kafka.value.schema.validation=true --config kop.kafka.value.subject.name.strategy=io.confluent.kafka.serializers.subject.RecordNameStrategy
```

Or

```bash theme={null}
bin/pulsar-admin \
    --admin-url <Pulsar Service Http Url> \
    --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
    --auth-params token:<API Key> \
  topics create-partitioned-topic persistent://public/default/my-other-topic-sv -p 4 -m kop.kafka.value.schema.validation=true -m kop.kafka.value.subject.name.strategy=io.confluent.kafka.serializers.subject.RecordNameStrategy
```

## Related

To use Kafka schemas from Pulsar Java clients:

* [Use External JSON Schema with Pulsar clients](/cloud/governance/kafka-schemas/external-json-schema)
* [Use External Avro Schema with Pulsar clients](/cloud/governance/kafka-schemas/external-avro-schema)
* [Use External Protobuf Schema with Pulsar clients](/cloud/governance/kafka-schemas/external-protobuf-schema)


# Private Networking for Databricks Unity Catalog (Iceberg)
Source: https://docs.streamnative.io/cloud/lakehouse/catalogs/private-networking/databricks-iceberg



This guide describes how to configure private network connections between StreamNative Cloud and Databricks Unity Catalog for Iceberg. Private connectivity ensures that traffic between your StreamNative BYOC cluster and Databricks Unity Catalog does not traverse the public internet.

Databricks Unity Catalog uses the same private connectivity infrastructure for both Iceberg and Delta Lake. If you use Delta Lake with Databricks, see [Private Networking for Databricks Unity Catalog (Delta Lake)](/cloud/lakehouse/catalogs/private-networking/databricks-unity-catalog).

The following diagram shows the network path between your StreamNative BYOC cluster and Databricks Unity Catalog over private connectivity.

```mermaid theme={null}
flowchart TB
    subgraph BYOC["StreamNative BYOC VPC"]
        Cluster["BYOC Cluster"]
        Endpoint["Private Endpoint"]
    end

    Databricks["Databricks Unity Catalog<br/>(Iceberg)"]

    Cluster -->|"catalog API requests"| Endpoint --> Databricks

    classDef byoc fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
    classDef edge fill:#fde68a,stroke:#b45309,color:#451a03
    classDef ext  fill:#ede9fe,stroke:#7c3aed,color:#4c1d95
    class Cluster byoc
    class Endpoint edge
    class Databricks ext
```

## Prerequisites

* A StreamNative BYOC cluster deployed on AWS, GCP, or Azure.
* A Databricks workspace in the same cloud provider and region as your StreamNative BYOC cluster.
* A prepared Databricks Unity Catalog for Iceberg. See [Prepare Lakehouse Catalogs](/cloud/lakehouse/prepare-lakehouse-catalogs) for the cloud-specific setup guides.
* Storage private connectivity configured (if applicable). See [Private Networking for Catalog Integration](/cloud/lakehouse/catalogs/private-networking/overview) for storage connectivity details.

## Configure private connectivity

Configure private connectivity to Databricks Unity Catalog based on the cloud provider where your StreamNative BYOC cluster is deployed.

<Tabs>
  <Tab title="AWS">
    Use AWS PrivateLink to establish a private connection between your StreamNative Cloud environment and Databricks Unity Catalog.

    Follow the Databricks documentation to configure inbound PrivateLink:
    [Configure Inbound PrivateLink for Databricks](https://docs.databricks.com/en/security/network/front-end/front-end-private-connect.html).
  </Tab>

  <Tab title="GCP">
    Use Google Cloud Private Service Connect to establish a private connection between your StreamNative Cloud environment and Databricks Unity Catalog.

    Follow the Databricks documentation to configure front-end Private Service Connect:
    [Configure Front-end Private Service Connect for Databricks on GCP](https://docs.databricks.com/gcp/en/security/network/front-end/front-end-private-connect.html).
  </Tab>

  <Tab title="Azure">
    Use Azure Private Link to establish a private connection between your StreamNative Cloud environment and Databricks Unity Catalog.

    Follow the Databricks documentation to configure inbound Private Link:
    [Configure Inbound Private Link for Databricks on Azure](https://learn.microsoft.com/en-us/azure/databricks/security/network/front-end/front-end-private-connect).
  </Tab>
</Tabs>

## Update the catalog URI

After enabling private connectivity, you may need to update the catalog URI in StreamNative Cloud. The catalog URI is configured during [catalog registration](/cloud/lakehouse/catalogs/register-catalog) and can be updated through [Manage Catalogs](/cloud/lakehouse/catalogs/manage-catalogs).

<Note>
  If your Databricks workspace is configured with private DNS, the existing workspace URL resolves to the private endpoint automatically and no URI change is needed. Otherwise, update the catalog URI to use the private endpoint hostname.
</Note>


# Private Networking for Databricks Unity Catalog (Delta Lake)
Source: https://docs.streamnative.io/cloud/lakehouse/catalogs/private-networking/databricks-unity-catalog



This guide describes how to configure private network connections between StreamNative Cloud and Databricks Unity Catalog for Delta Lake. Private connectivity ensures that traffic between your StreamNative BYOC cluster and Databricks Unity Catalog does not traverse the public internet.

Databricks Unity Catalog uses the same private connectivity infrastructure for both Delta Lake and Iceberg. If you use Iceberg with Databricks, see [Private Networking for Databricks Unity Catalog (Iceberg)](/cloud/lakehouse/catalogs/private-networking/databricks-iceberg).

The following diagram shows the network path between your StreamNative BYOC cluster and Databricks Unity Catalog over private connectivity.

```mermaid theme={null}
flowchart TB
    subgraph BYOC["StreamNative BYOC VPC"]
        Cluster["BYOC Cluster"]
        Endpoint["Private Endpoint"]
    end

    Databricks["Databricks Unity Catalog<br/>(Delta Lake)"]

    Cluster -->|"catalog API requests"| Endpoint --> Databricks

    classDef byoc fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
    classDef edge fill:#fde68a,stroke:#b45309,color:#451a03
    classDef ext  fill:#ede9fe,stroke:#7c3aed,color:#4c1d95
    class Cluster byoc
    class Endpoint edge
    class Databricks ext
```

## Prerequisites

* A StreamNative BYOC cluster deployed on AWS, GCP, or Azure.
* A Databricks workspace in the same cloud provider and region as your StreamNative BYOC cluster.
* A prepared Databricks Unity Catalog for Delta Lake. See [Prepare Lakehouse Catalogs](/cloud/lakehouse/prepare-lakehouse-catalogs) for the cloud-specific setup guides.
* Storage private connectivity configured (if applicable). See [Private Networking for Catalog Integration](/cloud/lakehouse/catalogs/private-networking/overview) for storage connectivity details.

## Configure private connectivity

Configure private connectivity to Databricks Unity Catalog based on the cloud provider where your StreamNative BYOC cluster is deployed.

<Tabs>
  <Tab title="AWS">
    Use AWS PrivateLink to establish a private connection between your StreamNative Cloud environment and Databricks Unity Catalog.

    Follow the Databricks documentation to configure inbound PrivateLink:
    [Configure Inbound PrivateLink for Databricks](https://docs.databricks.com/en/security/network/front-end/front-end-private-connect.html).
  </Tab>

  <Tab title="GCP">
    Use Google Cloud Private Service Connect to establish a private connection between your StreamNative Cloud environment and Databricks Unity Catalog.

    Follow the Databricks documentation to configure front-end Private Service Connect:
    [Configure Front-end Private Service Connect for Databricks on GCP](https://docs.databricks.com/gcp/en/security/network/front-end/front-end-private-connect.html).
  </Tab>

  <Tab title="Azure">
    Use Azure Private Link to establish a private connection between your StreamNative Cloud environment and Databricks Unity Catalog.

    Follow the Databricks documentation to configure inbound Private Link:
    [Configure Inbound Private Link for Databricks on Azure](https://learn.microsoft.com/en-us/azure/databricks/security/network/front-end/front-end-private-connect).
  </Tab>
</Tabs>

## Update the catalog URI

After enabling private connectivity, you may need to update the catalog URI in StreamNative Cloud. The catalog URI is configured during [catalog registration](/cloud/lakehouse/catalogs/register-catalog) and can be updated through [Manage Catalogs](/cloud/lakehouse/catalogs/manage-catalogs).

<Note>
  If your Databricks workspace is configured with private DNS, the existing workspace URL resolves to the private endpoint automatically and no URI change is needed. Otherwise, update the catalog URI to use the private endpoint hostname.
</Note>


# Private Networking for Google BigLake
Source: https://docs.streamnative.io/cloud/lakehouse/catalogs/private-networking/google-biglake



This guide describes how to configure private network connections between StreamNative Cloud and Google BigLake metastore. Private connectivity ensures that traffic between your StreamNative BYOC cluster and Google BigLake does not traverse the public internet.

Google BigLake is available only on GCP.

The following diagram shows the network path between your StreamNative BYOC cluster and Google BigLake metastore over private connectivity.

```mermaid theme={null}
flowchart TB
    subgraph BYOC["StreamNative BYOC VPC (GCP)"]
        Cluster["BYOC Cluster"]
        PGA["Private Google API Access"]
    end

    BL["Google BigLake Metastore"]

    Cluster -->|"catalog API requests"| PGA --> BL

    classDef byoc fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
    classDef edge fill:#fde68a,stroke:#b45309,color:#451a03
    classDef ext  fill:#ede9fe,stroke:#7c3aed,color:#4c1d95
    class Cluster byoc
    class PGA edge
    class BL ext
```

## Prerequisites

* A StreamNative BYOC cluster deployed on GCP.
* A prepared Google BigLake catalog. See [Prepare Google BigLake (Iceberg)](/cloud/lakehouse/prepare-catalogs/biglake/iceberg).
* Storage private connectivity configured (if applicable). See [Private Networking for Catalog Integration](/cloud/lakehouse/catalogs/private-networking/overview) for storage connectivity details.

## Configure private connectivity

Google BigLake metastore is a Google-managed service that runs within the Google Cloud network. On GCP, StreamNative configures private network connections to Google Cloud APIs by default in all StreamNative environments.

**No additional action is required on your side** in most cases. All traffic between your StreamNative BYOC cluster and Google BigLake stays within the Google private network automatically.

If your BYOC cluster runs in a [Shared VPC](https://cloud.google.com/vpc/docs/shared-vpc), you must configure private connectivity yourself. See [Private Networking for Catalog Integration](/cloud/lakehouse/catalogs/private-networking/overview#storage-private-connectivity) for details.


# Private Networking for Amazon S3 Tables
Source: https://docs.streamnative.io/cloud/lakehouse/catalogs/private-networking/s3-tables



This guide describes how to configure private network connections between StreamNative Cloud and Amazon S3 Tables. Private connectivity ensures that traffic between your StreamNative BYOC cluster and Amazon S3 Tables does not traverse the public internet.

Amazon S3 Tables is available only on AWS.

The following diagram shows the network path between your StreamNative BYOC cluster and Amazon S3 Tables over private connectivity.

```mermaid theme={null}
flowchart TB
    subgraph BYOC["StreamNative BYOC VPC (AWS)"]
        Cluster["BYOC Cluster"]
        VPCE["S3 Tables VPC Endpoint"]
    end

    S3T["Amazon S3 Tables"]

    Cluster -->|"catalog and data traffic"| VPCE --> S3T

    classDef byoc fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
    classDef edge fill:#fde68a,stroke:#b45309,color:#451a03
    classDef ext  fill:#ede9fe,stroke:#7c3aed,color:#4c1d95
    class Cluster byoc
    class VPCE edge
    class S3T ext
```

## Prerequisites

* A StreamNative BYOC cluster deployed on AWS.
* A prepared Amazon S3 Tables catalog. See [Prepare Amazon S3 Tables (Iceberg)](/cloud/lakehouse/prepare-catalogs/s3table/iceberg).
* Storage private connectivity configured (if applicable). See [Private Networking for Catalog Integration](/cloud/lakehouse/catalogs/private-networking/overview) for storage connectivity details.

## Configure private connectivity

Amazon S3 Tables uses S3 endpoints for both data storage and catalog operations. On AWS, StreamNative configures private network connections to Amazon S3 endpoints by default in all StreamNative environments. The S3 VPC endpoint is configured per VPC.

**No additional action is required on your side.** All traffic between your StreamNative BYOC cluster and Amazon S3 Tables stays within the AWS private network automatically.


# Private Networking for Snowflake Horizon Catalog
Source: https://docs.streamnative.io/cloud/lakehouse/catalogs/private-networking/snowflake-horizon-catalog



This guide describes how to configure private network connections between StreamNative Cloud and Snowflake Horizon Catalog. Private connectivity ensures that traffic between your StreamNative BYOC cluster and Snowflake Horizon Catalog does not traverse the public internet.

Snowflake Horizon Catalog uses the same Snowflake private connectivity infrastructure as Snowflake Open Catalog. The setup process is identical across both catalog types. See also [Private Networking for Snowflake Open Catalog](/cloud/lakehouse/catalogs/private-networking/snowflake-open-catalog).

The following diagram shows the network path between your StreamNative BYOC cluster and Snowflake Horizon Catalog over private connectivity.

```mermaid theme={null}
flowchart TB
    subgraph BYOC["StreamNative BYOC VPC"]
        Cluster["BYOC Cluster"]
        Endpoint["Private Endpoint"]
    end

    Snowflake["Snowflake Horizon Catalog"]

    Cluster -->|"catalog API requests"| Endpoint --> Snowflake

    classDef byoc fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
    classDef edge fill:#fde68a,stroke:#b45309,color:#451a03
    classDef ext  fill:#ede9fe,stroke:#7c3aed,color:#4c1d95
    class Cluster byoc
    class Endpoint edge
    class Snowflake ext
```

## Prerequisites

* A StreamNative BYOC cluster deployed on AWS, GCP, or Azure.
* A Snowflake account with Horizon Catalog enabled, in the same cloud provider and region as your StreamNative BYOC cluster.
* Storage private connectivity configured (if applicable). See [Private Networking for Catalog Integration](/cloud/lakehouse/catalogs/private-networking/overview) for storage connectivity details.

## Configure private connectivity

Configure private connectivity to Snowflake Horizon Catalog based on the cloud provider where your StreamNative BYOC cluster is deployed.

<Tabs>
  <Tab title="AWS">
    Use AWS PrivateLink to establish a private connection between your StreamNative Cloud environment and Snowflake Horizon Catalog.

    Follow the Snowflake documentation to configure PrivateLink:
    [Snowflake PrivateLink on AWS](https://docs.snowflake.com/en/user-guide/admin-security-privatelink).
  </Tab>

  <Tab title="GCP">
    Use Google Cloud Private Service Connect to establish a private connection between your StreamNative Cloud environment and Snowflake Horizon Catalog.

    Follow the Snowflake documentation to configure Private Service Connect:
    [Snowflake Private Service Connect on GCP](https://docs.snowflake.com/en/user-guide/private-service-connect-google).
  </Tab>

  <Tab title="Azure">
    Use Azure Private Link to establish a private connection between your StreamNative Cloud environment and Snowflake Horizon Catalog.

    Follow the Snowflake documentation to configure Private Link:
    [Snowflake Private Link on Azure](https://docs.snowflake.com/en/user-guide/privatelink-azure).
  </Tab>
</Tabs>

## Update the catalog URI

After enabling private connectivity, update the catalog URI in StreamNative Cloud to use the PrivateLink hostname. The catalog URI is configured during [catalog registration](/cloud/lakehouse/catalogs/register-catalog) and can be updated through [Manage Catalogs](/cloud/lakehouse/catalogs/manage-catalogs).

Change the URI from the public format:

```
https://<account-locator>.<catalog-region>.snowflakecomputing.com/polaris/api/catalog
```

to the PrivateLink format:

```
https://<account-locator>.privatelink.snowflakecomputing.com/polaris/api/catalog
```

<Note>
  The exact private endpoint hostname may vary by cloud provider. Refer to the Snowflake PrivateLink documentation for your cloud provider to determine the correct hostname.
</Note>


# Private Networking for Snowflake Open Catalog
Source: https://docs.streamnative.io/cloud/lakehouse/catalogs/private-networking/snowflake-open-catalog



This guide describes how to configure private network connections between StreamNative Cloud and Snowflake Open Catalog. Private connectivity ensures that traffic between your StreamNative BYOC cluster and Snowflake Open Catalog does not traverse the public internet.

Snowflake Open Catalog uses the same private connectivity infrastructure as Snowflake Horizon Catalog. If you use Horizon Catalog, see [Private Networking for Snowflake Horizon Catalog](/cloud/lakehouse/catalogs/private-networking/snowflake-horizon-catalog).

The following diagram shows the network path between your StreamNative BYOC cluster and Snowflake Open Catalog over private connectivity.

```mermaid theme={null}
flowchart TB
    subgraph BYOC["StreamNative BYOC VPC"]
        Cluster["BYOC Cluster"]
        Endpoint["Private Endpoint"]
    end

    Snowflake["Snowflake Open Catalog"]

    Cluster -->|"catalog API requests"| Endpoint --> Snowflake

    classDef byoc fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
    classDef edge fill:#fde68a,stroke:#b45309,color:#451a03
    classDef ext  fill:#ede9fe,stroke:#7c3aed,color:#4c1d95
    class Cluster byoc
    class Endpoint edge
    class Snowflake ext
```

## Prerequisites

* A StreamNative BYOC cluster deployed on AWS, GCP, or Azure.
* A Snowflake Open Catalog account in the same cloud provider and region as your StreamNative BYOC cluster.
* A prepared Snowflake Open Catalog. See [Prepare Lakehouse Catalogs](/cloud/lakehouse/prepare-lakehouse-catalogs) for the cloud-specific setup guides.
* Storage private connectivity configured (if applicable). See [Private Networking for Catalog Integration](/cloud/lakehouse/catalogs/private-networking/overview) for storage connectivity details.

## Configure private connectivity

Configure private connectivity to Snowflake Open Catalog based on the cloud provider where your StreamNative BYOC cluster is deployed.

<Tabs>
  <Tab title="AWS">
    Use AWS PrivateLink to establish a private connection between your StreamNative Cloud environment and Snowflake Open Catalog.

    Follow the Snowflake documentation to configure PrivateLink:
    [Snowflake PrivateLink on AWS](https://docs.snowflake.com/en/user-guide/admin-security-privatelink).
  </Tab>

  <Tab title="GCP">
    Use Google Cloud Private Service Connect to establish a private connection between your StreamNative Cloud environment and Snowflake Open Catalog.

    Follow the Snowflake documentation to configure Private Service Connect:
    [Snowflake Private Service Connect on GCP](https://docs.snowflake.com/en/user-guide/private-service-connect-google).
  </Tab>

  <Tab title="Azure">
    Use Azure Private Link to establish a private connection between your StreamNative Cloud environment and Snowflake Open Catalog.

    Follow the Snowflake documentation to configure Private Link:
    [Snowflake Private Link on Azure](https://docs.snowflake.com/en/user-guide/privatelink-azure).
  </Tab>
</Tabs>

## Update the catalog URI

After enabling private connectivity, update the catalog URI in StreamNative Cloud to use the PrivateLink hostname. The catalog URI is configured during [catalog registration](/cloud/lakehouse/catalogs/register-catalog) and can be updated through [Manage Catalogs](/cloud/lakehouse/catalogs/manage-catalogs).

Change the URI from the public format:

```
https://<account-locator>.<catalog-region>.snowflakecomputing.com/polaris/api/catalog
```

to the PrivateLink format:

```
https://<account-locator>.privatelink.snowflakecomputing.com/polaris/api/catalog
```

<Note>
  The exact private endpoint hostname may vary by cloud provider. Refer to the Snowflake PrivateLink documentation for your cloud provider to determine the correct hostname.
</Note>


# Lakehouse Observability
Source: https://docs.streamnative.io/cloud/lakehouse/lakehouse-observability



## Monitor data delivery progress in the Cloud Console

After you [enable the Lakehouse Table](/cloud/lakehouse/enable-lakehouse-integration) on a topic, the StreamNative Cloud Console shows a per-topic delivery dashboard. Use it to check the health of data delivery without any external monitoring setup.

Open a topic in the Cloud Console and select the **Lakehouse Table** tab.

<Frame>
  <img alt="Lakehouse Table tab on a topic, showing Streaming Lag, Last Success Commit Time, and Rejected Count" />
</Frame>

The dashboard reports three delivery indicators and the catalog the topic is delivering to.

| Indicator                    | What it means                                                                                                                                                                                                        | What to do if it looks wrong                                                                                                                                                                                                     |
| ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Streaming Lag**            | Number of messages produced to the topic but not yet committed to the lakehouse table. A small, steady value is expected; a continuously growing value indicates that delivery cannot keep up with the produce rate. | Check the produce rate, the catalog's availability, and recent **Rejected Count** changes. For deeper investigation, see the [Grafana dashboard](#grafana-dashboard) and the `pulsar_storage_compact_lag` metric.                |
| **Last Success Commit Time** | Time elapsed since the most recent successful commit to the lakehouse catalog. Updates regularly while the topic has traffic.                                                                                        | If this value keeps growing while the topic is actively receiving messages, delivery is stalled. Verify the catalog credentials and connectivity, then check the failure metrics in the [Grafana dashboard](#grafana-dashboard). |
| **Rejected Count**           | Number of messages that could not be written to the lakehouse table -- for example, messages that failed schema validation or exceeded size limits.                                                                  | A non-zero value means some messages were not delivered. Inspect the topic schema and producer payloads. Rejected messages are not retried automatically.                                                                        |

The **Catalog Settings** panel below confirms which catalog the topic is delivering to and shows whether the setting is inherited from the cluster, the namespace, or set directly on the topic. See [Configuration override priority](/cloud/lakehouse/enable-lakehouse-integration#configuration-override-priority) for how the effective catalog is resolved.

<Tip>
  The Cloud Console dashboard is the fastest way to verify delivery for a single topic. For fleet-wide monitoring, alerting, and historical trends, set up the [Grafana dashboard](#grafana-dashboard).
</Tip>

## Prerequisites

Before you can visualize Lakehouse metrics in your own Grafana instance, [enable Metrics Remote Write](/cloud/log-and-monitor/advanced-observability#metrics-remote-write-integration) on your Cloud Environment to forward StreamNative Cloud metrics to your Prometheus-compatible monitoring system or Datadog.

## Grafana Dashboard

A pre-built Grafana dashboard is available as [`CompactionScheduler.json`](https://github.com/streamnative/apache-pulsar-grafana-dashboard/tree/master/dashboards.kubernetes) in the [apache-pulsar-grafana-dashboard](https://github.com/streamnative/apache-pulsar-grafana-dashboard) repository. Import it into your Grafana instance for comprehensive monitoring.

### How to Import

1. Download [`CompactionScheduler.json`](https://github.com/streamnative/apache-pulsar-grafana-dashboard/tree/master/dashboards.kubernetes) from the repository.
2. Open Grafana -> **Dashboards** -> **Import**.
3. Upload `CompactionScheduler.json` or paste the JSON content.
4. Select your Prometheus data source.
5. Click **Import**.

### Dashboard Overview

The dashboard is organized into the following sections:

| Section                      | Description                                                                                                                                                                                                      |
| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Overview**                 | Topic count, task count, publish/compact/commit failed tasks, commit batch size                                                                                                                                  |
| **Compaction Write**         | Compaction lag, task publish lag, task stats, non-committable tasks, throughput (bytes/messages), latencies for compaction duration, WAL read, Parquet write, task commit, lakehouse commit, end-to-end pipeline |
| **Persistent API**           | Read throughput, read latencies (index+data, message, Oxia index, Oxia metadata)                                                                                                                                 |
| **WAL**                      | Read cache eviction/loading rate, WAL read latency, S3 cache loading latency                                                                                                                                     |
| **S3**                       | S3 read throughput, request rate, S3 read latency                                                                                                                                                                |
| **Compaction Read**          | Lakehouse read bytes/messages, read latency                                                                                                                                                                      |
| **Compaction Write Details** | Lakehouse write/encode/before-write/write-record latencies, Parquet write-record/write-metadata latencies                                                                                                        |
| **DLQ Tasks**                | Dead Letter Queue task statistics                                                                                                                                                                                |

***

## Key Alerts

These metrics should be monitored with alerting rules:

| Metric                                                                                       | Alert Condition                            | Severity |
| -------------------------------------------------------------------------------------------- | ------------------------------------------ | -------- |
| `pulsar_storage_compact_lag`                                                                 | Compaction lag exceeds threshold per topic | Warning  |
| `compaction_cluster_leaders_ratio`                                                           | Sum across cluster is not exactly 1        | Critical |
| `pulsar_storage_compact_quarantined_topics_count`                                            | Greater than 0                             | Warning  |
| `pulsar_storage_compact_topics_in_dlq`                                                       | Greater than 0                             | Critical |
| `pulsar_storage_compact_tasks_in_dlq`                                                        | Greater than 0                             | Critical |
| `pulsar_storage_compact_publish_task_failed_count_total`                                     | Increasing                                 | Warning  |
| `pulsar_storage_compact_failed_task_count_total`                                             | Increasing                                 | Warning  |
| `pulsar_storage_compact_task_commit_duration_seconds_count{pulsar_response_status="failed"}` | Increasing                                 | Critical |
| `pulsar_subscription_back_log`                                                               | Backlog exceeds threshold                  | Warning  |

***

## Compaction Service Metrics

The compaction service has three stages: task publishing (leader), WAL-to-Parquet conversion (worker), and commit to lakehouse (leader).

### Task Lifecycle

| Metric                                                  | Type  | Description                                      |
| ------------------------------------------------------- | ----- | ------------------------------------------------ |
| `pulsar_storage_compact_ongoing_topic_count`            | Gauge | Number of topics currently undergoing compaction |
| `pulsar_storage_compact_ongoing_task_count`             | Gauge | Number of active compaction tasks in progress    |
| `pulsar_storage_compact_tasks_in_init_state`            | Gauge | Tasks in initialization state                    |
| `pulsar_storage_compact_tasks_in_compacted_state`       | Gauge | Tasks in compacted state                         |
| `pulsar_storage_compact_tasks_in_prepared_commit_state` | Gauge | Tasks in prepared commit state                   |
| `pulsar_storage_compact_tasks_in_committed_state`       | Gauge | Tasks in committed state                         |

### Throughput

| Metric                                                | Type    | Description                                              |
| ----------------------------------------------------- | ------- | -------------------------------------------------------- |
| `pulsar_storage_compact_bytes_total`                  | Counter | Total bytes processed during compaction                  |
| `pulsar_storage_compact_messages_total`               | Counter | Total messages processed during compaction               |
| `pulsar_storage_compact_published_task_bytes`         | Gauge   | Size in bytes of messages batched in one compaction task |
| `pulsar_storage_compact_committed_parquet_file_bytes` | Gauge   | Size in bytes of committed Parquet files                 |
| `pulsar_storage_compact_commit_task_batch_size`       | Gauge   | Number of Parquet files in a single commit batch         |

### Offset Tracking

| Metric                                           | Type  | Description                                                        |
| ------------------------------------------------ | ----- | ------------------------------------------------------------------ |
| `pulsar_storage_compact_latest_message_offset`   | Gauge | Latest message offset for each topic                               |
| `pulsar_storage_compact_latest_published_offset` | Gauge | Latest published task's message offset                             |
| `pulsar_storage_compact_last_compacted_offset`   | Gauge | Latest offset confirmed as fully committed to lakehouse            |
| `pulsar_storage_compact_lag`                     | Gauge | Difference between latest message offset and last compacted offset |

### Latency

| Metric                                                                        | Type      | Description                                                            |
| ----------------------------------------------------------------------------- | --------- | ---------------------------------------------------------------------- |
| `pulsar_storage_compact_duration_seconds_bucket`                              | Histogram | Total latency of a compaction task                                     |
| `pulsar_storage_compact_read_messages_duration_seconds_bucket`                | Histogram | Latency for reading messages from WAL files                            |
| `pulsar_storage_compact_write_messages_duration_seconds_bucket`               | Histogram | Latency for decoding, converting, and writing to Parquet               |
| `pulsar_storage_compact_task_commit_duration_seconds_bucket`                  | Histogram | Latency for committing a task (includes Oxia index + catalog snapshot) |
| `pulsar_storage_compact_commit_to_lakehouse_duration_seconds_bucket`          | Histogram | Latency for committing snapshot to catalog service only                |
| `pulsar_storage_compact_message_from_ursa_to_parquet_duration_seconds_bucket` | Histogram | End-to-end latency: message write to Parquet file write                |
| `pulsar_storage_compact_message_end_to_end_duration_seconds_bucket`           | Histogram | End-to-end latency: message write to lakehouse commit                  |

### Failures

| Metric                                                               | Type      | Description                                   |
| -------------------------------------------------------------------- | --------- | --------------------------------------------- |
| `pulsar_storage_compact_publish_task_failed_count_total`             | Counter   | Total failed task publications                |
| `pulsar_storage_compact_failed_task_count_total`                     | Counter   | Total failed WAL-to-Parquet conversions       |
| `pulsar_storage_compact_quarantined_topics_count`                    | Gauge     | Topics quarantined due to compaction failures |
| `pulsar_storage_compact_topics_in_dlq`                               | Gauge     | Topics in Dead Letter Queue                   |
| `pulsar_storage_compact_tasks_in_dlq`                                | Gauge     | Tasks in Dead Letter Queue                    |
| `pulsar_storage_compact_non_committable_task_count`                  | Counter   | Non-committable tasks exceeding threshold     |
| `pulsar_storage_compact_non_committable_task_histogram_bytes_bucket` | Histogram | Size distribution of non-committable tasks    |

***

## WAL Storage Metrics

| Metric                                                         | Type      | Description                             |
| -------------------------------------------------------------- | --------- | --------------------------------------- |
| `pulsar_storage_wal_putEntry_count_total`                      | Counter   | Total entries written to WAL            |
| `pulsar_storage_wal_putEntry_rejected_count_total`             | Counter   | Total entries rejected during WAL write |
| `pulsar_storage_wal_putEntry_duration_seconds_bucket`          | Histogram | WAL write latency                       |
| `pulsar_storage_wal_putEntry_pending_duration_seconds_bucket`  | Histogram | Time entries wait in WAL buffer         |
| `pulsar_storage_wal_putEntry_cache_duration_seconds_bucket`    | Histogram | Write cache write latency               |
| `pulsar_storage_wal_getEntries_duration_seconds_bucket`        | Histogram | Batch read latency (cache or backend)   |
| `pulsar_storage_wal_getEntry_duration_seconds_bucket`          | Histogram | Single entry read latency               |
| `pulsar_storage_wal_writeCache_flush_duration_seconds_bucket`  | Histogram | Write cache flush latency               |
| `pulsar_storage_wal_readCache_loading_count_total`             | Counter   | Read cache loads from backend           |
| `pulsar_storage_wal_readCache_eviction_count_total`            | Counter   | Read cache evictions                    |
| `pulsar_storage_wal_readCache_loading_duration_seconds_bucket` | Histogram | Cache loading latency                   |
| `pulsar_storage_wal_read_cache_missed_total`                   | Counter   | Read cache misses                       |
| `pulsar_storage_wal_putEntry_pending_count`                    | Gauge     | Entries queued in WAL pending buffer    |
| `pulsar_storage_wal_writeCache_flushCallback_pending_count`    | Gauge     | Pending flush acknowledgments           |
| `pulsar_storage_wal_readCache_size_bytes`                      | Gauge     | Current read cache size                 |

### Write Cache Metrics

| Metric                                             | Type  | Description              |
| -------------------------------------------------- | ----- | ------------------------ |
| `pulsar_storage_wal_writeCache_used_bytes`         | Gauge | Write cache utilization  |
| `pulsar_storage_wal_writeCache_bufferSegment_used` | Gauge | Buffer segments in use   |
| `pulsar_storage_wal_writeCache_cacheSegment_used`  | Gauge | Cache segments in use    |
| `pulsar_storage_wal_writeCache_segment_count`      | Gauge | Total allocated segments |
| `pulsar_storage_wal_writeCache_capacity_bytes`     | Gauge | Max capacity per segment |

***

## File Storage Metrics

| Metric                                                         | Type      | Description                      |
| -------------------------------------------------------------- | --------- | -------------------------------- |
| `pulsar_storage_backend_storage_request_total`                 | Counter   | Total backend storage operations |
| `pulsar_storage_backend_write_duration_seconds_bucket`         | Histogram | Backend write latency            |
| `pulsar_storage_backend_read_duration_seconds_bucket`          | Histogram | Backend read latency             |
| `pulsar_storage_backend_metadata_read_duration_seconds_bucket` | Histogram | Metadata read latency            |
| `pulsar_storage_backend_crc_duration_seconds_bucket`           | Histogram | CRC calculation latency          |
| `pulsar_storage_backend_delete_duration_seconds_bucket`        | Histogram | Object deletion latency          |
| `pulsar_storage_backend_write_bytes_count_bytes_total`         | Counter   | Total bytes written to backend   |
| `pulsar_storage_backend_read_bytes_count_bytes_total`          | Counter   | Total bytes read from backend    |

***

## Lakehouse Read Metrics

| Metric                                                                | Type      | Description                                        |
| --------------------------------------------------------------------- | --------- | -------------------------------------------------- |
| `pulsar_storage_lakehouse_read_messages_total`                        | Counter   | Total messages read from lakehouse (Parquet files) |
| `pulsar_storage_lakehouse_read_bytes_bytes_total`                     | Counter   | Total bytes read from lakehouse                    |
| `pulsar_storage_lakehouse_read_request_total`                         | Counter   | Total read requests processed                      |
| `pulsar_storage_lakehouse_read_cache_hit_total`                       | Counter   | Parquet prefetch cache hits                        |
| `pulsar_storage_lakehouse_read_cache_miss_total`                      | Counter   | Parquet prefetch cache misses                      |
| `pulsar_storage_lakehouse_read_latency_seconds_bucket`                | Histogram | Read latency                                       |
| `pulsar_storage_lakehouse_read_request_queued_latency_seconds_bucket` | Histogram | Queue wait time before processing                  |

***

## Lakehouse Writer Metrics

| Metric                                                  | Type      | Description                     |
| ------------------------------------------------------- | --------- | ------------------------------- |
| `pulsar_storage_lakehouse_writer_before_write_duration` | Histogram | Pre-write operation latency     |
| `pulsar_storage_lakehouse_writer_write_all_duration`    | Histogram | Batch write latency             |
| `pulsar_storage_lakehouse_writer_write_record_duration` | Histogram | Individual record write latency |
| `pulsar_storage_lakehouse_writer_encode_duration`       | Histogram | Record encoding latency         |

## Lakehouse Reader Metrics

| Metric                                                 | Type      | Description                    |
| ------------------------------------------------------ | --------- | ------------------------------ |
| `pulsar_storage_lakehouse_reader_seek_duration`        | Histogram | Seek operation latency         |
| `pulsar_storage_lakehouse_reader_read_all_duration`    | Histogram | Batch read latency             |
| `pulsar_storage_lakehouse_reader_read_record_duration` | Histogram | Individual record read latency |
| `pulsar_storage_lakehouse_reader_decode_duration`      | Histogram | Record decoding latency        |

***

## Parquet File Metrics

### Writer

| Metric                                                     | Type      | Description                    |
| ---------------------------------------------------------- | --------- | ------------------------------ |
| `pulsar_storage_lakehouse_parquet_write_record_duration`   | Histogram | Parquet record write latency   |
| `pulsar_storage_lakehouse_parquet_write_metadata_duration` | Histogram | Parquet metadata write latency |

### Reader

| Metric                                                              | Type      | Description                     |
| ------------------------------------------------------------------- | --------- | ------------------------------- |
| `pulsar_storage_lakehouse_parquet_read_record_duration`             | Histogram | Parquet record read latency     |
| `pulsar_storage_lakehouse_parquet_read_metadata_duration`           | Histogram | Parquet metadata read latency   |
| `pulsar_storage_lakehouse_parquet_seek_by_offset_duration`          | Histogram | Seek by offset latency          |
| `pulsar_storage_lakehouse_parquet_seek_by_secondary_index_duration` | Histogram | Seek by secondary index latency |


# V4.0.10.1
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.10.1



# StreamNative Weekly Release Notes v4.0.10.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.10.1](https://github.com/streamnative/pulsar/releases/tag/v4.0.10.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.10.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.10.1/images/sha256-d2d42b5a1c0467cf66ed3b0a042941a16bf6f0dbd97e78db9e75101705a18e98)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.10.1/images/sha256-28b277e679d57df6ca19776887cb8ed1553d134c5292c0f39315c6980f2a9fef)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.10.1/images/sha256-28b277e679d57df6ca19776887cb8ed1553d134c5292c0f39315c6980f2a9fef)

## General Changes

### Apache Pulsar

([#25518](https://github.com/apache/pulsar/pull/25518)) \[improve]\[broker] Use full bundle name for namespace bundle destination affinity in ModularLoadManagerImpl

([#25566](https://github.com/apache/pulsar/pull/25566)) \[fix]\[test] Flaky SameAuthParamsLookupAutoClusterFailoverTest

([#25561](https://github.com/apache/pulsar/pull/25561)) \[fix]\[test] Fix flaky OffloadPrefixTest.testPositionOnEdgeOfLedger race with ledger rollover

([#25563](https://github.com/apache/pulsar/pull/25563)) \[fix]\[test] Extend SameAuthParamsLookupAutoClusterFailoverTest phase timeouts

([#25557](https://github.com/apache/pulsar/pull/25557)) \[fix]\[broker] pulsar admin stats internal with metadata command

([#25562](https://github.com/apache/pulsar/pull/25562)) \[fix]\[test] Relax BrokerRegistryIntegrationTest broker-close threshold

([#25389](https://github.com/apache/pulsar/pull/25389)) \[fix]\[test] Fix flaky OneWayReplicatorUsingGlobalZKTest.cleanup

([#25313](https://github.com/apache/pulsar/pull/25313)) \[fix]\[test] Fix flaky OneWayReplicatorUsingGlobalZKTest cleanup

([#25385](https://github.com/apache/pulsar/pull/25385)) \[fix]\[test] Fix flaky PersistentStickyKeyDispatcherMultipleConsumersClassicTest.testSkipRedeliverTemporally

([#24823](https://github.com/apache/pulsar/pull/24823)) \[fix]\[test] Fix flaky SingleThreadNonConcurrentFixedRateSchedulerTest.testPeriodicTaskCancellation

([#25307](https://github.com/apache/pulsar/pull/25307)) \[fix] Fix flaky testEstimatedTimeBasedBacklogQuotaCheckWhenNoBacklog

([#25463](https://github.com/apache/pulsar/pull/25463)) \[fix]\[test] Fix flaky BrokerRegistryIntegrationTest port binding race

([#25560](https://github.com/apache/pulsar/pull/25560)) \[fix]\[test] Recreate EventLoop in PublishRateLimiterTest setup

([#25502](https://github.com/apache/pulsar/pull/25502)) \[fix]\[broker] Unthrottle producers immediately when publish rate limiting is disabled

([#25558](https://github.com/apache/pulsar/pull/25558)) \[fix]\[broker] Lower log level of DrainingHashesTracker not-found entry to DEBUG

([#25098](https://github.com/apache/pulsar/pull/25098)) \[fix]\[broker] fix flaky test in SystemTopicBasedTopicPoliciesServiceTest

([#25500](https://github.com/apache/pulsar/pull/25500)) \[fix]\[test] Fix flaky ExtensibleLoadManagerTest.startBroker timeout

([#25509](https://github.com/apache/pulsar/pull/25509)) \[fix]\[test] Fix flaky ExtensibleLoadManagerImpl client reconnection tests: PulsarClientException\$AlreadyClosedException: Client already closed

([#25427](https://github.com/apache/pulsar/pull/25427)) \[fix]\[test] Fix flaky testLoadBalancerServiceUnitTableViewSyncer

([#25378](https://github.com/apache/pulsar/pull/25378)) \[fix]\[test] Fix flaky ExtensibleLoadManagerImplTest.testLoadBalancerServiceUnitTableViewSyncer

([#25497](https://github.com/apache/pulsar/pull/25497)) \[fix]\[test] Fix flaky ServerCnxTest.testCreateProducerTimeoutThenCreateSameNamedProducerShouldFail

([#25460](https://github.com/apache/pulsar/pull/25460)) \[fix]\[broker] Prevent timed-out producer creation from racing with retry

([#25358](https://github.com/apache/pulsar/pull/25358)) \[fix]\[test] Fix flaky ReplicatorTest.testResumptionAfterBacklogRelaxed

([#25551](https://github.com/apache/pulsar/pull/25551)) \[fix]\[broker]Namespaces can be created with may empty replication\_clusters policy

([#24463](https://github.com/apache/pulsar/pull/24463)) \[improve]\[broker] Improve the performance of TopicName constructor

([#25367](https://github.com/apache/pulsar/pull/25367)) \[improve]\[common] Optimize TopicName.get() to reduce lock contention on cache lookup

([#24875](https://github.com/apache/pulsar/pull/24875)) \[fix]\[test] Stabilize FunctionAssignmentTailerTest.testErrorNotifier by synchronizing mock stubbing with CountDownLatch

([#25017](https://github.com/apache/pulsar/pull/25017)) \[improve]\[client]\[branch-4.0] Deduplicate in-progress lookup requests also for HttpLookupService

([#25469](https://github.com/apache/pulsar/pull/25469)) \[cleanup]\[ci] Remove documentation label bot

([#25470](https://github.com/apache/pulsar/pull/25470)) \[cleanup]\[ci] Remove ready-to-test label enforcement

### AoP

86fd755 test(aop): ignore closed multibundle cleanup

9657c29 fix(aop): update jetty 12 handler APIs

### MoP

a2657bde fix(mqtt): adapt additional servlet api

947864b9 fix checkstyle

ea1f8dba Fix mqtt client not have the consumer metric

### KoP

f30e46cde Bump pulsar and sn bom versions to 4.0.10.1 and fix assertion message for invalid topic name in SimpleLoadBalanceTest

Fix KsnRestServlet issue due to jetty upgrading

### StreamNative Pulsar Plugins

9f7638887 fix(servlet): adapt additional servlets to jetty 12

Add keyword to filter the result

372d9b481 Add permission/ACL audit logging

### pulsarctl

Fix flaky tests

### Function Mesh Worker Service

feat: add default metadata and add metadata to labels

e2b016f7 apply api changes introduced by [https://github.com/apache/pulsar/pull/25534](https://github.com/apache/pulsar/pull/25534)

feat: Support sandbox agent

feat: Support setting extraDependenciesDir via CustomRuntimeOptions

fix(auth): preserve service account when updateAuthData is false

### StreamNative Unified RBAC

8da9057 fix: support Pulsar 4.0.10.x servlet loading

## Security Fixes

### Apache Pulsar

([#25569](https://github.com/apache/pulsar/pull/25569)) \[fix]\[sec] Upgrade BouncyCastle to 1.84 (CVE-2026-5588, CVE-2026-0636)

([#25534](https://github.com/apache/pulsar/pull/25534)) \[fix]\[sec]\[branch-4.0] Upgrade to Jetty 12.1.8 to address several CVEs

([#25546](https://github.com/apache/pulsar/pull/25546)) \[fix]\[sec] Upgrade to async-http-client 2.14.5 to address CVE-2026-40490


# V4.0.10.2
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.10.2



# StreamNative Weekly Release Notes v4.0.10.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.10.2](https://github.com/streamnative/pulsar/releases/tag/v4.0.10.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.10.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.10.2/images/sha256-5604d46033399693e69d5b4f922309b08cd4f3abfa5179430c995be973141ecf)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.10.2/images/sha256-b29d0c77cfe0d12db4fdb37c0d7f32a6505698922c044e51fe57632515d19de0)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.10.2/images/sha256-b29d0c77cfe0d12db4fdb37c0d7f32a6505698922c044e51fe57632515d19de0)

## General Changes

### Apache Pulsar

([#25638)](https://github.com/apache/pulsar/pull/25638))) Revert "\[fix]\[test] Reduce flakiness in testLoadBalancerServiceUnitTableViewSyncer

([#25638](https://github.com/apache/pulsar/pull/25638)) \[fix]\[test] Reduce flakiness in testLoadBalancerServiceUnitTableViewSyncer

([#25752](https://github.com/apache/pulsar/pull/25752)) \[improve]\[misc] Upgrade Jetty to 12.1.9

([#25679](https://github.com/apache/pulsar/pull/25679)) \[fix]\[test] Fix flaky OneWayReplicatorDeduplicationTest.testDeduplication

([#25641](https://github.com/apache/pulsar/pull/25641)) \[fix]\[test] Make NamespacesTest.cleanupAfterMethod tolerant of transient infra failures

([#25640](https://github.com/apache/pulsar/pull/25640)) \[fix]\[test] Fix flaky testGetExcludedBookiesWithIsolationGroups

([#25581](https://github.com/apache/pulsar/pull/25581)) \[fix]\[broker] Decrement unacked counter when removeAllUpTo removes pending acks

([#25514](https://github.com/apache/pulsar/pull/25514)) \[fix]\[broker] Clean up orphan ledger on concurrent initial schema creation in BookkeeperSchemaStorage

([#25736](https://github.com/apache/pulsar/pull/25736)) \[fix]\[broker] Merge broker offload extra configurations

([#25681](https://github.com/apache/pulsar/pull/25681)) \[fix]\[broker] Correct two race conditions in the tracker code and logic bug in InMemoryDelayedDeliveryTracker that failed with NoSuchElementException

([#25684](https://github.com/apache/pulsar/pull/25684)) \[fix]\[broker] Skip backlog-quota eviction on fenced/closing topics

([#25730](https://github.com/apache/pulsar/pull/25730)) ([#25740](https://github.com/apache/pulsar/pull/25740)) \[fix]\[client] Make ClientBuilder serializable

([#25725](https://github.com/apache/pulsar/pull/25725)) \[fix]\[client]Broker-side producer handle leak if closes a producer which state is regitering schema

([#25583](https://github.com/apache/pulsar/pull/25583)) \[fix]\[broker]\[fix]\[broker]Replication stats is empty when the cluster is the target cluster of a one-way replication

([#25625](https://github.com/apache/pulsar/pull/25625)) \[fix]\[broker]Replication is stuck because failed to read entries

([#25644](https://github.com/apache/pulsar/pull/25644)) \[fix]\[broker] ConcurrentLongHashMap throw ArrayIndexOutOfBoundsException

([#25572](https://github.com/apache/pulsar/pull/25572)) \[fix]\[broker] Race condition causes perpetual backlog on internal topics

([#25578](https://github.com/apache/pulsar/pull/25578)) \[fix]\[client] Stabilize scaleReceiverQueueHint against concurrent enqueue/take

### AoP

6c929d3 test(aop): ignore closed multibundle cleanup

4c23aa6 fix(aop): update jetty 12 handler APIs

### KoP

add8acc2e Merge branch 'branch-4.0' into branch-4.0.10.2

ea3cf684b \[branch-4.0] Remove proxy from build.sh

b24b4a442 \[branch-4.0] remove dependencies from proxy

Fix fetch request duplicate key issue

Fix rdkafka requests might be forward to non-leader brokers and never recovered

Skip permission check for internal stats consumer

Stabilize generated swagger definitions

Fix possible message loss from idempotent producers during ledger rollover

Remove proxy module and related tests

### StreamNative Pulsar Plugins

c227b39f4 fix(rest): retry reader after consumer reconnect

10e99a94d chore(proxy): fix mock servlet import order

2976c8b1d fix(rest): handle jetty ee8 request redirects

Add rest v2 consume timeout to help return the current data

fix: upgrade Netty to 4.1.133.Final and exclude epoll native to fix CVEs in pulsar-metadata-tool and bookie-rackinfo

Fix to allow message id seek for non-partitioned topics

### pulsarctl

Bump go version to 1.25.10 to fix CVE

Add JDK path

### Function Mesh Worker Service

85f2b82a fix ci

feat: Support pin agent version when create session

feat: support filter based on metadata and use CRD for sessions

feat: Align with Anthropic api

fix(registry-service): mirror agent rename onto binding display-name

### StreamNative Unified RBAC

refactor: remove Pulsar servlet compatibility logic on main

63f1b65 Bump version to 1.13.2-rc10

41b673e Bump version to 1.13.2-rc9

b2f01df Bump version to 1.13.2-rc8

feat: add more agent related permissions

6dfb0ea Bump version to 1.13.2-rc7

c2fd5c0 Bump version to 1.13.2-rc6

\[codex] Fix maintenance notification action RBAC mappings

0ad4ea1 Bump version to 1.13.2-rc5

feat: add MaintenanceNotification permissions to unified RBAC

9c144cf Bump version to 1.13.2-rc4

ba6b888 Bump version to 1.13.2-rc3

feat: add test cases for workspaces and update sdk-apiserver

bde7d6e Bump version to 1.13.2-rc2

fix: update workspaces' api group to compute.streamnative.io

66bbb7a Bump version to 1.13.2-rc1

fix: support cloud integration with specified image

## Security Fixes

### Apache Pulsar

([#25744](https://github.com/apache/pulsar/pull/25744)) \[fix]\[sec] Upgrade thrift to 0.23.0 to address CVE-2026-43869

([#25745](https://github.com/apache/pulsar/pull/25745)) \[fix]\[sec] Upgrade vertx to 4.5.27 to address CVE-2026-6860

([#25737](https://github.com/apache/pulsar/pull/25737)) \[fix]\[sec] Upgrade vert.x to 4.5.25 to address CVE-2026-6860

([#25670](https://github.com/apache/pulsar/pull/25670)) \[fix]\[sec] Upgrade Netty to 4.1.133.Final to address CVEs


# V4.0.10.3
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.10.3



# StreamNative Weekly Release Notes v4.0.10.3

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.10.3](https://github.com/streamnative/pulsar/releases/tag/v4.0.10.3)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.10.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.10.3/images/sha256-9479a9fc08f1601dc4c2bd142c79dbb4e9f901ccd5c748ceb67548a6beeb5666)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.10.3/images/sha256-d5e94efe4bec30f7bd05683887f8d262f1ed9507d6b21cd03ca764c999e2c437)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.10.3/images/sha256-d5e94efe4bec30f7bd05683887f8d262f1ed9507d6b21cd03ca764c999e2c437)

## General Changes

### Apache Pulsar

([#25781](https://github.com/apache/pulsar/pull/25781)) \[fix]\[broker] Use effective offload policies for extra configs

([#25626](https://github.com/apache/pulsar/pull/25626)) \[improve]\[broker] optimize namespaceBundle validation to fix single-thread 100% CPU during unloading entire namespaces

([#25767](https://github.com/apache/pulsar/pull/25767)) \[improve]\[broker] Prevent stale replicator pending reads after termination

([#25790](https://github.com/apache/pulsar/pull/25790)) \[refactor]\[fn] Use Map instead of TreeMap for connector/function API types

([#25773](https://github.com/apache/pulsar/pull/25773)) \[improve]\[fn] make built-in connector reload incremental

([#25785](https://github.com/apache/pulsar/pull/25785)) \[improve]\[build] Upgrade org.apache.kerby:kerb-simplekdc from 1.1.1 to 2.1.1

([#25777](https://github.com/apache/pulsar/pull/25777)) \[fix]\[broker] Fix PulsarService.closeAsync where Condition.signalAll was called without holding a lock

([#25770](https://github.com/apache/pulsar/pull/25770)) \[fix]\[proxy] Close channel on connection failure

([#25759](https://github.com/apache/pulsar/pull/25759)) \[fix]\[client] Apply Avro logical type conversions when decoding schema without classloader

([#25624](https://github.com/apache/pulsar/pull/25624)) \[fix]\[broker] Close pending acks cleanup gap in BacklogQuotaManager

([#25592](https://github.com/apache/pulsar/pull/25592)) \[fix]\[broker] Move pending acks cleanup to selected mark-delete callbacks

([#25538](https://github.com/apache/pulsar/pull/25538)) \[improve]\[client] Implement tls\_client\_auth for AuthenticationOAuth2

([#25363](https://github.com/apache/pulsar/pull/25363)) \[improve]\[client] Enable configurable preemptive OAuth2 token refresh

([#25589](https://github.com/apache/pulsar/pull/25589)) \[fix]\[broker] Fix race in pending acks removal in redeliverUnacknowledgedMessages

([#25579](https://github.com/apache/pulsar/pull/25579)) \[fix]\[broker] Wait for orphan schema ledger cleanup before retry

([#25638](https://github.com/apache/pulsar/pull/25638)) \[fix]\[test] Reduce flakiness in testLoadBalancerServiceUnitTableViewSyncer

([#25596](https://github.com/apache/pulsar/pull/25596)) \[fix]\[test] Fix flaky ExtensibleLoadManagerImplTest.testLoadBalancerServiceUnitTableViewSyncer

### Function Mesh Worker Service

feat: update agents api to match Claude managed agent api

feat: align api updates from Claude

### StreamNative Ursa storage

Compatible with the avro converted java type

## Security Fixes

### Apache Pulsar

([#25818](https://github.com/apache/pulsar/pull/25818)) \[fix]\[sec] Bump org.asynchttpclient:async-http-client from 2.14.5 to 2.15.0

([#25788](https://github.com/apache/pulsar/pull/25788)) \[fix]\[sec]\[branch-4.0] Upgrade avro to 1.11.5 to address CVE-2025-33042


# V4.0.10.4
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.10.4



# StreamNative Weekly Release Notes v4.0.10.4

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.10.4](https://github.com/streamnative/pulsar/releases/tag/v4.0.10.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.10.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.10.4/images/sha256-43bdd81cf7036b3104e498262b8d865296e244decd9a0619e6f3326f2177fb15)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.10.4/images/sha256-9614990bd4bf609a11adf5f62fb55efe6e617c6525c6f5e0ea80a00448630ba9)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.10.4/images/sha256-9614990bd4bf609a11adf5f62fb55efe6e617c6525c6f5e0ea80a00448630ba9)

## General Changes

### Apache Pulsar

([#25805](https://github.com/apache/pulsar/pull/25805)) \[fix]\[client] Fix failed to close consumer because of the error: param memorySize is a negative value

([#25854](https://github.com/apache/pulsar/pull/25854)) \[improve]\[client] In cases where there is a risk of message loss, adjust the log level to error

([#25855](https://github.com/apache/pulsar/pull/25855)) \[improve]\[build] Remove kotlin-stdlib override; upgrade okhttp3 5.3.2 and okio 3.17.0

([#25852](https://github.com/apache/pulsar/pull/25852)) \[fix]\[test] Fix flaky ResendRequestTest.testSharedSingleAckedPartitionedTopic() test

([#25828](https://github.com/apache/pulsar/pull/25828)) \[fix]\[test] Add timeout to initial receives in ResendRequestTest.testSharedSingleAckedPartitionedTopic

([#25840](https://github.com/apache/pulsar/pull/25840)) \[fix]\[fn] Fix functions update issue where artifact is provided as a http url

([#25819](https://github.com/apache/pulsar/pull/25819)) \[improve]\[fn] Avoid gRPC timeout when getting status of a dead process runtime

([#25796](https://github.com/apache/pulsar/pull/25796)) \[fix]\[broker] Fix ManagedLedgerImpl.advanceCursorsIfNecessary() method may lose non-durable cursor properties in race condition

### KoP

feat: integrate group lag to Pulsar subscription's backlog in stats

Fix leader epoch capability advertisement and unknown epoch handling

### Function Mesh Worker Service

feat: remove "organization", "instance" label from metrics

5d14ead6 fix: fix AgentTriggerManager build error

feat: expose agent session metrics

Implement agent trigger

## Security Fixes


# V4.0.10.5
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.10.5



# StreamNative Weekly Release Notes v4.0.10.5

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.10.5](https://github.com/streamnative/pulsar/releases/tag/v4.0.10.5)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.10.5/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.10.5/images/sha256-51b6137d8740ee6546f4525f17f07b14d436fb35700c5849412845ae8ae1a516)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.10.5/images/sha256-fd48b31e2cc0e4b14d9b86592ffbf6b42d2c2b35e0c123ce5a5bd918f164e7e9)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.10.5/images/sha256-fd48b31e2cc0e4b14d9b86592ffbf6b42d2c2b35e0c123ce5a5bd918f164e7e9)

## General Changes

### Apache Pulsar

([#25707](https://github.com/apache/pulsar/pull/25707)) \[feat]\[broker] PIP-469: Legacy-aware topic policies backend routing and metadata-store topic policies

([#25547](https://github.com/apache/pulsar/pull/25547)) \[feat]\[pip] PIP-469: Legacy-aware topic policies backend routing and metadata-store topic policies

([#25943](https://github.com/apache/pulsar/pull/25943)) \[improve]\[misc] Upgrade Jetty to 12.1.10

([#25942](https://github.com/apache/pulsar/pull/25942)) \[fix]\[fn] Fix orphan exclusive producer on creation timeout in WorkerUtils.createExclusiveProducerWithRetry

([#25907](https://github.com/apache/pulsar/pull/25907)) \[fix]\[client] Preserve equals in FieldParser map values

([#25923](https://github.com/apache/pulsar/pull/25923)) \[improve]\[client] Clean up unacked message tracker when topics are removed in multi-topic consumers

([#25921](https://github.com/apache/pulsar/pull/25921)) \[fix]\[client] Match logical topic when removing unacked messages

([#25872](https://github.com/apache/pulsar/pull/25872)) \[improve]\[functions] Allow customizing Kubernetes service domain suffix in Function Worker

([#25924](https://github.com/apache/pulsar/pull/25924)) \[improve]\[misc] Upgrade vert.x to 4.5.28

([#25900](https://github.com/apache/pulsar/pull/25900)) \[fix]\[test] Stabilize testSecondaryIsolationGroupsBookiesNegative() test

([#25793](https://github.com/apache/pulsar/pull/25793)) \[improve]\[offload] Coalesce automatic offload triggers to reduce retry loops and ledger scans

([#25899](https://github.com/apache/pulsar/pull/25899)) \[fix]\[client] Prevent duplicate ServiceUrlProvider initialization

([#25919](https://github.com/apache/pulsar/pull/25919)) \[fix]\[proxy] Avoid intermittent 502 when admin proxy follows a broker redirect for a request with a body

([#25916](https://github.com/apache/pulsar/pull/25916)) \[fix]\[client] Clean up unacked messages when unsubscribing a topic with ack timeout backoff

([#25868](https://github.com/apache/pulsar/pull/25868)) \[improve]\[fn] make built-in functions reload incremental

([#25892](https://github.com/apache/pulsar/pull/25892)) \[fix]\[test] Fix flaky SameAuthParamsLookupAutoClusterFailoverTest.testAutoClusterFailover() test

([#25910](https://github.com/apache/pulsar/pull/25910)) \[fix]\[meta] Fix ZooKeeper session reconnect race condition in PulsarZooKeeperClient.clientCreator

([#25913](https://github.com/apache/pulsar/pull/25913)) \[fix]\[meta] Fix PulsarZooKeeperClient async addWatch callback retry behavior

([#23549](https://github.com/apache/pulsar/pull/23549)) \[improve]\[broker] PIP-380: Support-setting-up-specific-namespaces-to-skipping-the-load-shedding

([#25426](https://github.com/apache/pulsar/pull/25426)) \[fix]\[test] Fix flaky testMsgDropStat in NonPersistentTopicTest

([#25365](https://github.com/apache/pulsar/pull/25365)) \[fix]\[test] Fix flaky MessagePublishBufferThrottleTest.testBlockByPublishRateLimiting

([#25826](https://github.com/apache/pulsar/pull/25826)) \[fix]\[client] Reset higher-index states on recovery in SameAuthParamsLookupAutoClusterFailover

([#25388](https://github.com/apache/pulsar/pull/25388)) \[fix]\[client] Fix stale Healthy state in SameAuthParamsLookupAutoClusterFailover causing flaky test

([#25620](https://github.com/apache/pulsar/pull/25620)) \[fix]\[broker] Fix stuck chunks in SharedConsumerAssignor permit tracking

([#25594](https://github.com/apache/pulsar/pull/25594)) \[fix]\[broker] Fix precision loss in DataSketchesSummaryLogger by replacing LongAdder with DoubleAdder for sum accumulation

([#25525](https://github.com/apache/pulsar/pull/25525)) \[improve]\[client] Best-effort retry for individual/batch-index acks on send failure when ackReceiptEnabled=false

([#25817](https://github.com/apache/pulsar/pull/25817)) \[fix]\[broker] Fix non-batched null-value messages not removed during topic compaction

([#25825](https://github.com/apache/pulsar/pull/25825)) \[fix]\[bk] Fix NPE in IsolatedBookieEnsemblePlacementPolicy when policy class does not match

([#25803](https://github.com/apache/pulsar/pull/25803)) \[fix]\[broker] Fix PersistentMessageExpiryMonitor findEntryComplete() method may lose mark-delete properties in race condition

([#25865](https://github.com/apache/pulsar/pull/25865))  Return 400 for invalid reader messageId query parameter

([#25862](https://github.com/apache/pulsar/pull/25862)) \[fix]\[broker] Fix compaction cursor reset may lose mark-delete properties

([#25889](https://github.com/apache/pulsar/pull/25889)) \[fix]\[test] Fix flaky PulsarFunctionTlsTest.testFunctionsCreation() test

([#25864](https://github.com/apache/pulsar/pull/25864)) \[fix]\[test] Fix flaky ProducerCleanupTest timer cleanup

([#25867](https://github.com/apache/pulsar/pull/25867)) \[fix]\[fn] Fix Go function runtime to continue after user exceptions and add neg-ack tests

([#25870](https://github.com/apache/pulsar/pull/25870)) \[improve]\[misc] Upgrade Netty to 4.1.134

### MoP

f1bbc1e0 fix compile issue

Change error log to warn

Fix not release entry issue

### KoP

330aa061f \[branch-4.0] Fix ProtobufSchema incompatibility

Fix build failure after wire-schema-jvm is upgraded to 6.3.0

Ensure stats subscriptions (\_\_ksn\_xxx) are created even if the Pulsar subscription exists

\[fix] Improve error log for OauthValidatorCallbackHandler

Fix destination broker for NotOwnedBundleHandler

5b62396db Increase the unload time for NotOwnedBundleHandler

Initialize PID with carried producer ID

### StreamNative Pulsar Plugins

fix: patch metadata tool CVE dependencies on branch-4.0

### pulsarctl

Update Go toolchain to 1.25.11

### Function Mesh Worker Service

961c01fb fix build

feat: support Kafka functions in registry service

registry-service: Support agent trigger client pagination

feat: support orca managed agents

### StreamNative Ursa storage

fix cve check failed

## Security Fixes

### Apache Pulsar

([#25918](https://github.com/apache/pulsar/pull/25918)) \[fix]\[sec] Upgrade Netty to 4.1.135.Final to address several CVEs

([#25844](https://github.com/apache/pulsar/pull/25844)) \[fix]\[sec] Upgrade commons-configuration2 to 2.15.0 to address CVE-2026-45205


# V4.0.10.6
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.10.6



# StreamNative Weekly Release Notes v4.0.10.6

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.10.6](https://github.com/streamnative/pulsar/releases/tag/v4.0.10.6)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.10.6/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.10.6/images/sha256-56b7c869e0cc2fea450f7ea87666f5c0079c9a974323db61181d216949b894bc)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.10.6/images/sha256-3dc5b5ecc0d98c271ab493bae6a21aabb2ac40de38999d81b44051cc9ff3540b)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.10.6/images/sha256-3dc5b5ecc0d98c271ab493bae6a21aabb2ac40de38999d81b44051cc9ff3540b)

## General Changes

### Apache Pulsar

([#25915](https://github.com/apache/pulsar/pull/25915)) \[fix]\[broker]Do not trigger topic GC if replication is still active

([#26079](https://github.com/apache/pulsar/pull/26079)) \[feat]\[broker] Expose managed ledger properties via topic internal stats

([#26075](https://github.com/apache/pulsar/pull/26075)) \[fix]\[broker] Avoid attaching a consumer to a migrated non-persistent topic on subscribe

([#26065](https://github.com/apache/pulsar/pull/26065)) \[fix]\[meta] Run ledger-underreplication notification callbacks off the metadata-store listener thread

([#26064](https://github.com/apache/pulsar/pull/26064)) \[fix]\[client] Run the failover health probe off the Netty event-loop thread

([#25675](https://github.com/apache/pulsar/pull/25675)) \[fix]\[test] Make SameAuthParamsLookupAutoClusterFailoverTest less timing-sensitive

([#26038](https://github.com/apache/pulsar/pull/26038)) \[improve]\[test]Add test: test/testTopicPartitionCannotBeCreatedAfterTopicDeleted

([#26002](https://github.com/apache/pulsar/pull/26002)) \[fix]\[broker] Fix geo-replication stuck after a failed publish to the remote cluster

([#26059](https://github.com/apache/pulsar/pull/26059)) \[fix] functions: Run worker leader-election off the consumer event-listener thread

([#26054](https://github.com/apache/pulsar/pull/26054)) \[fix]\[broker] Avoid blocking the bundle-throughput lookup on per-bundle metadata reads

([#26053](https://github.com/apache/pulsar/pull/26053)) \[fix]\[broker] Avoid blocking the dispatcher close path on delayed-delivery tracker close

([#26052](https://github.com/apache/pulsar/pull/26052)) \[fix]\[proxy] Avoid blocking the proxy IO thread on a cold broker cache

([#26051](https://github.com/apache/pulsar/pull/26051)) \[fix]\[broker] Avoid blocking metadata read on the IO thread when redirecting migrated producers/consumers

([#26044](https://github.com/apache/pulsar/pull/26044)) \[fix]\[broker] Prevent topic policy initialization race with a buffering listener wrapper

([#26049](https://github.com/apache/pulsar/pull/26049)) \[fix]\[test] Fix flaky testPrepareInitPoliciesCacheAsyncThrowExceptionAfterCreateReader

([#26040](https://github.com/apache/pulsar/pull/26040)) \[fix]\[broker] Run the message expiry check off the topic policy update path

([#26042](https://github.com/apache/pulsar/pull/26042)) \[fix]\[broker] Run topic policy notifications on the topic-ordered executor

([#26046](https://github.com/apache/pulsar/pull/26046)) \[fix]\[fn] Make exclusiveLeaderProducer volatile in FunctionMetaDataManager

([#26033](https://github.com/apache/pulsar/pull/26033)) \[improve]\[fn] Upgrade pulsar-client-python to 3.12.0

([#26031](https://github.com/apache/pulsar/pull/26031)) \[fix]\[broker] Fail fast for load balancer misconfigurations instead of falling back to SimpleLoadManagerImpl

([#26025](https://github.com/apache/pulsar/pull/26025)) \[fix]\[broker] Don't let a stuck or aborted topic policies cache init make a namespace's topics unloadable

([#26026](https://github.com/apache/pulsar/pull/26026)) \[fix]\[broker] Fix forced topic/namespace deletion still hanging when the compaction reader reconnect stalls

([#26016](https://github.com/apache/pulsar/pull/26016)) \[fix]\[broker] Fix forced topic/namespace deletion hanging or failing when compaction is in progress

([#26015](https://github.com/apache/pulsar/pull/26015)) \[fix]\[broker] Prevent subscribe rate limit from stalling compaction and blocking forced deletion

([#26000](https://github.com/apache/pulsar/pull/26000)) \[fix]\[meta] Keep the leader value in the election cycle and make leader reads authoritative

([#25998](https://github.com/apache/pulsar/pull/25998)) \[fix]\[broker] Fix compacted read could be stuck forever or message loss due to cursor mark delete

([#25963](https://github.com/apache/pulsar/pull/25963)) \[improve]\[misc] Upgrade Apache Commons libraries and Apache Http components

([#25974](https://github.com/apache/pulsar/pull/25974)) \[fix]\[test] Deflake TopicPoliciesTest.setupTestTopic by retrying forced namespace deletion

([#25977](https://github.com/apache/pulsar/pull/25977)) \[fix]\[test] Fix flaky ExtensibleLoadManagerImplTest.initializeState by recovering wedged channel ownership

([#25976](https://github.com/apache/pulsar/pull/25976)) \[fix]\[test] Fix flaky ExtensibleLoadManagerImplTest by re-serving the channel topic in initializeState

([#25946](https://github.com/apache/pulsar/pull/25946)) \[fix]\[broker] Fix tableview divergence in ServiceUnitStateTableViewSyncer causing flaky tests

### KoP

\[fix] Improve NotOwnedBundleHandler logic

Fix inflight reads limiter permits leak when handleEntries fails

Fix the internal Kafka client could be configured with TLS but no certificates

### StreamNative Pulsar Plugins

fix(package-storage-cloud): set numeric GCS fs config for Hadoop 3.5 compatibility

405aa78dd fix(ci): use go 1.26.4 for plugin release builds

fix(snoidc-sdk-go): upgrade golangci-lint to v2 for Go 1.26 compatibility

46cccc374 Remove unused OpenTelemetry test dependency from pulsar-rest tests

### pulsarctl

feat(cmd): add custom runtime options injection

### Function Mesh Worker Service

e1fed5f2 fix cherry-pick error

feat: support as service account runtime auth

fix: fix AgentTrigger restart and support list AgentTrigger by agent id

worker-service: Support Kubernetes 1.35 pod status

a1c93d5c feat: upgrade pulsar to 4.0.10.5

### StreamNative Ursa storage

fix cve check failed

## Security Fixes

### Apache Pulsar

([#26068](https://github.com/apache/pulsar/pull/26068)) \[fix]\[sec] Upgrade jline to 4.2.1 and picocli to 4.7.7, drop unused jline2

### StreamNative Pulsar Plugins

\[fix]\[sec] Update branch-4.0 plugin CVE dependencies


# V4.0.10.7
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.10.7



# StreamNative Weekly Release Notes v4.0.10.7 (stable)

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.10.7](https://github.com/streamnative/pulsar/releases/tag/v4.0.10.7)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.10.7/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.10.7/images/sha256-3d950329f9398616c51711f736babb7be6270f70722662fc5c1f78b906f48117)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.10.7/images/sha256-09680ac45c4dbcc35e429c97876a3f45b5fdeaed89c294b86da5486c053f6f90)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.10.7/images/sha256-09680ac45c4dbcc35e429c97876a3f45b5fdeaed89c294b86da5486c053f6f90)

## General Changes

### Apache Pulsar

([#26132](https://github.com/apache/pulsar/pull/26132)) \[fix]\[broker] Don't let a closing topic-policies reader abort a concurrent cache-init reload

([#26123](https://github.com/apache/pulsar/pull/26123)) \[fix]\[test] Run makeReadEntryProbFail's errorOrNot on a caller-provided executor

([#26083](https://github.com/apache/pulsar/pull/26083)) \[fix]\[test] Fix flaky PersistentTopicsTest setup caused by concurrent Mockito stubbing

([#25645](https://github.com/apache/pulsar/pull/25645)) \[fix]\[test] Fix flaky SchemaServiceTest.testSchemaRegistryMetrics

([#26122](https://github.com/apache/pulsar/pull/26122)) \[fix]\[test] Fix flaky AuditorBookieTest.testBookieClusterRestart

([#26106](https://github.com/apache/pulsar/pull/26106)) \[fix]\[broker] Fix replication stall when a cursor rewind skips an in-flight read

([#26038)](https://github.com/apache/pulsar/pull/26038))) Revert "\[improve]\[test]Add test: test/testTopicPartitionCannotBeCreatedAfterTopicDeleted

([#26110](https://github.com/apache/pulsar/pull/26110)) \[fix]\[broker] Forward topic policy updates after init failures

([#26005](https://github.com/apache/pulsar/pull/26005)) \[fix]\[broker] Fix replicator getting stuck under rate limiter throttling and honor readBatchSize/maxReadSizeBytes on the default read path

([#26055](https://github.com/apache/pulsar/pull/26055)) \[improve]\[broker] Improve dispatch performance by summing entry bytes with a loop

([#26080](https://github.com/apache/pulsar/pull/26080)) \[fix]\[broker] Guard BucketDelayedDeliveryTracker.nextDeliveryTime against empty queues

([#25984](https://github.com/apache/pulsar/pull/25984)) \[improve]\[broker] Trim orphaned bucket snapshots when ledgers are deleted

### MoP

8dc19588 test: isolate mock ZooKeeper sessions per broker

### KoP

Support handling non-partitioned topics for DescribeTopicPartitions and configs

\[SchemaRegistry] Add `ALWAYS_INCOMPATIBLE` compatibility mode

\[feature] Refactor JSON schema compatibility checker

\[branch-4.0] Use sn-bom to manage all dependency versions

Improve earliest offset query performance by recording start offset in managed ledger properties

\[fix] lock-ordering deadlock between partitionLock and group lock in storeOffsetMessageAsync

### StreamNative Pulsar Plugins

7575e1614 Upgrade the bk version

fix: upgrade golang.org/x/net to v0.55.0 to fix multiple CVEs

### pulsarctl

feat: add set-replication-clusters command for topics

### Function Mesh Worker Service

feat: use agent id for agent trigger and support agent version

Support cron agent triggers in worker service

feat: Align managed agent vault API validation

\[codex] Support Claude memory and file APIs

registry-service: Split agent session token metrics

worker-service: Support JWT auth for Kafka Connect

## Security Fixes

### Apache Pulsar

([#26098](https://github.com/apache/pulsar/pull/26098)) \[fix]\[sec]\[branch-4.0] Upgrade Jackson version to 2.18.8


# V4.0.10.8
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.10.8



# StreamNative Weekly Release Notes v4.0.10.8

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.10.8](https://github.com/streamnative/pulsar/releases/tag/v4.0.10.8)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.10.8/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.10.8/images/sha256-6a1d1c36322578089845a12922b39eb6ee54d36d09c6ee49d4e8a4ce94b954a9)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.10.8/images/sha256-ebdab06b422e2144079ad12703fd9b8aa45e6962750c4259063ed16074bf7a67)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.10.8/images/sha256-ebdab06b422e2144079ad12703fd9b8aa45e6962750c4259063ed16074bf7a67)

## General Changes

### Apache Pulsar

([#26158](https://github.com/apache/pulsar/pull/26158)) \[fix]\[metadata] Fix orphaned UR parent nodes not cleaned up with Oxia metadata backend

([#26136](https://github.com/apache/pulsar/pull/26136)) \[fix]\[fn] Reorder Function Worker shutdown to stop scheduler before runtime manager

([#26139](https://github.com/apache/pulsar/pull/26139)) \[improve]\[fn] Upgrade pulsar-client-python to 3.13.0

([#26134](https://github.com/apache/pulsar/pull/26134)) \[improve]\[broker] Load topic policies on non-persistent topic load and gate the policy replay

### KoP

Fix OffsetFetch duplicate partition handling

Implement DeleteRecords request correctly

Implement lazy recovery for producer state management for classic engine

Skip Pulsar message deduplication snapshot for Kafka topics

### Function Mesh Worker Service

fix(registry): add 'type' field to EnvironmentPackages

03298465 fix cherry-pick error

feat: test registry agents api with orca managed agent

registry-service: Type environment config unions

feat: support session-local agent tool/mcp\_servers overrides on session update

feat: Align managed agent endpoints

## Security Fixes

### Apache Pulsar

([#26142](https://github.com/apache/pulsar/pull/26142)) \[fix]\[sec] Bump github.com/go-jose/go-jose/v4 from 4.1.3 to 4.1.4 in /pulsar-function-go

([#26140](https://github.com/apache/pulsar/pull/26140)) \[fix]\[sec] Upgrade pulsar-client-go to v0.20.0 in pulsar-function-go, also address CVEs


# V4.0.10.9
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.10.9



# StreamNative Weekly Release Notes v4.0.10.9

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.10.9](https://github.com/streamnative/pulsar/releases/tag/v4.0.10.9)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.10.9/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.10.9/images/sha256-f74967c433b80ec7e2451482a468081a010cb429740f2a159de1063026bc874a)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.10.9/images/sha256-f4ec6603f1a1e7843dd64e88d883a23ae9d6b161943b22c8a18f3ea913367e19)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.10.9/images/sha256-f4ec6603f1a1e7843dd64e88d883a23ae9d6b161943b22c8a18f3ea913367e19)

## General Changes

### Apache Pulsar

([#26184](https://github.com/apache/pulsar/pull/26184)) \[fix]\[broker] Fix `getEstimatedSizeSinceMarkDeletePosition` throw `IllegalArgumentException`

([#26196](https://github.com/apache/pulsar/pull/26196)) \[fix]\[client] Preserve null values in pulsar-admin schema output

([#26165](https://github.com/apache/pulsar/pull/26165)) \[improve]\[monitor]\[branch-4.0] Upgrade OpenTelemetry libraries

### AoP

\[fix] Fix flaky test MultiBundlesTest

### MoP

6226f39f Fix maven cache issue

### KoP

Skip metadata lookup for topics with errors in async lookup

Fix pending txn offsets never removed due to deleted TXN markers after compaction

### StreamNative Pulsar Plugins

Fix OIDC auth metrics recording and add pool match warning log

### pulsarctl

Upgrade go version to avoid cve

### Function Mesh Worker Service

registry-service: Expose session cache token metrics

ci: Seed managed-agents API key fingerprint

0c5ba45d fix ci

ci: Support managed-agent sandbox isolation

fix: remove debug logs

registry-service: Add snServiceAccount registry config mapping

feat(agents): add agent binding update functionality

ci: Add agent trigger e2e coverage

registry-service: Support session-scoped files

### StreamNative Ursa storage

adc5b3309 Align OpenTelemetry dependencies with 1.62.0

## Security Fixes

### Apache Pulsar

([#26195](https://github.com/apache/pulsar/pull/26195)) \[fix]\[sec]\[branch-4.0] Upgrade Hadoop to 3.5.0

([#26187](https://github.com/apache/pulsar/pull/26187)) \[fix]\[sec]\[branch-4.0] Upgrade Jackson version to 2.18.9

([#26170](https://github.com/apache/pulsar/pull/26170)) \[fix]\[sec]\[branch-4.0] Upgrade Netty to 4.1.136.Final


# V4.0.11.1
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.11.1



# StreamNative Weekly Release Notes v4.0.11.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.11.1](https://github.com/streamnative/pulsar/releases/tag/v4.0.11.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.11.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.11.1/images/sha256-49af0ea229f9c5cd6b133418cb3d9fb5bbc94a432c337421c20d21399aaa7bc2)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.11.1/images/sha256-64b5fd11b2a0c701c8c4f8d91812ba5cf5c54443463928dbb84c090a94af76a5)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.11.1/images/sha256-64b5fd11b2a0c701c8c4f8d91812ba5cf5c54443463928dbb84c090a94af76a5)

## General Changes

### Apache Pulsar

([#26248](https://github.com/apache/pulsar/pull/26248)) \[fix]\[meta]\[branch-4.0] Tolerate concurrent creation of the underreplication LAYOUT node

([#26089](https://github.com/apache/pulsar/pull/26089)) \[fix]\[broker] Release entry on GetLastMessageId when parseMessageMetadata throws

([#26146](https://github.com/apache/pulsar/pull/26146)) \[fix]\[broker] Prevent stale service unit callbacks from dropping active lookup and cleanup jobs

([#26243](https://github.com/apache/pulsar/pull/26243)) \[fix]\[broker] Fix TableViewLoadDataStoreImpl close deadlock that stalls broker shutdown

([#26001](https://github.com/apache/pulsar/pull/26001)) \[fix]\[client] Fix unAckedMessageTracker cleanup on multi-topics batch ack

([#26191](https://github.com/apache/pulsar/pull/26191)) \[fix]\[fn] Forward source message properties in Python runtime

([#26237](https://github.com/apache/pulsar/pull/26237)) \[fix]\[broker] Fix silently dropped acknowledgement failures in PulsarMetadataEventSynchronizer

([#26240](https://github.com/apache/pulsar/pull/26240)) \[fix]\[broker] Fix delayed message index data loss when trimming overlapping bucket snapshots

([#26245](https://github.com/apache/pulsar/pull/26245)) \[fix]\[broker] Fix incorrect listener URLs returned by ModularLoadManager lookups

([#26119](https://github.com/apache/pulsar/pull/26119)) \[fix]\[broker] Prevent completing replicated snapshot before marker publish

([#26145](https://github.com/apache/pulsar/pull/26145)) \[fix]\[broker] Prevent stale topic unload cleanup from removing active cache entries

([#26236](https://github.com/apache/pulsar/pull/26236)) \[fix]\[broker] Fix Key\_Shared delivery stall when look-ahead triggers at the end of the topic

([#26228](https://github.com/apache/pulsar/pull/26228)) \[fix]\[ml] Preserve ledger entries/size when transformLedgerInfo callback completes after a concurrent close

([#26174](https://github.com/apache/pulsar/pull/26174)) \[fix]\[broker] Prevent stale read completions from stranding Failover subscriptions

([#26199](https://github.com/apache/pulsar/pull/26199)) \[fix]\[meta] Complete handleMetadataEvent future exceptionally when the initial get fails

([#26201](https://github.com/apache/pulsar/pull/26201)) \[fix]\[meta] Record get op stats on the correct completion branch in AbstractMetadataStore

([#26218](https://github.com/apache/pulsar/pull/26218)) \[fix]\[meta] Fix RocksdbMetadataStore instanceId not advancing across restarts

([#26232](https://github.com/apache/pulsar/pull/26232)) \[improve]\[offload] Support credentials from offload policies for S3 and Aliyun OSS drivers

([#26233](https://github.com/apache/pulsar/pull/26233)) \[improve]\[misc] Upgrade Jetty to 12.1.11

([#26234](https://github.com/apache/pulsar/pull/26234)) \[fix]\[broker] Trigger max read position callback for messages published during transaction buffer recovery

([#26230](https://github.com/apache/pulsar/pull/26230)) \[fix]\[broker] Check deliverAt before containsMessage in bucket addMessage

([#26225](https://github.com/apache/pulsar/pull/26225)) \[improve]\[build] Upgrade docker base image Alpine to 3.24

([#25180](https://github.com/apache/pulsar/pull/25180)) \[improve]\[misc] Upgrade to Alpine 3.23

([#26226](https://github.com/apache/pulsar/pull/26226)) \[improve]\[build] Upgrade slog to 0.10.0

([#26163](https://github.com/apache/pulsar/pull/26163)) ([#26224](https://github.com/apache/pulsar/pull/26224)) \[improve]\[broker]\[branch-4.0] Trace the asynchronous tasks in logs when loading topics

([#26198](https://github.com/apache/pulsar/pull/26198)) \[fix]\[test] Fix flaky test `testCompactionPriority `

([#26223](https://github.com/apache/pulsar/pull/26223)) \[fix]\[broker]\[branch-4.2] Fix admin API HTTP 400 FAIL\_ON\_TRAILING\_TOKENS when a broker interceptor is loaded

([#26217](https://github.com/apache/pulsar/pull/26217)) \[fix]\[fn] Return inputSpecs consumerProperties in function GET info

([#26200](https://github.com/apache/pulsar/pull/26200)) \[fix]\[meta] Fix NPE in shouldIgnoreEvent when MetadataEvent options is null

([#26179](https://github.com/apache/pulsar/pull/26179)) \[fix]\[broker] Prevent partition expansion from inheriting delayed-delivery bucket state

([#26149](https://github.com/apache/pulsar/pull/26149)) \[improve]\[broker] Skip system cursor when check inactive cursor.

([#26143](https://github.com/apache/pulsar/pull/26143)) \[fix]\[client] Fix lookup permit double-release, waiting queue starvation and timeout-response races in ClientCnx

([#26193](https://github.com/apache/pulsar/pull/26193)) \[improve]\[meta] Upgrade Oxia client to 0.9.4

([#25964](https://github.com/apache/pulsar/pull/25964)) \[improve]\[meta] Upgrade Oxia client to 0.8.0

([#26171](https://github.com/apache/pulsar/pull/26171)) \[fix]\[broker] Fix bucket delayed message index metrics reset on scrape

([#26159](https://github.com/apache/pulsar/pull/26159)) \[fix]\[broker] Read subscription properties directly from cursor

([#26043](https://github.com/apache/pulsar/pull/26043)) \[fix]\[client] Fix UnAckedMessageRedeliveryTracker to skip cancelled timeouts

([#26135](https://github.com/apache/pulsar/pull/26135)) \[fix]\[client] Sync ackSet in client with broker to stop acked messages reaching the DLQ

([#26169](https://github.com/apache/pulsar/pull/26169)) \[fix]\[ci] Upgrade sandboxed-trivy-action to approved sha

([#25480](https://github.com/apache/pulsar/pull/25480)) \[improve]\[ci] Replace trivy-action with sandboxed-trivy-action

([#25038](https://github.com/apache/pulsar/pull/25038)) \[fix]\[client] Fix lookup request semaphore not release problem

### KoP

\[branch-4.0] Fix Oxia range scan consumer compatibility

### StreamNative Ursa storage

Trigger Claude code review by label

Update the dependency version to fix the build issue

## Security Fixes


# V4.0.11.2
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.11.2



# StreamNative Weekly Release Notes v4.0.11.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.11.2](https://github.com/streamnative/pulsar/releases/tag/v4.0.11.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.11.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.11.2/images/sha256-496c429c8cbea4457ce842a9b81a059fa6b03077a99e91e6965c3240784f1dbc)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.11.2/images/sha256-fb9ba9b22acacc66c9582af5f650eff2d145dc849c1d65f398a88c1c188e84cc)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.11.2/images/sha256-fb9ba9b22acacc66c9582af5f650eff2d145dc849c1d65f398a88c1c188e84cc)

## General Changes

### Apache Pulsar

([#26242](https://github.com/apache/pulsar/pull/26242)) \[fix]\[broker] Fix delayed-delivery bucket merge failures when delayedDeliveryMaxNumBuckets is 1-3

([#26247](https://github.com/apache/pulsar/pull/26247)) \[fix]\[ml] Tolerate concurrent creation of the managed ledger z-node

([#26219](https://github.com/apache/pulsar/pull/26219)) \[improve]\[broker]\[branch-4.2] Upgrade bookkeeper to 4.17.4

([#25857](https://github.com/apache/pulsar/pull/25857)) \[fix]\[client] Avoid exception in ConsumerImpl hasMessageAvailable before first receive

([#26203](https://github.com/apache/pulsar/pull/26203)) \[fix]\[broker] Log exception in PulsarMetadataEventSynchronizer failure path

([#26121](https://github.com/apache/pulsar/pull/26121)) \[improve]\[fn] Standardize log4j2 Root logger configuration to use system property

### KoP

\[fix] Remove the NotOwnedBundleHandler

Introduce size and time based snapshottable metadata for producer state

### StreamNative Pulsar Plugins

Upgrade Pulsar and Oxia schema store dependencies

Add new produce method for Pulsar-Rest

### Cloud Pulsar Plugins

Check auth provider name in the authorization stage

### Function Mesh Worker Service

registry-service: handle missing connector config definitions

registry-service: Align list pagination with Claude

Align with orca-managed-agents

registry-service: Support PostgreSQL managed-agent storage

## Security Fixes

### Apache Pulsar

([#26250](https://github.com/apache/pulsar/pull/26250)) \[fix]\[sec] Upgrade lz4-java to 1.11.1 to address CVE-2026-59949

([#26235](https://github.com/apache/pulsar/pull/26235)) \[fix]\[sec] Upgrade grpc in pulsar-function-go to 1.82.1 to fix GHSA-hrxh-6v49-42gf

([#26231](https://github.com/apache/pulsar/pull/26231)) \[fix]\[sec] Bump google.golang.org/grpc from 1.79.3 to 1.82.1 in /pulsar-function-go/examples

([#26270](https://github.com/apache/pulsar/pull/26270)) \[fix]\[sec]\[branch-4.2] Upgrade Spring to 7.0.8


# V4.0.12.1
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.12.1



# StreamNative Weekly Release Notes v4.0.12.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.12.1](https://github.com/streamnative/pulsar/releases/tag/v4.0.12.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.12.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.12.1/images/sha256-0db82521cb17abd8e4ee0a69d64e14d5fa610d4d58bbac75541dbcb1b5a2072c)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.12.1/images/sha256-69b4d5bcb2a36eb5b74074c5ca9f54f196e4932e2fb449043aa2dab91a9fbb81)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.12.1/images/sha256-69b4d5bcb2a36eb5b74074c5ca9f54f196e4932e2fb449043aa2dab91a9fbb81)

## General Changes

### KoP

Introduce size and time based snapshottable metadata for producer state

### StreamNative Pulsar Plugins

Upgrade Pulsar and Oxia schema store dependencies

Add new produce method for Pulsar-Rest

### Cloud Pulsar Plugins

Check auth provider name in the authorization stage

### Function Mesh Worker Service

Align with orca-managed-agents

registry-service: Support PostgreSQL managed-agent storage

## Security Fixes


# V4.0.9.4
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.9.4



# StreamNative Weekly Release Notes v4.0.9.4

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.9.4](https://github.com/streamnative/pulsar/releases/tag/v4.0.9.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.9.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.9.4/images/sha256-4dc52e3ec35e189e177eccbe89e74e55ecd45d3f74db748aae12422ad16ca78c)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.9.4/images/sha256-168ffb5b4c6e468311036a8ebd2fa78c3173f2082e1ba711974b38fe32072f56)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.9.4/images/sha256-168ffb5b4c6e468311036a8ebd2fa78c3173f2082e1ba711974b38fe32072f56)

## General Changes

### Apache Pulsar

([#25371](https://github.com/apache/pulsar/pull/25371)) \[fix]\[broker] Fix IllegalArgumentException in BucketDelayedDeliveryTracker.addMessage

([#25312](https://github.com/apache/pulsar/pull/25312)) \[fix]\[broker]system topic was created with different partitions acrossing clusters after enabled namespace-level replication

([#25266](https://github.com/apache/pulsar/pull/25266)) \[fix]\[broker] Handle missing replicator during snapshot request processing

([#25325](https://github.com/apache/pulsar/pull/25325)) \[fix]\[io]\[kca] kafka headers silently dropped

([#25317](https://github.com/apache/pulsar/pull/25317)) \[fix]\[client] Fail messages immediately in ProducerImpl when in terminal state

([#25316](https://github.com/apache/pulsar/pull/25316)) \[fix] Fix flaky OneWayReplicatorTest.testTopicPoliciesReplicationRule

([#25314](https://github.com/apache/pulsar/pull/25314)) \[fix]\[test] Fix flaky PulsarDebeziumOracleSourceTest

([#25346](https://github.com/apache/pulsar/pull/25346)) \[fix]\[broker] Fix concurrency bug in BucketDelayedDeliveryTracker

([#25296](https://github.com/apache/pulsar/pull/25296)) \[fix]\[offload] Close all resources in BlobStoreBackedReadHandleImplV2.closeAsync

([#25276](https://github.com/apache/pulsar/pull/25276)) \[fix]\[broker] Support namespace unsubscribe when bundles are unloaded

### StreamNative Pulsar Plugins

Use `CLOUDSTORAGE_S3_BUCKET` for AWS cloud storage tests

Remove deprecated checkCluster method using TopicName.getCluster()

\[feature] A new detector for loading topics

\[fix]\[audit-log] Log ProducerQueueIsFullError at INFO level to prevent log flooding

fix: override log4j to 2.25.3 for metadata tool

### pulsarctl

test: tolerate namespace not found errors

bump go version to 1.25.8 to fix CVE-2026-25679 and CVE-2026-27142

feat(topic): add subscription dispatch rate commands

### StreamNative Unified RBAC

d97b1f4 Bump version to 1.9.3

fix ci

02741f6 Bump version to 1.9.2

feat: Add workspace related permissions

a635ede Bump version to 1.9.1

631693d Bump version to 1.9.0

f4f7515 Bump version to 1.8.4

ad74270 Bump version to 1.8.3

bb800e8 Bump version to 1.8.2

220f10b Bump version to 1.8.1

9dd9a09 Bump version to 1.8.0

feat: upgrade sdk-go version to v0.15.0

feat: support instances permissions mapping

00bccf8 Bump version to 1.7.4

fixes the schedule release workflow

2034f5d fixes: configure maven

5ec5eb2 fixes: use the pulsar image directly

d863187 fixes: fix the wrong image

feat: support features and featuregates permissions mapping

## Security Fixes

### Apache Pulsar

([#25303](https://github.com/apache/pulsar/pull/25303)) \[fix]\[sec] Bump org.apache.zookeeper:zookeeper from 3.9.4 to 3.9.5


# V4.0.9.5
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.9.5



# StreamNative Weekly Release Notes v4.0.9.5

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.9.5](https://github.com/streamnative/pulsar/releases/tag/v4.0.9.5)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.9.5/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.9.5/images/sha256-55c6518c4668fb8d1190e1e4140d7bbc5f2407bd4dbad6ca014c369bb8484b53)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.9.5/images/sha256-2a7b7a126e418052d3d47fa36c6d1b4a5e83deb3c216041e5cf3646fc1903e87)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.9.5/images/sha256-2a7b7a126e418052d3d47fa36c6d1b4a5e83deb3c216041e5cf3646fc1903e87)

## General Changes

### Apache Pulsar

([#25371](https://github.com/apache/pulsar/pull/25371)) \[fix]\[broker] Fix IllegalArgumentException in BucketDelayedDeliveryTracker.addMessage

([#25312](https://github.com/apache/pulsar/pull/25312)) \[fix]\[broker]system topic was created with different partitions acrossing clusters after enabled namespace-level replication

([#25266](https://github.com/apache/pulsar/pull/25266)) \[fix]\[broker] Handle missing replicator during snapshot request processing

([#25325](https://github.com/apache/pulsar/pull/25325)) \[fix]\[io]\[kca] kafka headers silently dropped

([#25317](https://github.com/apache/pulsar/pull/25317)) \[fix]\[client] Fail messages immediately in ProducerImpl when in terminal state

([#25316](https://github.com/apache/pulsar/pull/25316)) \[fix] Fix flaky OneWayReplicatorTest.testTopicPoliciesReplicationRule

([#25314](https://github.com/apache/pulsar/pull/25314)) \[fix]\[test] Fix flaky PulsarDebeziumOracleSourceTest

([#25346](https://github.com/apache/pulsar/pull/25346)) \[fix]\[broker] Fix concurrency bug in BucketDelayedDeliveryTracker

### KoP

([#1838)](https://github.com/streamnative/ksn/pull/1838))) Revert "Improve performance for finding position by offset

Improve performance for finding position by offset

### StreamNative Pulsar Plugins

90be1b311 fix(build): manage zookeeper version in parent pom

Fix Vault testcontainer compatibility in integration tests

fix: exclude stream-storage-server from pulsar-metadata-tool

fix: upgrade spring-beans to 6.2.12 and spring-ldap-core to 3.2.16 to fix CVEs in broker-auth-ldap

fe47ab3f5 Update bom version to 4.0.0-SNAPSHOT, the 4.0.0-SNAPSHOT will keep update when bom branch-4.0 have any change

Remove zookeeper version defiine

0d3d7c518 Upgrade bookie-rackinfo netty version

Fix vault docker version issue

### pulsarctl

Update Trivy GitHub Action to v0.35.0

Gate Docker login and snstage image usage on streamnativebot actor

Add platform and compute teams as CODEOWNERS

### Function Mesh Worker Service

df8505ee fix: use 4.0.9.4 base image in CI

Support unified RBAC for registry service

feat: expose clusterRef field to ConnectionConfig

fix: do not set static bootstrapServers when in registry mode

fix: fix build error

feat: add a new endpoint to validate connection

### StreamNative Unified RBAC

feat: add workspace packages permissions

b963039 Bump version to 1.11.2

89d0215 Bump version to 1.11.1

fix(deps): make caffeine compileOnly to avoid BOM convergence conflict

aac4247 Bump version to 1.11.0

fix(build): use afterEvaluate to resolve Maven artifactId from archivesName

740835b Bump version to 1.10.2

afa1350 Bump version to 1.10.1

fix(build): set Maven artifactId from archivesName

f7f6131 Bump version to 1.10.0

perf(authz-provider): add allowExtraOperationAsync result cache

ci: add workflow to sync release branches from main

docs: add comprehensive project documentation

build: remove legacy Maven pom.xml files

docs: add comprehensive project documentation

perf(enforcer): avoid thread switching with thenCompose/thenApply

build(management-servlet): conditional compilation for Pulsar 4.0.x/4.2.x compat

refactor(authz-provider): remove jjwt dependency, decode JWT payload directly

style: apply google java format via spotless

build: migrate Java build from Maven to Gradle

fix(authz-provider): pass org/instance/cluster condition in isSuperUser

ed0031c Bump version to 1.9.4

fix(sdk-go-oxia): merge role bindings on apply instead of overwriting

e683943 Bump version to 1.9.3

fix ci

367bae4 Bump version to 1.9.2

feat: Add workspace related permissions

10ca8ca Bump version to 1.9.1

16ac8ee Bump version to 1.9.0

9bc931e Bump version to 1.8.4

0b14a22 Bump version to 1.8.3

83a2430 Bump version to 1.8.2

26e32f4 Bump version to 1.8.1

e4f2838 Bump version to 1.8.0

feat: upgrade sdk-go version to v0.15.0

feat: support instances permissions mapping

b245b96 Bump version to 1.7.4

fixes the schedule release workflow

fdf637a fixes: configure maven

f0a9bcd fixes: use the pulsar image directly

597e9e4 fixes: fix the wrong image

feat: support features and featuregates permissions mapping

## Security Fixes


# V4.0.9.6
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.9.6



# StreamNative Weekly Release Notes v4.0.9.6

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.9.6](https://github.com/streamnative/pulsar/releases/tag/v4.0.9.6)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.9.6/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.9.6/images/sha256-87efd75ec5353cde88a62a7be08a56bb9e87ff59b77828a511195ee80854e433)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.9.6/images/sha256-343638a96cd7fd69473d1114811eade023a504c5f83d4d27695117134f2c9614)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.9.6/images/sha256-343638a96cd7fd69473d1114811eade023a504c5f83d4d27695117134f2c9614)

## General Changes

### Apache Pulsar

([#25437](https://github.com/apache/pulsar/pull/25437)) \[fix]\[broker]Producer with AUTO\_PRODUCE schema failed to reconnect, which caused by schema incompatible

([#25293](https://github.com/apache/pulsar/pull/25293)) \[improve]\[broker]Reduce the lock range of SimpleCache to enhance performance

### MoP

933e3701 Ignore some flaky test

### pulsarctl

Update version from v4.0.6.1 to v4.1.3.4

### Function Mesh Worker Service

fix: fix AgentFunction missing auth for package service

71984859 fix ci

fix(runtime): preserve connection fields during updates

feat(registry): add support for short-form package URLs

fix: make pulsar package service always use internal auth

Implement package service

Support volume and volume mounts in CustomRuntimeOptions

### StreamNative Unified RBAC

fix: remove Cloud Integration workflow

fix: switch default Pulsar to snstage/pulsar:4.0.9.6 and simplify integration tests

fix: exclude integration tests from PR CI

fix: remove duplicate CI runs on branch-\* pushes

fix: make pulsar-broker Maven group configurable

0c6a183 Bump version to 1.13.1

9aa9527 Bump version to 1.13.1-rc5

fix: remove duplicate npm version in Publish JS step

60db2c5 Bump version to 1.13.1-rc4

fix: bump sdk-js package.json version in release step

82eb61a Bump version to 1.13.1-rc3

feat: validate before release, fix pipeline ordering

00b7cc6 Bump version to 1.13.1-rc2

refactor: independent versioning, sdk-java-admin split, shade cel

feat: upgrade sdk-go version to v0.16.0

220bc57 Bump version to 1.13.1-rc1

feat: RC-based release workflow

1bbe365 fix: silence TypeScript moduleResolution deprecation warning

f67f8a6 Bump version to 1.13.0

refactor: migrate to Gradle version catalog, restructure project, and fix NAR packaging

2816c22 Bump version to 1.12.0

fix: fix cv image

fix: use different base image based on branch

feat: Add CV workflow to release procedure

b43ce72 Bump version to 1.11.5

fix: include Pulsar 4.1 in pulsar40 compat source set

fix: trigger CI workflows on branch-\* PRs

87cb0b8 Bump version to 1.11.4

fix: add authToken to authz cache key and use refreshAfterWrite

a0a90e0 Bump version to 1.11.3

## Security Fixes


# V4.0.9.7
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.9.7



# StreamNative Weekly Release Notes v4.0.9.7

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.9.7](https://github.com/streamnative/pulsar/releases/tag/v4.0.9.7)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.9.7/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.9.7/images/sha256-661f047a24e2849cc7eb1befade5c2eeadae4c11edbe5641a4171872304b14e1)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.9.7/images/sha256-96e8bb987c01e22f3ca1933b1baf74902e82ffb5486f1a5bb06c41e99a309b43)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.9.7/images/sha256-96e8bb987c01e22f3ca1933b1baf74902e82ffb5486f1a5bb06c41e99a309b43)

## General Changes

### Apache Pulsar

([#25510](https://github.com/apache/pulsar/pull/25510)) \[fix]\[ci] Ensure discard\_max\_bytes is set to 0 only for existing block devices

([#25373](https://github.com/apache/pulsar/pull/25373)) \[fix]\[ci] Disable trivy-action

([#25075](https://github.com/apache/pulsar/pull/25075)) \[fix]\[ci] Fix .github/actions/ssh-access which is used for debugging Pulsar CI in forks

([#25272](https://github.com/apache/pulsar/pull/25272)) \[fix]\[broker] Fix backlog clearing for unloaded namespace bundles

([#25478](https://github.com/apache/pulsar/pull/25478)) \[fix]\[admin] Refactor namespace migration operation to async in rest api

([#25289](https://github.com/apache/pulsar/pull/25289)) \[fix]\[broker] Return failed future instead of throwing exception in async methods

([#25287](https://github.com/apache/pulsar/pull/25287)) \[fix]\[client] Fix async APIs to return failed futures on validation errors

([#25086](https://github.com/apache/pulsar/pull/25086)) \[fix]\[admin] Refactor namespace anti affinity group sync operations to async in rest api

([#25272)](https://github.com/apache/pulsar/pull/25272))) Revert "\[fix]\[broker] Fix backlog clearing for unloaded namespace bundles

([#25478)](https://github.com/apache/pulsar/pull/25478))) Revert "\[fix]\[admin] Refactor namespace migration operation to async in rest api

([#25384)](https://github.com/apache/pulsar/pull/25384))) Revert "\[refactor]\[broker] Decouple delayed delivery trackers from dispatcher

([#25272](https://github.com/apache/pulsar/pull/25272)) \[fix]\[broker] Fix backlog clearing for unloaded namespace bundles

([#25352](https://github.com/apache/pulsar/pull/25352)) \[fix]\[broker] Fix race condition in ServerCnx producer/consumer async callbacks

([#25370](https://github.com/apache/pulsar/pull/25370)) \[feat]\[bookkeeper] add certs refresh

([#25379](https://github.com/apache/pulsar/pull/25379)) \[fix]\[broker] Fix ExtensibleLoadManagerImpl stuck Assigning bundle state after broker restart

([#25384](https://github.com/apache/pulsar/pull/25384)) \[refactor]\[broker] Decouple delayed delivery trackers from dispatcher

([#25400](https://github.com/apache/pulsar/pull/25400)) \[fix]\[client] Fix thread-safety and refactor MessageCryptoBc key management

([#25444](https://github.com/apache/pulsar/pull/25444)) \[improve]\[ci] Cleanup tune-runner-vm and clean-disk actions

([#25478](https://github.com/apache/pulsar/pull/25478)) \[fix]\[admin] Refactor namespace migration operation to async in rest api

([#25483](https://github.com/apache/pulsar/pull/25483)) \[fix]\[broker] Change the schema incompatible log from ERROR to WARN level

([#25520](https://github.com/apache/pulsar/pull/25520)) \[improve]\[broker] Close connection when close consumer write fails

### KoP

\[branch-4.0] Bump pulsar and sn bom versions to 4.0.9.7

Increase timeout values and ensure topic creation is completed in transaction test

Return UNKNOWN\_TOPIC\_OR\_PARTITION error for partitioned metadata loss

Include client id as the suffix of producer name in topic stats

Prevent concurrent metadata requests in each connection

Improve the performance of metadata request processing

Support listing non-partitioned topics

Speed up maven build by adjusting the repository order

Only try creating missed partition when the partition does not exist

fix: respect max bytes limit for both requests and partitions

Fix "LastConfirmedEntry is xxx when reading" read failures after ledger rollover

### StreamNative Pulsar Plugins

0887fed87 Upgrade zookeeper version to 3.9.5

fix: patch CVE-2026-33870 in pulsar-metadata-tool

fix: upgrade alpine to 3.23 and patch musl, libssl3, libcrypto3 for CVEs

fix: upgrade log4j to 2.25.4 to fix CVE-2026-34480, CVE-2026-34481

### pulsarctl

Bump go version to 1.25.9 to fix CVE-2026-32280

### Cloud Pulsar Plugins

\[ApiKeys] Include role in auth failure log during new/refresh auth state

### Function Mesh Worker Service

feat: add more cases for registry CI

## Security Fixes

### Apache Pulsar

([#25353](https://github.com/apache/pulsar/pull/25353)) \[fix]\[sec] Bump google.golang.org/grpc from 1.60.0 to 1.79.3 in /pulsar-function-go

([#25399](https://github.com/apache/pulsar/pull/25399)) \[fix]\[sec] Upgrade to Netty 4.1.132.Final to address CVEs


# V4.1.0.0
Source: https://docs.streamnative.io/release-notes/pulsar/v4.1/v4.1.0.0



## StreamNative Weekly Release Notes v4.1.0.0

#### General Changes

### AoP

Add qpid dependencies

Add Jiwei Guo as code owner of AoP

Add back qpid-test-utils package

Use Github package wildcard repository url

\[test] fix mock zookeeper change

Upgrade artifact github action version

Fix SN bom version

Fix build script

Use SN bom

remove useless check in tests

Fix the publish latency unit

### MoP

Fix MQTT message error handling and improve connection responses

Add support MQTT5 features related doc

Add Cong to the codeowners

Fix subscription authorization PREFIX mode

Fix listener error

fix topic authentication issue

Fix connection event error

Fix user properties lost when enable authorization

Remove jacoco upload jobs

Use Github package wildcard repository url

Fix proxy conn multi broker

Fix mock zookeeper change

Upgrade artifact github action version

Fix build script

Use SN bom

Remove yahoo dependency

MoP accept ws proxy connections

Add mop proxy admin part

Fix mop producer publish metric

Make the proxy adapter worker thread configrable

Fix the auth data is NPE error

Fix broker enable dedup cause client publish failed

Seperate proxy and broker a single module

Refactor MoP to prepare for split Proxy to seperate module

### KoP

Fix Kafka Connect's topic replay loop might be stuck when all messages have been compacted out

Pin avro version to 1.12.0 for tests

Fix possible deadlock of system topic access due to blocking call when holding the lock

Fix incorrect ListOffsets result on a compacted topic

Add partition name to error logs in PartitionLog and UrsaPartitionLog classes

Fix retention.ms may overflow when converting to the Pulsar retention policy

Adapt TopicCompactionService interface changes

Auth SN github maven repo before claude review

Do not set setReplicationClusters on createTopicIfNotExist

Fix consumer close might be stuck when SyncGroup is in progress

Fix topic name reference in AlterPartitionReassignments

Ignore proxy module in CI

Add support for checking schema compatibility against specific versions

Remove key distribution verification from PulsarNonBatchedFormatTester

Increase timeout for testTwoTopicsGroupState to improve reliability

Disable context7 mcp server in Claude github actions

Fix message consumption count in BasicEndToEndPulsarTest

Add Claude Code GitHub Workflow

Support configurable separator for topic

Use a scheduler to handle the KafkaClientPool expired key removal

Fix broken master due to upstream change

Add lookup cache for transaction marker channel manager

Don't replay topic for producer state when the topic is empty

Fix invalid negative timestamp when fetch offset and timestamp

\[Ursa] Support unload topic via rest api

Fix flaky UrsaPartitionLogTest.testClosingWithPendingProduce

\[test] Remove ignore annotation for compaction test

Enhance AdminProxyTest to wait for topic policy updates before alter configs

Fix wrong content length when getting schemas containing non-ASCII text

Remove kafka tenant and kafka namespace config

Add developers and project name

Fix Schema Registry returns internal error if the request body is an invalid JSON

Remove migration service

Support registering the same schema with different subjects for Oxia schema registry

Support consume message via rest api

Fix consume failure caused by duplicate entry recycling

Support a custom managed ledger that extends ManagedLedgerImpl

Add namespace filter in dashboard

Fix InitProducerId could always fail for the same transactional id

Update dispatchable position after update KSN transaction max read position

\[Ursa] Support unload all zones when the zone is not specified in the clientId

Fix ClientIdBasedLookupTest flaky test

Disable automatic group metadata migration in setup to fix flaky test

Migrate nexus staging to maven central

Improve logs for topic lookup

Improve KSN Metrics Dashboard

Update to Oxia 0.6.0 and use new group-id

Add Swagger Maven plugin for REST API documentation generation

Fix get topic by name in KsnRestServlet and update related tests

Reduce unnecessary time-consuming topic replay for producer state recovery

\[refactor] Create PartitionLog only after ProducerStateManager#recover is done

Change the default value of the kopAllowedNamespaces

Support topic retention policy configuration

Update build script to copy kafka-rest-servlet as a jar instead of nar

Remove unnecessary dependencies for rest api

Bump Pulsar to 4.1.0-SNAPSHOT

Add kafka-rest-servlet to asset directory in build script

Support produce message via rest api

Unified metrics topic scope format

Exclude netty-codec from test

Support schema registry RBAC verification at cluster level

Fix UrsaPartitionLog not taking snapshot during shutdown

\[schema-registry] Fix get schema by subject and id behavior

Fix test compatibility request

\[Ursa] Reduce unnecessary time-consuming topic replay for producer state recovery

Support Scram with JWT token Sasl

Return timestamp for the ListOffset request

Fixed NPE when removing partition log

\[Ursa] Skip producer state recovery for consumers and non-idempotent producers

Add kafka-common jar to build script assets

Avoid replaying topics when handling ListOffsets requests

Fix InitProducerID request transaction operation

Support configuring default compatibility level

Change log level from error to info for successful authorization in SimpleAclAuthorizer

\[schema-registry] Use the dot for all subject related requests

Add KSN rest admin servlet

Fix flaky testProducerStateRecovery

\[Ursa] Add per-topic partition storage size metrics for ursa

Filter system topic when list kafka topic

Remove the log when the EventManager receives events after shutdown

Use try load group and offsets to avoid print scheduling loading log

Improve error logging for topic deletion failures

Remove unused jaxb-api dependency from pom.xml

Add namespace label to metrics

Fix ServiceConfiguration cast to KafkaServiceConfiguration

Fix pulsar-kafka-schema-registry compilation failure: Compilation failure

Schedule load group metadata on owner broker

\[Ursa] Fix namespace bundle for topic not served by this instance

\[Ursa] Support unload ursa partition log

Support KSN RBAC part3

Support handle DESCRIBE\_LOG\_DIRS request

Bump Pulsar to 4.1.0-SNAPSHOT

\[Ursa] Support idempotent producer

Remove error logs if a response completes with a known Kafka exception

Support KSN RBAC Part2

Add a separated config to specify number of worker threads

\[ursa] Fix messages might be skipped for a consumer that subscribes multiple partitions

Fix NPE when Accept Header is empty in SchemaRegistry request

\[proxy] Fix memory leak for requests that are not forwarded to brokers

In the `pulsar_non_batched` entry format a single batched message is encoded as a non-batched message

Only take snapshot for producer state except when the topic is unloaded

Exclude lz4-java dependency due to CVE

Don't block requests for unknown request field

Fix pulsar\_non\_batched out of order sequence number error

Fix lookup reference schema

Fix DeadLock issue when resolving the reference schema

SNIP-143: Fallback to base64 encoding only when the byte array is not a valid UTF-8 string

Fix OxiaSchemaStorage use the wrong range scan start key

Add new PulsarNonBatchedEntryFormatter class to encode batched messages to non-batched messages

Support content negotiation for schema registry HTTP service

\[ursa] Support configuring consistent hashing virtual nodes

Fix dead group might not be removed when storing group metadata in metadata store

Retry loading a partition of transaction metadata if it failed to load last time

Disable geo-replication for system topics

Fix memory leak due to temporary buffers are not released in decode

\[ursa] Update default load balance strategy to CONSISTENT\_HASHING

Remove nexus staging repository

Fix connections will always be disconnected when RbacAuthorizationProvider is not configured

Use Github package wildcard repository url

Remove the override updateRates method since it's removed from PersistentMessageExpiryMonitor

Change the ConsumerGroupMetricsCollector error log to warn

Support RBAC authorization validation - Part1

Don't print error logs when creating a topic that already exists

Don't print error logs when receiving GET\_TELEMETRY\_SUBSCRIPTION requests

\[CI] Run smoke-tests on CI

Do not modify the Entry's data when decode

Cherry pick PRs from branch-4.0 to master branch

Avoid blocking I/O threads when lookup due to unstable DNS

Refactor the Ursa partition log cache to make fields immutable as much as possible

Support independent schema registry service

\[test] Update SimpleLoadBalanceTest's load manager config to use `ServiceUnitStateMetadataStoreTableViewImpl`

\[ursa] Re-balance the topic when broker up

Fix UrsaStorageTest.testListOffsetsFromClosedManagedLedger

Remove the mixed\_kafka format and simplify ProducerAppendInfo

Fix failed to generate schemaId/version if the schema-id-gen/version-gen nodes are empty

Fix testIdempotentProduce will fail when running individually

Support wildcard characters for kopAllowedNamespaces

\[test] fix mock zookeeper change

\[ursa] Move the consume logic to use managed cursor directly

Support import mode for oxia schema registry

Fix get schema to return the correct Protobuf type instead of Avro for Protobuf schemas

Fix DescribeConfigs compatibility with Sarama 1.42.1 or earlier

Add a simple consistent hashing based load balancer implementation

Remove the complicated multi-tenant metadata feature and enable transaction by default

Change LEO outdated log to Debug log

Catch the `TooLongFrameException` as warn log

Invalidate UrsaPartitionLog from cache if it's failed

Use slf4j-log4j2-impl as the logger in tests

Fix the ConsumerGroupMetricsCollector admin client url

Revert "Revert using sn-bom (#957)"

Add a simple partition index based load balancer implementation

Update Pulsar stats for Ursa

Upgrade the kafka-clients dependency to 3.9.0 for new requests

\[ursa] Fix shadow managed ledgers' metadata nodes not deleted

Revert using sn-bom

Fix group metadata inconsistency after loading from the metadata store

Update pulsar consumer stats from ksn

Fix flaky OffsetTopicWriteTimeoutTest.testSyncGroup

Fix possible thread safety issue when accessing GroupMetadata

Update the default offset retention time from 3 days to 7 days

Handle schema registry authorization compatibility issue

Fix system topics not filtered for billing metrics

Add pulsar-kafka-schema-registry jar to image

Support schema registry RBAC

Use SN bom

Delete the corresponding shadow topics when the source topic is deleted

Add metric for consumer lag

Fix possible deadlock cause by NamespaceBundleFactory.getBundle

Close the topic during ListOffsets if ML is already closed

Remove the unnecessary asyncMarkDelete call after reading from the cursor

Add more logs for empty assignment

Update clientId format to allow including both zone id and additional information from users

Update the shadow namespace name format

\[Ursa] Don't fail with OFFSET\_OUT\_OF\_RANGE when LEO is less than the fetch offset

Unload topic when fail to create cursor

Retain only the value associated with the valid topic name key when fetching offsets

Fix data\_in\_bytes is always 0 when Ursa is enabled

Fix managed ledger config not respected for Ursa producers

Fix Ursa might not be able to produce after the topic is unloaded

Handle Kafka multi-tenant format topic name in schema registry

Remove producer state snapshot when topic deleted

Filter the pulsar.dedup cursor z-node on the topic deletion event handler

Fix memory leak for producers when Ursa is enabled

Ignore all compaction tests when entryFormat is pulsar

Disable bundle ownership transferring for bundles in shadow namespaces

Remove useless log for the PulsarSchemaStorage

Fix UrsaPartitionLog initialization

Introduce DelayedRemovalCache to cleanup ManagedLedger when Ursa is enabled

Add metrics for Ursa topic initialization

Fix Kafka headers are not converted correctly when entryFormat is pulsar

Return a user-friendly error when the Fetch requests are not supported or topic is blank

Add message duplication tests when Ursa is disabled

Support billing metrics for SN cloud

Support handling produce requests from non-owner brokers

\[fix] Filter duplicated topic when fetch offsets

Adopt a more efficient and reliable approach for compacted topic replay

Enable geo replication test

Support librdkafka's PARTITION\_EOF feature

Re-implement the EventManager

Avoid creating transaction related systems topics when enabling ursa engine

Remove all coordinator epoch usages

Change the default entryFormat to kafka

Refactor the transaction implementation to make it align with Kafka

Fix response status code for Schema Registry

### StreamNative Pulsar Plugins

upgrade avro to 1.12.0

fix export duplicated JVM metrics on AuditLogMetrics

Upgrade pulsar placement policy bk dependency to 4.17.2

Upgrade zk version to 3.9.3 to avoid CVE-2024-51504

Bump com.fasterxml.jackson.core:jackson-core from 2.13.0 to 2.15.0 in sn-pulsar-tool

Fix the build issue

\[detector] Print more info when a corrupted value is received by Kafka consumer

Return empty when getting global topic policies instead of returning exception

Upgrade commons-beanutils version to fix CVE-2025-48734

Add test to verify the sts module

\[detector] Enable idempotence for Kafka detector

Fix the backup tool can not use sts to authenticate

\[cluster-rollout] add orphan unload job cleanup logic

Fix the time ticker leak issue which caused high CPU usage

Update Go SDK rbac & oidc dependencies

Update x/net and go 1.24

Fix jlink command's compress argument

fix(detector): enable pprof by default

feat(detector): support disable pulsar protocol detection.

\[graceful-rollout] set unload retry max

make integration test for all release branch

Add environment to e2e pipeline

add cloud package integration tests

Use Github package wildcard repository url ursa-storage

Fix rest api due to producer min compress size

4a25f8bca Update Go toolchain to 1.22.12

Upgrade deps to fix CVEs

Fix go x/net cve

\[fix]\[cluster-rollout] handle namespace not found greacefully

\[test] Fix mock zookeeper change

Add cert expiration detector

feat(detector): support pprof

fix: discard superuser flag to avoid deadlock

fix(detector): compatible with non-partitioned aliveness topic

Change error log to warn when topic closed

fix initialize for AuthenticationProviderMTls

feat(IdentityPool): Add basic authentication support for identitypool

Add AuthenticationProviderMTls for snoidc

Fix deploy workflow

Fix deploy package workflow

Add plugin module deploy workflow

fix azure package failed when list non-exists directory

Support Topic Level Tracing and LogTraceRecorder

Azure Blob Storage backed Package Management Service

Exclude commons-io to fix cve

Fix license commons plugin jar package

Use WebIdentity way when AWS\_WEB\_IDENTITY\_TOKEN\_FILE exists

fix test in BrokerUnloadJobResourcesTest

Just add non fat jar to image

Use SN bom

Add pulsar-oxia-state-store jar package

Change authentication failed log level to warn

Upgrade aws sdk dependency version to v2

exlude netty for aws-jdk

Reduce pulsar-rollout-plugin nar package size

### pulsarctl

Upgrade go to 1.24.6 to fix CVE-2025-47907

### Cloud Pulsar Plugins

Change to use commons-lang3

### Function Mesh Worker Service

b8175ef4 fix build.sh

Create a new sub module mesh-worker-common

Generate OpenAPI docs for agent-functions

Support set agent tools config

Make MeshWorker able to run standalone and load additional servlets

Support load ConnectorCatalog using label

Update error msg in status

Implement agent function

Set minReplicas to parallelism when HPA is enabled

Set default VPA by default when HPA is not enabled

Support invalid name

Fix resource error during update and get connectors

Fix trigger function not support partitioned input topics error

Find specified ServiceAccount using oauth2's client role and use it when exist

validate function-mesh v0.24.1

exclude lz4-java for CVE reasons

Unified resource scale for all Objects and make it configurable

add security schemas to kafka connect openapi

better error responses

Use Github package wildcard repository url

bump function-mesh to 0.24.0

masking the sensitive data in logs

support insecure auth secret override

Use ConnectRestException for Kafka Connectors

Add default liveness probe:

Validate k8s secret before creating functions/connectors

inferring the tenant and namespace from request if empty

Fix update error for functions/sinks/sources with secrets injected

Support set log topic from configs for Sinks\&Sources

Support set resources for kafka connect

Use SN bom SNAPSHOT version

Do not allow using system topics when creating sink\&source

Reject request when kafka connect's name is longer than 29

Use SN bom to reduce artifact size

Use large runner to avoid disk full issue

Fix free disk job of ubuntu-latest runner

bump kafka dep to 3.9.0

Replace the deprecated `getZooKeeperSessionTimeoutMillis`

Add `extraDependency` field to FunctionMeshConnectorDefinition

Support set pod annotations via CustomRuntimeOptions

Set processingGuarantee for window functions

Bump function-mesh to v0.23.0

### StreamNative Tiered storage

Introduce flag to control delta add file stats

### StreamNative Ursa storage

Remove failed to deserilize log

Add more metrics

Run with different commit runner if task properties changed

Expose parquet file reader cache config

Fix NaN serialization issue in UrsaParquetFileWriter

Fix nested enum serialization failed

fix: Handle NPE when nested record default value has mismatched field names

Refactor the commit process to allow recreate commit runner

Pulsar worker support offload to delta

Remove tmate in CI

USe Hessian2 as the new task serialization.

Fix delete package task bug

Fix topic medata not found issue.

Fix: Use shared static thread pool for PulsarLakehouseReader idle timeout

Upgrade iceberg to 1.9.2

Unblock the compact process when encounter task deserialize exception

Adapt new changes for TopicCompactionService interface

Refresh the catalog instance when using open catalog

Make update and delete compactTask async

Uniform all places configuration name for the data source type

Update the offload flag according to the each ledger state

Rename engineType to dataSourceForCompaction

Fix publish time

Fix committed task delete leak

Add engine type configuration

Refactor the offload format by adding a util to convert the entry to KafkaMessage

Fix deadlock by making getStreamId async in OffloadReadHandler

Fix task compability issue

Fix resource leak: close IndexFileReader in ParquetFileReader

Only publish BlobNotFound exception task to DLQ

Exclude OutOfMemoryError for DLQ

Support publish commit failed tasks to DLQ

Fix oxia lock leak when not acquired

Optimize Compaction Service heap memory usage

fix\[lock]: fixes memory leak on oxia distributed lock

Introduce catalog factory for iceberg

Add external table protobuf support for Ursa and Pulsar protocol

Make the pulsar compaction worker not record column stats.

Using bookkeeperStorageApi when configured pulsar client

Add compaction leader metric doc

Delete committed tasks and update oxia index

Fix the error handling

Remove unnecessary synchronized lock

fix: handle UUID logical type with string base type in AvroToIcebergConverter

Fix ConcurrentModificationException in CompactionTaskProvider.getTask

Cleanup stream when deleting unloaded topics

Add multi catalog user document

Add compaction service throguhput rate limiter for reading from BookKeeper

Support multiple catalog in namespace and topic level

Optimize reset cursor

Refactor update iceberg table properties

Separate managed and external writer for ursa

Add compaction leader metric

Add failure reason for the iceberg external writer

Fix CI

Upgrade delta kernel to 4.0.0

Make the RawReader object pooled to avoid creating each time

Delete the compact task if the topic doesn't exist

\[refactor] Only open one parquet file for the lakehouse reader

Using the GlobalOpenTelemetry to register the reader metrics

Fix reader read failed in parallel

Delta external table introduce temporary credential

Fix the task compatibility issue.

Introduce CustomKernelParquetWriter to support put write mode to improve memory usage

Fix readIndexes bug

Fix the typo for the classname of PreparedCompactStreamTask

Introduce the task type to control the compaction handling

Add offload cursor to block data expire

Send the failed parse messages into a failure topic

Fix the metrics tests

Load configuration from the pulsar-client.conf file

Close the catalog resrouce after using

Delete the compaction task if it compacting the expired data

Add ML Cloud Storage Developer Guide

Reenable the pulsarE2ETest

Speed up the CI process from 1 hour to 20mins by separating to the different groups

Adapt for unity iceberg rest api

Fix the offloaded ledgermetadata is not synced with the offload state

chore: Add Claude Code Agents

Allow to disable sync UrsaMLMetadata

Support delete ledger from pulsar offload handler

Add metrics for pulsar read/writer lakehouse path

Redirect maven twitter repo to central

Fix build script

Refactor pulsar external table

Support load pulsar client token from file

Use the existing resources to init lakehouse reader

Renable the pulsar e2e tests

Get ledger metadata from Oxia

Change default entrySerDeType to PULSAR\_BATCHED\_RAW\_PARQUET

support register managedledger meatadata in oxia

Pulsar support reading from parquet

store messageId into parquet file

Store ledger metadata to oxia

Use jar instead of nar for offloader

Add serialization type in the metadata

Revert to use normal file as the index file

Support save pulsar entry without parsing batch

Support skip system topic and black topics

Fix the bookkeeperApi can not get the index by secondary key

Using MapFile to speed up the seek performance

Support deleting the compacted data

Update the offload flag according to the ursa storage state

Make pulsar compaction worker enable iceberg external table writer

Reuse the pulsar storage configuration for the pulsar offloader

Update metadata store ledgers info after checking offloaded flag.

Fix the prepare task name compatibility issue

\[cleanup] remove unused code in the managedLedgerWithTs

Support generate the Ursa offset when committing the task

Update to Oxia 0.6.0 and use new group-id

Make bookkeeperStorageApi implement the StorageApi

Pulsar offloader leader support

Support write without parse content with schema

Support compress the index files

Make the tasks in the CommitTaskProvider sorted by the stream id and start offset

Pulsar offloader support offload to iceberg format

Support read/write index file with hadoop lib to cloud storage

Fix pulsar lakehouse reader memory leak issue

Integration test for the pulsar protocol compaction process

Refactored to allow support for low-latency storage class

Using the 4.1.0-SNAPSHOT of pulsar

Adapt the comapction process with pulsar related code

Fix the NPE when serializing the bytes schema version

Use static instances for compression codecs

Pin version for commons-lang3

Support bookkeeper entry reader to let compaction service read from bookkeeper

Support read and write bookkeeper entries

Introduce the api for bookkeeper generate offsets

Support trimming topic with the mark-deleted-offset

Fail back to normal config when the external config miss.

Optimize entry reader read batch

Allow to configure the http client used by Azure

Fix build script typo

Support pulsar entry write into / read from the parquet file process

\[doc] WAL Cloud Storage Developer Guide

Update Metrics.md with accurate and concise descriptions

Fix topic quarantine bug

Implement new interface for Pulsar

Add default timeout for all integration tests

External table support wal compact

Disable expire Iceberg snapshot by default

Support fencing a managed ledger after closing

Use nonRetriableQuaratine for task publish

Improve the compaction worker handle stream task logic.

Flaky test in SimpleStorageImplTest.java

\[improve] Use ReentrantReadWriteLock in  EntryCache

Move out the integration test containers to a new module

Reduce primitive schema retry times

Format compaction service quarantine logic

Support configure iceberg table properties with topic properties

Throw exception when read empty entries in lakehouse worker.

\[fix] throw exception upon EntryCache apis calls after EntryCache is closed

Move the persistStorageApi initialize in the common place

Fix real offset incorrect

Fix semaphore not release bug

Fix read lock not release bug

Fix build failure

GCS supports delete with lifecycle

Throw exceptions when handle the recursive schema

throw exception when hitting non-binary index while processing RAW type

Support azure to delete object using lifecycle rules

Delete the compact task when compaction worker read compacted wal file.

Quarantine topic when get topic failed or get topic metadata failed

fix deadlock from EntryIndexCache.invalidate

Skip topics in pulsar tenant in compaction service

Fix compaction service generates a lot of small parquet files

Add Iceberg bigquery metastore catalog support

Fix publish thread executor block issue.

Skip some tests to speed up integration test

Fix s3Table name format

Remove sn-bom build in CI

\[fix] write api support for v2 format

Make the entry read instance in compaction not shared betweet the different task

Fix the IllegalReferenceCountException by duplicated method execute

Fix npe when check lakehouse commit table failed.

Do ursa GCS performance and improvement

Fix GCS deadlock issue

Fix priorityqueue concurrentModification exception

Recreate LakehouseStreamCompactWorker when encountered S3Exception

Add more metrics for compaction service

Use Github package wildcard repository url

Support write entry with specified initial offset

Support streamnative Delta External table.

Add metrics for the read cache size in bytes

Fix the aws config socket time not work issue.

Add more netty config

Fix the NPE when getting the non-exists blob

Remove keys from lock when unlock

\[improve] Ursa Live Entry Index Compaction

Improve parquet read prefetch

Use SN bom based image

Fix taskWriter shared bug

Fix avro convert to iceberg npe

Format lakehouse exception

Fix the memory leak issue in the test.

Fix memory leak when write parquet failed

Fix the record and schema convert issue for avro to iceberg

Fix LakehouseKafkaReader NPE

optimize the jvm opts for compact service.

Remove serailize failed noise log

tmp

Do not catch handleRecord exception

Fix dockerfile

Remove the test jar from the image

Revert #615 to avoid the caffine cache performance issue.

Enable v3 serialization and deserialization

Make compaction service compatible with Confluent schema registry

Fix maxPendingAddRequestsUsedBytes integer overflow

Fix the map value struct type issue in unity catalog.

Fix avro record convert to Iceberg Record failed

Fix retry the compact stream task failed issue.

Fix parquet row writer issue

Introduce delta kernel to replace of delta-standalone.

Fix unity catalog schema issue

\[Fix] Fix the iceberg namespace on s3 table catalog

Limit the readcache by bytes

Limit the pending add by bytes

Support nested protobuf schema for the compaction

Move the create table to the writer instead committer

Add test case to cover msg payload schema content

Remove copied class

Remove the ursa-lakehouse dependency from the ursa-ml

Add iceberg external table support

Support kafka protobuf schema for the compaction

Fix BrokerInterceptorTest.testAddBrokerEntryMetadataEntryRef flaky test.

Add test unit to cover get wrong streamId issue.

Fix CI

Fix the json schema convert issue.

Fix always use the wrong streamId when publish stream task.

Set partitionkey for the rangeScan in the persistStorageApi

Add the persistCache size into the meta

Improve PersistCache with a simpler and thread-safe implementation

Add EntryIndex protobuf

feat: move the info log to debug to prevent server keep logging

\[cleanup] Remove unused api

Refactor compaction worker interface

Replace guava cache with caffeine cache

\[cleanup] Move the read cache into a single class

Clean up the unused code of ursa-storage-core

Remove the unused code

Unity Catalog support clientId and clientSecret to authenticate.

Use ubuntu-latest to run ci

Fix Github Packages publish workflow

Remove nexus snapshot distribution

Use `____` to replace '-' for table name when create unity catalog table.

Support publishing to GitHub Packages

Fix ManagedCursorTest#testReadEntriesWithSkipDeletedEntries flaky test.

Fix StorageWalManagedCursor#internalReadEntries only read one entry at each round.

Fix s3 compact flaky test.

\[TEST] Run integration test with image ksn 3.9.0

Fix parquet prefetch bug

Add iceberg azure dependency

Use separate thread pool for parquet reading

Test master CI

Fix default compaction thread bug

Support load credential from file

Fix flaky test TestCompactionServiceBaseFileStorage#simple

fixed flaky test\_whenReadAgaint

add cloud region

9d6fd76f \[Bug] fix duplicated entry id put in PersistCache.index

Adjust the compaction default configuration according to the performance test

Fix flaky test in TestGarbageCollection

Add the ProfileCredentialsProvider into the auth chain

Disable nonblocking dns and get rid of request limitation of S3 prefix

improvement in dispatch logic

Fix the default write buffer segment configuration

Unity catalog support config User-agent.

Adding stress-ng to CI

Fix oxia read failed with small range

Improve CI by running S3 and GCS integration tests In parallel

Optimize PersistCache serialization to only persist used segments

Support disable read cache expire by time

Compaction service support GCS

Delta table support partition column using topic partition index

Compact service support azure blob protocol

Parse the storage account name and blob container name from bucket for azure

Check the table whether register to the unity catalog when commit action. If not, register the table to unity catalog.

Compaction service support Azure storage.

Modify delta table mapping name in unity catalog

Rename azure to azureblob to make it more precise

fix azure make data in heap

Quarantine the compact task if it read WAL data fialed.

Add the storage metrics for all the file storage type

Allow to disable the lakehouse reader in the managedledger

Enable ChecksumCRC32C for getting object from S3

Add core and lakehouse jars to image

Support unity catalog.

Add iceberg support

Remove jar with dependencies when release

Switch to sn-bom pom dependency

\[cleanup] Remove the duplicated code

Limit the read request from compaction to the storage api

Get the compact task in the start offset order

Separate the integration tests and ut in workflows

Enable all the primitive types tests

Grouping pending add requests by stream id

Add read request limitation for the persistStorageApi read

Guarantee managed ledger's stream id is always valid and never modified

Add Yunze and Zike as code owner

Avoid concurrently update metadata for ShadowManagedLedger

Provided storage endpoint for azure storage

\[WIP] Add primitive type support

Fix the null filed value can't decode issue.

Introduce lakehouse read prefetch cache manager

Speed up get all task

Separte the maxRequest config and write buffer segment config

Fix the commit runner race condition issue.

Update streamId if Shadow Managed Ledger stream ID is invalid

Trigger the metadata update when calling getLastConfirmed entry

\[improve]\[tests] covered Pulsar protocol retention and backlog quota

\[improve]\[tests] covers Pulsar protocol encryption and compression tests

Remove awssdk bundle dependency

Avoiding the risk of pending add buffer release

fix netty and json cve

Fix the stream id is duplicated

Do not reject the entries bigger than write buffer

Support the azure file storage

\[improve]\[tests] enabled ExtensibleLoadBalancer in integ tests and added delayed messaging test

\[improve]\[tests] enabled concurrent tests in integ and enabled new load balancer

Support creating the managedLedger in different brokers

Add GCS FileStorage implementation

Introduce task manager to avoid acquire lock between threads (Re-Commit)

LakehouseKafkaReader add prefetch cache support

\[fix]\[ml] fixed npe in getLastIndividualDeletedRange

Improve the WAL catch up read throughput by prefetching cache

\[improve]\[tests] cover Pulsar messaging tests

\[fix]\[tests] Enable ursa-storage-test tests

Fix trivy downloading db rate limitation

Make mockito test scope

([#401)](https://github.com/streamnative/ursa-storage/pull/401)))  Revert "Introduce compaction task provider to avoid acquire lock between multiple threads

Fix the integer key leak issue

Introduce compaction task provider to avoid acquire lock between multiple threads

Fix aws credential not match issue

Reduce package size

Upgrade aws sdk version to fix cve

Fix CVE-2024-7254

Trigger publish task by self, not wait all the topics.

Add error log when put entry failed in the managedLedgder

Update the Docker resource and readme to the latest stat

Add json support check in publish task

Bump org.scala-lang:scala-library from 2.13.0 to 2.13.9

CompactService support json schema

Optimize quarantine logs in publish tasks

Bump project version and pulsar version to 4.1.0-SNAPSHOT

\[metrics] Add metrics for the write buffer

Use ConcurrentHashMap to reduce risks

\[cleanup] move the write cache out of the storage impl

Bump org.apache.hadoop:hadoop-common from 3.3.6 to 3.4.0

Support write entry for the StorageApi

Filter topics with not support schema

Optimize logs in compaction service

Refactor compaction metrics

Optimize compaction service default configurations

Make the stream map evict by time.

\[cleanup]Remove the unused code in the persistStorageApi

Fix the flaky test testOperationRejection

Fix wrong metadata updates for first and last entry headers

Increase commit runner concurrency

Fix som cve

feat(distributed-lock): use metadata oxia client as distributed lock client

Use concurrent map for the cache strings in storageApi

\[cleanup] Move the metrics part out of the implementation

Make the maxPendingAddRequest dynamic

Correct the dockerfile used file location

Make the sync method call the async method to get result

Fix the request rejection of SimpleStorageImpl

Fix the missed exception handling for getting the next read index

Fix prepared publish task may failed issue.

Fix the ledger deletion for PersistCache

Parquet read perfomance improvement

Delay metadata update task for managedLedger

Introduce compact cordinator.

Fix flaky test `testReadAfterTrimmed`

Decrease s3OpsRateLimitPerSecond to 100

Enable the CI on all branches

Fix delete topic failed when not load

Fix compaction otel paramter


# V4.1.0.1
Source: https://docs.streamnative.io/release-notes/pulsar/v4.1/v4.1.0.1



# StreamNative Weekly Release Notes v4.1.0.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.1.0.1](https://github.com/streamnative/pulsar/releases/tag/v4.1.0.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.1.0.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.1.0.1/images/sha256-0ce649625ae5389aa368b1b2253fd957104175d5b158b0409f33040a113a6c8e)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.1.0.1/images/sha256-ae9bd01f7bb406b5cec9f2428bc4aaad1c27565e37a7f6cf3bcee883ce2f6993)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.1.0.1/images/sha256-ae9bd01f7bb406b5cec9f2428bc4aaad1c27565e37a7f6cf3bcee883ce2f6993)

## General Changes

### Apache Pulsar

([#24719](https://github.com/apache/pulsar/pull/24719)) \[fix]\[broker] Fix memory leak when metrics are updated in a thread other than FastThreadLocalThread

([#24712](https://github.com/apache/pulsar/pull/24712)) \[improve]\[io] Upgrade to Debezium 3.2.2

([#24682](https://github.com/apache/pulsar/pull/24682)) \[improve]\[broker] Reduce unnecessary MessageMetadata parsing by caching the parsed instance in the broker cache

([#24594](https://github.com/apache/pulsar/pull/24594)) \[improve]\[build] Disable javadoc build failure

([#24706](https://github.com/apache/pulsar/pull/24706)) \[fix]\[broker] Fix NPE and annotate nullable return values for ManagedCursorContainer

([#23942](https://github.com/apache/pulsar/pull/23942)) \[improve]\[client]  PIP-407 Add newMessage with schema and transactions

([#24699](https://github.com/apache/pulsar/pull/24699)) \[improve]\[ml] Improve cache insert performance by removing exists check since it's already covered by putIfAbsent

([#24689](https://github.com/apache/pulsar/pull/24689)) \[feat]\[misc] upgrade oxia version to 0.6.2

([#24691](https://github.com/apache/pulsar/pull/24691)) \[fix]\[client] Fix potential NPE in TypedMessageBuilderImpl

([#24488](https://github.com/apache/pulsar/pull/24488)) \[improve]\[client] PIP-420: Supports users implement external schemas

([#24684](https://github.com/apache/pulsar/pull/24684)) \[improve]\[doc] Cleanup some legacy PIP documents and improve PIP listing

([#23351](https://github.com/apache/pulsar/pull/23351)) \[improve] \[pip] PIP-382: Add a label named reason for topic\_load\_failed\_total

([#24648](https://github.com/apache/pulsar/pull/24648)) \[fix]\[broker]User topic failed to delete after removed cluster because of failed delete data from transaction buffer topic

([#23222](https://github.com/apache/pulsar/pull/23222)) \[improve] \[pip] PIP-375 Expose the Admin client configs: readTimeout, requestTimeout, and connectionTimeout

([#23336](https://github.com/apache/pulsar/pull/23336)) \[fix]\[client] Fix ArrayIndexOutOfBoundsException when using SameAuthParamsLookupAutoClusterFailover

([#24681](https://github.com/apache/pulsar/pull/24681)) \[improve]\[test]Add new test PartitionCreationTest.testGetPoliciesIfPartitionsNotCreated

([#24678](https://github.com/apache/pulsar/pull/24678)) \[improve]\[doc] Update PIP links in PIP documents converted from the wiki and remove trailing whitespace

([#24679](https://github.com/apache/pulsar/pull/24679)) \[fix]\[broker]Fix flaky test PartitionCreationTest.testCreateMissedPartitions

([#24623](https://github.com/apache/pulsar/pull/24623)) \[improve]\[broker] Implement PIP-430 Pulsar Broker cache improvements

([#24622](https://github.com/apache/pulsar/pull/24622)) \[improve]\[broker]Find the target position at most once, during expiring messages for a topic, even though there are many subscriptions

([#24651](https://github.com/apache/pulsar/pull/24651)) \[fix]\[broker]Failed to create partitions after the partitions were deleted because topic GC

([#24665](https://github.com/apache/pulsar/pull/24665)) \[fix]\[meta] Use `getChildrenFromStore` to read children data to avoid lost data

([#23977](https://github.com/apache/pulsar/pull/23977)) \[fix]\[broker] Invalid regex in PulsarLedgerManager causes zk data notification to be ignored

([#24663](https://github.com/apache/pulsar/pull/24663)) \[fix]\[client] Skip schema validation when sending messages to DLQ to avoid infinite loop when schema validation fails on an incoming message

([#24669](https://github.com/apache/pulsar/pull/24669)) \[improve]\[io] Support specifying Kinesis KPL native binary path with 1.0 version specific path

([#24668](https://github.com/apache/pulsar/pull/24668)) \[improve]\[build] Use org.apache.nifi:nifi-nar-maven-plugin:2.1.0 with skipDocGeneration=true

([#24666](https://github.com/apache/pulsar/pull/24666)) \[improve]\[build] Increase maven resolver's sync context timeout

([#24427](https://github.com/apache/pulsar/pull/24427)) \[fix]\[broker] PIP-428: Fix corrupted topic policies issues with sequential topic policy updates

([#24661](https://github.com/apache/pulsar/pull/24661)) \[improve]\[io] Upgrade AWS SDK v1 & v2, Kinesis KPL and KPC versions

([#24662](https://github.com/apache/pulsar/pull/24662)) \[fix]\[client] fix ArrayIndexOutOfBoundsException in SameAuthParamsLookupAutoClusterFailover

([#24655](https://github.com/apache/pulsar/pull/24655)) \[improve]\[ml] Optimize ledger opening by skipping fully acknowledged ledgers

([#24660](https://github.com/apache/pulsar/pull/24660)) \[improve]\[doc] Add all legacy PIPs from Pulsar wiki and GitHub issues as files to pip directory

([#24659](https://github.com/apache/pulsar/pull/24659)) \[fix]\[misc] Upgrade fastutil to 8.5.16

### AoP

fix managed cursor container constructor in test

### KoP

Support broker side schema validation

Ursa: fix incorrect warn log when appending new messages

Correct token extraction for Kafka internal client

Remove useless authorization warning log

\[flaky-test] Fix ListConsumerGroupTest

Fix build issue caused by upstream interface changes

Remove dependencies with Confluent Community License

Ignore the read\_committed field in Ursa

Resolve RBAC compatibility issue

Fix topic read authorization not applied for OffsetDelete requests

Add module name

### StreamNative Pulsar Plugins

Adapt to the latest topic policies interface from PIP-428

Add audit logging support for non-partitioned topic creation

### pulsarctl

Update stable version

### StreamNative Ursa storage

Fix the external table commit state failed to check issue

Fix build failure on the latest snapshot

upgrade oxia version to 0.6.2

feat: upgrade oxia version to 0.6.1

IcebergTable support schema evolution

Add test for #1253.

Use number to convert the digital type at AvroToGenericRowConvert

## Security Fixes

### Apache Pulsar

([#24717](https://github.com/apache/pulsar/pull/24717)) \[fix]\[sec] Upgrade Netty to 4.1.127.Final to address CVEs


# V4.1.0.10
Source: https://docs.streamnative.io/release-notes/pulsar/v4.1/v4.1.0.10



# StreamNative Weekly Release Notes v4.1.0.10

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.1.0.10](https://github.com/streamnative/pulsar/releases/tag/v4.1.0.10)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.1.0.10/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.1.0.10/images/sha256-34cdfdff42c6a6d4bb9bac4055b3e0e2d5a7d25f517699ec6293e33d917692e5)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.1.0.10/images/sha256-6e5c4096dc7839aadf05ac73a898519d69c70ca6a72c356624da7127e4a2d387)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.1.0.10/images/sha256-6e5c4096dc7839aadf05ac73a898519d69c70ca6a72c356624da7127e4a2d387)

## General Changes

### Apache Pulsar

([#25073](https://github.com/apache/pulsar/pull/25073)) \[fix]\[broker]Infinitely failed to delete topic if the first time failed and enabled transaction

([#25047](https://github.com/apache/pulsar/pull/25047)) \[fix]\[broker]Fix incorrect backlog if use multiple acknowledge types on the same subscription

([#24980](https://github.com/apache/pulsar/pull/24980)) \[fix]\[broker] fix prepareInitPoliciesCacheAsync in SystemTopicBasedTopicPoliciesService

([#24658](https://github.com/apache/pulsar/pull/24658)) \[improve]\[broker] Optimize Reader creation in TopicPoliciesService

([#25053](https://github.com/apache/pulsar/pull/25053)) \[improve]\[broker] Use atomic counter for ongoing transaction count

([#25069](https://github.com/apache/pulsar/pull/25069)) \[fix]\[client] Fix invalid parameter type passed to Map.get in TopicsImpl.getListAsync method

([#25066](https://github.com/apache/pulsar/pull/25066)) \[fix]\[broker] PIP-442: Fix race condition in async semaphore permit updates that causes memory limits to become ineffective

([#25044](https://github.com/apache/pulsar/pull/25044)) \[improve]\[broker] Improve replicated subscription snapshot cache so that subscriptions can be replicated when mark delete position update is not frequent

([#25067](https://github.com/apache/pulsar/pull/25067)) \[fix]\[broker] Force EnsemblePolicies to resolve network location after rackInfoMap is updated due to changes in /ledgers/available znode

([#25050](https://github.com/apache/pulsar/pull/25050)) \[fix]\[admin] Refactor bookie affinity group sync operations to async in rest api

([#25059](https://github.com/apache/pulsar/pull/25059)) \[fix]\[broker] Fix various error-prone detected errors mainly in logging and String.format parameters

([#25054](https://github.com/apache/pulsar/pull/25054)) \[improve]\[build] Upgrade errorprone to 2.45.0 version

([#25056](https://github.com/apache/pulsar/pull/25056)) \[fix]\[cli] Fix output of --print-metadata in cli consume

([#25051](https://github.com/apache/pulsar/pull/25051)) \[fix]\[cli] Fix some pulsar-admin topicPolicies commands exiting before async operations complete

([#16651](https://github.com/apache/pulsar/pull/16651)) \[improve]\[broker] Fix replicated subscriptions race condition with mark delete update and snapshot completion

([#25027](https://github.com/apache/pulsar/pull/25027)) \[improve]\[misc] Add log4j-layout-template-json to server distribution to enable e.g. ECS template support in log4j configurations for Pulsar server components.

([#25032](https://github.com/apache/pulsar/pull/25032)) \[fix]\[test] Replace LZ4FastDecompressor with LZ4SafeDecompressor in test

([#25034](https://github.com/apache/pulsar/pull/25034)) \[improve]\[misc]introduce log4j Console appender ConsoleJson

([#25039](https://github.com/apache/pulsar/pull/25039)) \[fix]\[broker] Fix potential NPE in InMemTransactionBuffer.appendBufferToTxn by returning a valid Position

([#25026](https://github.com/apache/pulsar/pull/25026)) \[improve]\[broker]Add test for getting partitioned topic metadata with PulsarAdmin client

([#25029](https://github.com/apache/pulsar/pull/25029)) \[improve]\[io] Upgrade Debezium version to 3.2.5.Final

([#25036](https://github.com/apache/pulsar/pull/25036)) \[improve]\[client] Add null checks for MessageAcknowledger methods to prevent NullPointerException

([#25037](https://github.com/apache/pulsar/pull/25037)) \[fix]\[broker]Incorrect backlog that is larger than expected

### KoP

Handle no zone case for ursa storage

Fix schema-registry docker image build workflow

Build docker image for schema-registry

\[Ursa] Recover producer state according to the client id's zone

Upgrade Confluent Schema Registry version to 7.9.4

\[Ursa] Fix producer state recovery will never complete when messages are written concurrently

Bump-branch-4.1 to 4.1.0.9

Add detailed logging for Schema Registry error responses

### StreamNative Pulsar Plugins

Upgrade pulsar and sn bom version to 4.1.0.9

Fix setup-go action version

### pulsarctl

Fix setup-go action issue

Fix setup-go action version and upgrade go version to fix CVE

### Cloud Pulsar Plugins

Add FileBasedJwksResolver

### Function Mesh Worker Service

Add ComponentLimits to custom config

support plain auth for KafkaConnect and individual functions worker deployment

## Security Fixes

### Apache Pulsar

([#25045](https://github.com/apache/pulsar/pull/25045)) \[fix]\[sec] Bump at.yawk.lz4:lz4-java from 1.9.0 to 1.10.1 in /pulsar-common


# V4.1.0.11
Source: https://docs.streamnative.io/release-notes/pulsar/v4.1/v4.1.0.11



# StreamNative Weekly Release Notes v4.1.0.11

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.1.0.11](https://github.com/streamnative/pulsar/releases/tag/v4.1.0.11)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.1.0.11/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.1.0.11/images/sha256-54b8cf97959b2cdf2dc11662157e3342acc43fd11a05d8bcbd1692a64959221f)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.1.0.11/images/sha256-285cfd5da8b0675112c22583fbec34dfa3dbe39e29eb414949ebaa9639d991c6)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.1.0.11/images/sha256-285cfd5da8b0675112c22583fbec34dfa3dbe39e29eb414949ebaa9639d991c6)

## General Changes

### Apache Pulsar

([#25105](https://github.com/apache/pulsar/pull/25105)) \[fix]\[broker]pulsar\_ml\_reads\_inflight\_bytes and pulsar\_ml\_reads\_available\_inflight\_bytes are 0 at the same time

([#25087](https://github.com/apache/pulsar/pull/25087)) \[fix]\[broker] Fix cursor position persistence in ledger trimming

([#25085](https://github.com/apache/pulsar/pull/25085)) \[improve]\[io] Replace Qpid in tests with RabbitMQ in Testcontainers and upgrade RabbitMQ client version

([#25084](https://github.com/apache/pulsar/pull/25084)) \[fix]\[build] Activate jdk21 and jdk24 profiles on Java 25

### MoP

Fix mqtt disconnect due to appId permission bug when namespace policies update

### KoP

\[Ursa] Fix the pulsar internal topic owner issue caused create topic stuck

Fix potential NPE issue when initializing schema storage reader

Fix inflight reads limiter permits leak when offsetsForTimes is called

Return UNKNOWN\_TOPIC\_OR\_PARTITION error for partitioned metadata loss

Fix retention and TTL policies on metadata namespace

Fix producer state manager snapshot buffer start issue when use global zk

Improve logging for read entry errors

fix(schemaregistry): add schema type check before compatibility checking

fix(schema-registry): validate JSON schema format during registration

Remove the immature producer side throttling feature

Correct token extraction for Kafka internal schema registry client

### pulsarctl

Fix assertion on TopiCreateTimeStamp

Upgrade pulsar go client to latest and golang to 1.25

### Function Mesh Worker Service

Uncomment connector copy commands in Dockerfile

fix: kafka sink auth inject not working

Update authorization error msg

build(deps): bump function-mesh.version to v0.26.1

## Security Fixes

### Apache Pulsar

([#25102](https://github.com/apache/pulsar/pull/25102)) \[fix]\[sec] Upgrade log4j to 2.25.3 to address CVE-2025-68161

([#25095](https://github.com/apache/pulsar/pull/25095)) \[fix]\[sec] Upgrade jose4j to 0.9.6 to address CVE-2024-29371

([#25078](https://github.com/apache/pulsar/pull/25078)) \[fix]\[sec] Upgrade Netty to 4.1.130.Final


# V4.1.0.12
Source: https://docs.streamnative.io/release-notes/pulsar/v4.1/v4.1.0.12



# StreamNative Weekly Release Notes v4.1.0.12

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.1.0.12](https://github.com/streamnative/pulsar/releases/tag/v4.1.0.12)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.1.0.12/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.1.0.12/images/sha256-7cf4659844956c9515ed2275840ce7c0305c933052449eb4917edf662cec2d21)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.1.0.12/images/sha256-6435500d6c4c7fe49ce02dd445754970b6298ca729e8526b9f99f32f128dc71a)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.1.0.12/images/sha256-6435500d6c4c7fe49ce02dd445754970b6298ca729e8526b9f99f32f128dc71a)

## General Changes

### Apache Pulsar

([#25106](https://github.com/apache/pulsar/pull/25106)) \[fix]\[client]Producer stuck or geo-replication stuck due to wrong value of message.numMessagesInBatch

### KoP

Add github deploy profile

### StreamNative Pulsar Plugins

f7d807b63 ignore flaky test

552c8cc06 Fix checkstyle issue and opentel sdk spi version

b269d9683 Revert "add opentelemetry version"

3f1d5b089 fix checkstyle issue

Support protobuf-native type for rest-v2

0d10f56cf add opentelemetry version

Add new rest-consume api

### Function Mesh Worker Service

feat(auth): add API Keys authentication support

### StreamNative Unified RBAC

perf(authz): optimize JWT parsing and metadata extraction with caching

## Security Fixes


# V4.1.0.13
Source: https://docs.streamnative.io/release-notes/pulsar/v4.1/v4.1.0.13



# StreamNative Weekly Release Notes v4.1.0.13

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.1.0.13](https://github.com/streamnative/pulsar/releases/tag/v4.1.0.13)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.1.0.13/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.1.0.13/images/sha256-cbceda11fb1947bbd76317aeb6c4d209d6d5672d1a2ffdd6c0ecf4c8cdca5c3a)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.1.0.13/images/sha256-7e34265e7a3e73669628e7ee4ab794f0103430dca019543af88fee8d2a503340)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.1.0.13/images/sha256-7e34265e7a3e73669628e7ee4ab794f0103430dca019543af88fee8d2a503340)

## General Changes

### Apache Pulsar

([#25125](https://github.com/apache/pulsar/pull/25125)) \[fix]\[test] Wait for txn.abort() to complete to avoid AdminApiTransactionTest.testAnalyzeSubscriptionBacklogWithTransactionMarker() flaky test

([#25114](https://github.com/apache/pulsar/pull/25114)) \[fix]\[broker]Topic deleting failed after removed local cluster from namespace policies

([#25130](https://github.com/apache/pulsar/pull/25130)) \[improve]\[broker] Change the log level from error to info when throwing NotAllowedException

([#25048](https://github.com/apache/pulsar/pull/25048)) \[improve]\[broker] Enhance logging for adding schema failures in ServerCnx

([#25121](https://github.com/apache/pulsar/pull/25121)) \[fix]\[broker] Fix MultiRolesTokenAuthorizationProvider error when subscription prefix doesn't match.

([#25119](https://github.com/apache/pulsar/pull/25119)) \[fix]\[broker] Fix compaction horizon might be reset to an old position when phase two is interrupted

([#25104](https://github.com/apache/pulsar/pull/25104)) \[improve]\[broker] Fix thread safety issue in ManagedCursorImpl.removeProperty

([#25091](https://github.com/apache/pulsar/pull/25091)) \[improve]\[admin] Add counter for marker messages in PersistentTopics.analyzeSubscriptionBacklog() rest api

([#25089](https://github.com/apache/pulsar/pull/25089)) \[fix]\[ml] Fix cursor backlog size to account for individual acks

([#25077](https://github.com/apache/pulsar/pull/25077)) \[fix]\[broker] Fix chunked message loss when no consumers are available

([#25101](https://github.com/apache/pulsar/pull/25101)) \[fix]\[test] Fix ManagedCursorTest and NonDurableCursorTest flaky tests

### KoP

Cache Maven dependencies to speed up CI

Set shadow namespace load manager to void interceptor test fail

Reduce total test time of all workflows

\[Ursa] Prevent possible partitioned metadata loss that fails ksqlDB's SHOW TOPICS command

test(transaction): increase timeout for transaction recovery test

Print maven effect setting before build

Remove ShadowTopicManager

Support complex oxia config for Schema Registry service

Fix schema evolution version resetting after deleting a subject version in oxia schema registry

Add cloud plugin common library when building schema registry docker image

### StreamNative Pulsar Plugins

05add0d3d fix checkstyle issue

2787a1a37 fix(rest): fix producer leak

feat(compaction): Support `compact,delete` cleanup Policy for Kafka topics

### Function Mesh Worker Service

Support custom agent framework and hpa for Agent

### StreamNative Tiered storage

Fix some CVE

## Security Fixes


# V4.1.0.14
Source: https://docs.streamnative.io/release-notes/pulsar/v4.1/v4.1.0.14



# StreamNative Weekly Release Notes v4.1.0.14

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.1.0.14](https://github.com/streamnative/pulsar/releases/tag/v4.1.0.14)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.1.0.14/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.1.0.14/images/sha256-dfe4b8fb8e32f4e9ebe43d932c904cbaefcb1828ce5af4f78664c096bf9fc4be)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.1.0.14/images/sha256-96e051f02e34f32a112af86ea7c854932cc6e18fa0664ccc8f0d5a41ac83443f)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.1.0.14/images/sha256-96e051f02e34f32a112af86ea7c854932cc6e18fa0664ccc8f0d5a41ac83443f)

## General Changes

### Apache Pulsar

([#25136](https://github.com/apache/pulsar/pull/25136)) \[fix]\[broker] Fix regex matching of namespace name which might contain a regex char

([#25110](https://github.com/apache/pulsar/pull/25110)) \[fix]\[broker] Fix markDeletedPosition race condition in ManagedLedgerImpl.maybeUpdateCursorBeforeTrimmingConsumedLedger() method

### KoP

Ignore the exception for duplicated release on an entry

Return failed future instead of null when cursor manager is closed

Avoid closing a Kafka admin with small request timeout to speed up DescribeConsumerGroupTest

### StreamNative Pulsar Plugins

\[pulsar-detector] Extract the auth username from the JWT token

## Security Fixes


# V4.1.0.15
Source: https://docs.streamnative.io/release-notes/pulsar/v4.1/v4.1.0.15



# StreamNative Weekly Release Notes v4.1.0.15

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.1.0.15](https://github.com/streamnative/pulsar/releases/tag/v4.1.0.15)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.1.0.15/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.1.0.15/images/sha256-586e3a6ba446656fc5c245b3a2f9679ef906f0978e08b937347b1a7cf0bac016)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.1.0.15/images/sha256-a040915246b6057729f60e144d212fbbf1f6f30d94b8b37c4b58e18755ab592d)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.1.0.15/images/sha256-a040915246b6057729f60e144d212fbbf1f6f30d94b8b37c4b58e18755ab592d)

## General Changes

### Apache Pulsar

([#25177](https://github.com/apache/pulsar/pull/25177)) \[fix]\[ml] Fix NoSuchElementException in EntryCountEstimator caused by a race condition

([#25166](https://github.com/apache/pulsar/pull/25166)) \[improve]\[broker] Upgrade bookkeeper to 4.17.3

([#25132](https://github.com/apache/pulsar/pull/25132)) \[improve]\[broker] Ensure metadata session state visibility and improve Unstable observability for ServiceUnitStateChannelImpl

([#25070](https://github.com/apache/pulsar/pull/25070)) \[improve]\[broker] PIP-442: Add memory limits for topic list watcher (part 2)

([#25157](https://github.com/apache/pulsar/pull/25157)) \[fix]\[fn] Fix graceful Pulsar Function shutdown so that consumers and producers are closed

([#25151](https://github.com/apache/pulsar/pull/25151)) \[fix]\[broker] Fence reset cursor by timestamp to avoid concurrent timestamp-based position lookups

([#25148](https://github.com/apache/pulsar/pull/25148)) \[fix]\[ml] Retry offload reads when OffloadReadHandleClosedException is encountered

([#25149](https://github.com/apache/pulsar/pull/25149)) \[fix]\[admin] Fix offload policy incompatible issue.

([#25142](https://github.com/apache/pulsar/pull/25142)) \[fix]\[proxy] Fix memory leaks in ParserProxyHandler

([#25140](https://github.com/apache/pulsar/pull/25140)) \[fix]\[fn] complete flushAsync before closeAsync in ProducerCache and wait for completion in closing the cache

([#25031](https://github.com/apache/pulsar/pull/25031)) \[fix]\[broker] Avoid split non-existent bundle

### KoP

Fix missing fallback when oxiaSchemaRegistryUrl is unset

Add warning logs and fliter out incorrect schemas returned by range scans in the OxiaSchemaStorage

### pulsarctl

fix: patch Go stdlib CVEs in pulsarctl (update to go 1.25.5)

### Cloud Pulsar Plugins

Add commons-lang dependency to sn-broker-inteceptors to fix the compile isuse

### Function Mesh Worker Service

\[branch-4.1] fix: apikeys auth handler uses incorrect issuer

### StreamNative Unified RBAC

fix the illegal license format

e41d505 Bump version to 1.7.3

b6e5296 Bump version to 1.7.2

cff40fa Bump version to 1.7.1

742bb20 fixes license and spotless

df7d236 Bump version to 1.7.0

fixes: workflow issue

feat: support schedule release for rbac maven sdk

## Security Fixes

### Apache Pulsar

([#25152](https://github.com/apache/pulsar/pull/25152)) \[fix]\[sec] Upgrade vertx to address CVE-2026-1002


# V4.1.0.16
Source: https://docs.streamnative.io/release-notes/pulsar/v4.1/v4.1.0.16



# StreamNative Weekly Release Notes v4.1.0.16

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.1.0.16](https://github.com/streamnative/pulsar/releases/tag/v4.1.0.16)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.1.0.16/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.1.0.16/images/sha256-d9aeb6a105c0cbe5749c0c0dd385afaf9bd76fe00b97dc4612f03938de1b0fe7)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.1.0.16/images/sha256-bfe8b9f124b9acbf9ded92007d4d6f42daaf7791760fdae6ff4a4c0fa8ce8ddd)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.1.0.16/images/sha256-bfe8b9f124b9acbf9ded92007d4d6f42daaf7791760fdae6ff4a4c0fa8ce8ddd)

## General Changes

### Apache Pulsar

([#25187)](https://github.com/apache/pulsar/pull/25187))) Revert "\[improve]\[meta] PIP-453: Improve the metadata store threading model

([#25231](https://github.com/apache/pulsar/pull/25231)) \[fix]\[broker] Fix transactionMetadataFuture completeExceptionally with null value

([#25229](https://github.com/apache/pulsar/pull/25229)) \[fix]\[client] Send all chunkMessageIds to broker for redelivery

([#25221](https://github.com/apache/pulsar/pull/25221)) \[improve]\[broker] Give the detail error msg when authenticate failed with AuthenticationException

([#25227](https://github.com/apache/pulsar/pull/25227)) \[fix]\[test] Fix Mockito stubbing race in TopicListServiceTest

([#25228](https://github.com/apache/pulsar/pull/25228)) \[fix]\[broker] Fix incomplete futures in topic property update/delete methods

([#25224](https://github.com/apache/pulsar/pull/25224)) \[improve]\[broker] Add idle timeout support for http

([#25052](https://github.com/apache/pulsar/pull/25052)) \[improve]\[client] Make authorization server metadata path configurable in AuthenticationOAuth2

([#24944](https://github.com/apache/pulsar/pull/24944)) \[feat]\[client] oauth2 trustcerts file and timeouts

([#25185](https://github.com/apache/pulsar/pull/25185)) \[improve]\[broker] Add strictAuthMethod to require explicit authentication method

([#25223](https://github.com/apache/pulsar/pull/25223)) \[fix]\[broker] Fix httpProxyTimeout config

([#25200](https://github.com/apache/pulsar/pull/25200)) \[improve]\[broker] Change log level from warn to debug when cursor mark-deleted position ledger doesn't exist

([#25195](https://github.com/apache/pulsar/pull/25195)) \[feat]\[io] implement pip-297 for jdbc sinks

([#25127](https://github.com/apache/pulsar/pull/25127)) \[improve]\[admin] Add client side looping to analyze-backlog in Topics to avoid potential HTTP call timeout

([#25188](https://github.com/apache/pulsar/pull/25188)) \[fix]\[broker] Prevent missed topic changes in topic watchers and schedule periodic refresh with patternAutoDiscoveryPeriod interval

([#25207](https://github.com/apache/pulsar/pull/25207)) \[fix]\[client] Fix producer synchronous retry handling in failPendingMessages method

([#25199](https://github.com/apache/pulsar/pull/25199)) \[fix]\[broker]Fix ledgerHandle failed to read by using new BK API

([#25165](https://github.com/apache/pulsar/pull/25165)) \[fix]\[broker] Fix ManagedCursorImpl.asyncDelete() method may lose previous async mark delete properties in race condition

([#25216](https://github.com/apache/pulsar/pull/25216)) \[fix]\[test]Fix flaky ExtensibleLoadManagerImplTest\_testGetMetrics

([#25211](https://github.com/apache/pulsar/pull/25211)) \[improve]\[proxy] Add regression tests for package upload with 'Expect: 100-continue'

([#24994](https://github.com/apache/pulsar/pull/24994)) \[improve]\[monitor] Upgrade OpenTelemetry to 1.56.0, Otel instrumentation to 2.21.0 and Otel semconv to 1.37.0

([#25187](https://github.com/apache/pulsar/pull/25187)) \[improve]\[meta] PIP-453: Improve the metadata store threading model

([#25208](https://github.com/apache/pulsar/pull/25208)) \[fix]\[client] Fix race condition between isDuplicate() and flushAsync() method in PersistentAcknowledgmentsGroupingTracker due to incorrect use Netty Recycler

([#25209](https://github.com/apache/pulsar/pull/25209)) \[fix] \[test] Upgrade docker-java to 3.7.0

([#25179](https://github.com/apache/pulsar/pull/25179)) \[fix]\[proxy] Close client connection immediately when credentials expire and forwardAuthorizationCredentials is disabled

([#25197](https://github.com/apache/pulsar/pull/25197)) \[fix]\[misc] Allow JWT tokens in OpenID auth without nbf claim

([#25186](https://github.com/apache/pulsar/pull/25186)) \[fix]\[test] Bump org.assertj:assertj-core from 3.27.5 to 3.27.7

([#25182](https://github.com/apache/pulsar/pull/25182)) \[improve]\[misc] Upgrade snappy version to 1.1.10.8

([#25178](https://github.com/apache/pulsar/pull/25178)) \[fix]\[client] ControlledClusterFailover avoid unnecessary reconnection.

([#25172](https://github.com/apache/pulsar/pull/25172)) \[improve]\[client]Reduce unnecessary getPartitionedTopicMetadata requests when using retry and DLQ topics.

### KoP

Fix cursor leak from KafkaTopicConsumerManager

Upgrade testcontainers and docker-java to address min api version issue

Fix list/rangeScan in OxiaSchemaStorage

Some operations can't work with super-user role

Fix race condition in concurrent Schema Registry requests handling

\[branch-4.1] Upgrade pulsar to 4.1.0.16

Add auth info for oxia configuration

\[branch-4.1] Upgrade unified rbac dependency to 1.7.3

Remove rbac download step when building schema registry image

Return references when getting schema by subject and version

Fix potential concurrent modification issue

Fix flaky test IdempotentProducerTest

### StreamNative Pulsar Plugins

7a182f825 Upgrade testcontainers and docker-java to address min api version issue

a5e38247f fix incompatible with pulsar

Upgrade detector build image to 1.25

9d4d519ed upgrade opentel version

1c273a437 build detector multi-platform

fix: patch CVE-2025-61726, CVE-2025-61728, CVE-2025-61730 in stdlib

Fix OIDCServlet to use local metadata store instead of configuration metadata store

fix: upgrade zookeeper to 3.9.4 to patch CVE-2025-58457

### pulsarctl

fix: upgrade Go to 1.25.7 to fix CVE-2025-68121

fix: upgrade Go from 1.25.5 to 1.25.6 to patch CVE-2025-61726, CVE-2025-61728, CVE-2025-61730

### Function Mesh Worker Service

06120fe2 Fix CI

Use FunctionWorker crd to deploy registry service in CI

Do not allow to update connection and packageConnection

Add integration tests and OpenAPI docs for registry service

Implement registry endpoint

41a7786f Fix CI

Reuse authorization service when possible

9643c58f Enhance CI

### StreamNative Tiered storage

a18ecdc0 Fix test

## Security Fixes

### Apache Pulsar

([#25095](https://github.com/apache/pulsar/pull/25095)) \[fix]\[sec] Upgrade jose4j to 0.9.6 to address CVE-2024-29371

([#25206](https://github.com/apache/pulsar/pull/25206)) \[fix]\[sec] Upgrade OpenSearch to 2.19.4 to remediate CVE-2025-9624

([#25198](https://github.com/apache/pulsar/pull/25198)) \[fix]\[sec] Exclude org.lz4:lz4-java and standardize on at.yawk.lz4-java to remediate CVE-2025-12183 and CVE-2025-66566

([#25175](https://github.com/apache/pulsar/pull/25175)) \[fix]\[sec] Bump org.apache.solr:solr-core from 9.8.0 to 9.10.1 in /pulsar-io/solr


# V4.1.0.2
Source: https://docs.streamnative.io/release-notes/pulsar/v4.1/v4.1.0.2



# StreamNative Weekly Release Notes v4.1.0.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.1.0.2](https://github.com/streamnative/pulsar/releases/tag/v4.1.0.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.1.0.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.1.0.2/images/sha256-5276c248983485b62232629f5412cfc1ddc56d97879e189b8cc9837d13f32ff2)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.1.0.2/images/sha256-9cc65a554b18f8a47bcf65f4b9a9c8268a535beb5823ff59e677226c8ea0f7f9)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.1.0.2/images/sha256-9cc65a554b18f8a47bcf65f4b9a9c8268a535beb5823ff59e677226c8ea0f7f9)

## General Changes

### Apache Pulsar

([#24752](https://github.com/apache/pulsar/pull/24752)) \[fix]\[client] rollback TopicListWatcher retry behavior

([#24698](https://github.com/apache/pulsar/pull/24698)) \[fix]\[client]TopicListWatcher not closed when calling PatternMultiTopicsConsumerImpl.closeAsync() method

([#24756](https://github.com/apache/pulsar/pull/24756)) \[fix]\[broker] Fix testServiceConfigurationRetentionPolicy unit test

([#24733](https://github.com/apache/pulsar/pull/24733)) \[improve]\[broker] Allow deletion of empty persistent topics regardless of retention policy

([#24696](https://github.com/apache/pulsar/pull/24696)) \[fix]\[broker]Fix dirty reading of namespace level offload thresholds

([#24683](https://github.com/apache/pulsar/pull/24683)) \[fix]\[broker]Fix the wrong logic of the test PartitionCreationTest.testCreateMissedPartitions

([#24634](https://github.com/apache/pulsar/pull/24634)) \[fix]\[broker]Dispatcher did unnecessary sort for recentlyJoinedConsumers and printed noisy error logs

([#24749](https://github.com/apache/pulsar/pull/24749)) \[fix] Exclude commons-lang dep from bookkeeper

([#24743](https://github.com/apache/pulsar/pull/24743)) \[fix]\[client] Fix receiver queue auto-scale without memory limit

([#24742](https://github.com/apache/pulsar/pull/24742)) \[improve]\[build] Upgrade Apache Parent POM to version 35

([#24722](https://github.com/apache/pulsar/pull/24722)) \[fix]\[ml] Negative backlog & acked positions does not exist & message lost when concurrently occupying topic owner

([#24730](https://github.com/apache/pulsar/pull/24730)) \[fix]\[broker] Ensure KeyShared sticky mode consumer respects assigned ranges

([#24736](https://github.com/apache/pulsar/pull/24736)) \[fix]\[broker] Key\_Shared subscription doesn't always deliver messages from the replay queue after a consumer disconnects and leaves a backlog

([#24731](https://github.com/apache/pulsar/pull/24731)) \[fix]\[broker] Fix cannot shutdown broker gracefully by admin api

([#24654](https://github.com/apache/pulsar/pull/24654)) \[fix]\[io] Improve Kafka Connect source offset flushing logic

([#24725](https://github.com/apache/pulsar/pull/24725)) \[fix]\[client] Avoid recycling the same ConcurrentBitSetRecyclable among different threads

([#24721](https://github.com/apache/pulsar/pull/24721)) \[feat]\[fn] Fallback to using `STATE_STORAGE_SERVICE_URL` in `PulsarMetadataStateStoreProviderImpl.init`

([#24580](https://github.com/apache/pulsar/pull/24580)) \[fix]\[broker]Fix never recovered metadata store bad version issue if received a large response from ZK

### KoP

Handle unexpected exception in decode for safe producer state recovery

Avoid blocking when the previous consumer closed without sending SyncGroup requests

### StreamNative Pulsar Plugins

Fix LicenseAdditionalServletTest

Use snstage/pulsar image to integration test

Removed pinned version for nimbus-jose-jwt-9.37.2

Pin Netty version for bookie\_rackinfo

Fix BK exclusions to avoid commons-configuration and commons-langs deps

Fix sn bom plugin typo

### pulsarctl

Preserve tenant fields on partial update

upgrade client go version to 0.16.0

### Function Mesh Worker Service

Support trigger agent function with properties

Update function-mesh version to v0.25.0 in pom.xml

Remove ConnectRestException from mesh-worker-common module

Support input-type-class and output-type-class arguments for Functions

Support set extra env for kafka connect

Support streamable http for AgentFunction and make trigger timeout value configurable

### StreamNative Ursa storage

Use the azure latest image for testing

Remove parquet compression property from iceberg table

Fix topic properties can not be fetched on the serverless cluster

Fix cache reference staleness race condition in ObjectWalStorageImpl

Control sdt behavior by the properties

Fix offload cursor doesn't show in the consumer stats

Fix unity catalog can't convert decimal type issue.

Fix global open telemetry confict issue

Add detailed label for the compaction commit and error metrics

Fix record type with instant field write failed issue.

Close the writer when there is exceptions

Pause the reader when release it

Split offload cursor update interval

Fix metrics name

Add more metrics for the error, commit time

2e957f84 fix ci

Introduce exception code to improve the exception handling

Support disable the publish task by topic

Use latest snapshot version image to fix tests

## Security Fixes


# V4.1.0.3
Source: https://docs.streamnative.io/release-notes/pulsar/v4.1/v4.1.0.3



# StreamNative Weekly Release Notes v4.1.0.3

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.1.0.3](https://github.com/streamnative/pulsar/releases/tag/v4.1.0.3)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.1.0.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.1.0.3/images/sha256-6d1394bf412ab6bc19d49ab066c02db22010298c99ad250d8a9e840631ad752a)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.1.0.3/images/sha256-81eed97801d1f0fbcea315058637d36d8769bb142d62a325c3cf3efdbec5e98d)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.1.0.3/images/sha256-81eed97801d1f0fbcea315058637d36d8769bb142d62a325c3cf3efdbec5e98d)

## General Changes

### Apache Pulsar

([#24779](https://github.com/apache/pulsar/pull/24779)) Bump org.apache.zookeeper:zookeeper from 3.9.3 to 3.9.4

([#24596](https://github.com/apache/pulsar/pull/24596)) \[improve]\[broker]Call scheduleAtFixedRateNonConcurrently for scheduled tasks, instead of scheduleAtFixedRate

([#23386](https://github.com/apache/pulsar/pull/23386)) \[improve]\[broker] PIP-402: Optionally prevent role/originalPrincipal logging

([#24772](https://github.com/apache/pulsar/pull/24772)) \[fix]\[misc] Fix compareTo contract violation for NamespaceBundleStats, TimeAverageMessageData and ResourceUnitRanking

([#24753](https://github.com/apache/pulsar/pull/24753)) \[fix]\[ml]Fix EOFException after enabled topics offloading

([#24769](https://github.com/apache/pulsar/pull/24769)) \[fix]\[test] Flaky-test: BrokerServiceTest.testShutDownWithMaxConcurrentUnload

([#24764](https://github.com/apache/pulsar/pull/24764)) \[improve]\[build] Upgrade Mockito, AssertJ and ByteBuddy to fully support JDK25

([#24761](https://github.com/apache/pulsar/pull/24761)) \[fix]\[client] Exclude io.prometheus:simpleclient\_caffeine from client-side dependencies

([#24763](https://github.com/apache/pulsar/pull/24763)) \[improve]\[build] Upgrade Lombok to 1.18.42 to fully support JDK25

([#23634](https://github.com/apache/pulsar/pull/23634)) \[improve]\[broker] If there is a deadlock in the service, the probe should return a failure because the service may be unavailable

([#24738](https://github.com/apache/pulsar/pull/24738)) \[fix]\[broker] First entry will be skipped if opening NonDurableCursor while trimmed ledger is adding first entry.

([#24768](https://github.com/apache/pulsar/pull/24768)) \[improve]\[build] Upgrade SpotBugs to a version that supports JDK25

([#24767](https://github.com/apache/pulsar/pull/24767)) \[fix]\[ci] Fix CI for Java 25 including upgrade of Gradle Develocity Maven extension

([#24741](https://github.com/apache/pulsar/pull/24741)) \[fix]\[broker] Prevent unexpected recycle failure in dispatcher's read callback

### AoP

Fix configuration potential NPE in test

### KoP

Use new CompactedTopicUtils.asyncReadCompactedEntries API from apache/pulsar#24725

### StreamNative Pulsar Plugins

\[feat]\[topic-compaction-service] Implement clean expired  message during compaction

Fix gcs-connector cve

### Function Mesh Worker Service

Reject request when agent name is too long or sessionId is not valid

feat: support multiple mcp servers

### StreamNative Tiered storage

Yong/test branch 4.1

### StreamNative Unified RBAC

feat: treat NotFound exception as success for deleting

feat: upgrade sdk-go version to v0.14.0

Add conditions for Catalog, CC and CE

feat: support extract variable claim from token

### StreamNative Ursa storage

Fix catalog instance not shared

Move the iceberg table creation params to a common place

Fix the integration test mount jar issue.

fix: mask sensitive properties in CatalogKey toString to avoid credential leaks in logs

## Security Fixes


# V4.1.0.4
Source: https://docs.streamnative.io/release-notes/pulsar/v4.1/v4.1.0.4



# StreamNative Weekly Release Notes v4.1.0.4

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.1.0.4](https://github.com/streamnative/pulsar/releases/tag/v4.1.0.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.1.0.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.1.0.4/images/sha256-c49aecb7969caa30e419aff3e912004555aa8e6484ebe287980c5168df681e58)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.1.0.4/images/sha256-cfbadf1830b4d9c1675050d3162e5900f1f5f5f0f7d301eb7096799e05189ef8)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.1.0.4/images/sha256-cfbadf1830b4d9c1675050d3162e5900f1f5f5f0f7d301eb7096799e05189ef8)

## General Changes

### Apache Pulsar

([#24863](https://github.com/apache/pulsar/pull/24863)) \[fix]Fixed getChildren('/') on Oxia based provider

([#24852](https://github.com/apache/pulsar/pull/24852)) \[fix]\[ml] Fix `getNumberOfEntries` may point to deleted ledger

([#24855](https://github.com/apache/pulsar/pull/24855)) \[fix]\[ml] Fix ledger trimming race causing cursor to point to deleted ledgers

([#24838](https://github.com/apache/pulsar/pull/24838)) \[fix]\[broker] Ensure LoadSheddingTask is scheduled after metadata service is available again

([#24841](https://github.com/apache/pulsar/pull/24841)) \[improve]\[ci] Upgrade GitHub Actions workflows to use ubuntu-24.04

([#24840](https://github.com/apache/pulsar/pull/24840)) \[improve] Upgrade Alpine base image to 3.22 version

([#24836](https://github.com/apache/pulsar/pull/24836)) \[fix]\[ml] PIP-430: Fix concurrency issue in MessageMetadata caching and improve caching

([#24830](https://github.com/apache/pulsar/pull/24830)) \[fix]\[client] Fix getPendingQueueSize for PartitionedTopicProducerStatsRecorderImpl: avoid NPE and implement aggregation

([#24832](https://github.com/apache/pulsar/pull/24832)) \[fix] Fix mixed lookup/partition metadata requests causing reliability issues and incorrect responses

([#24821](https://github.com/apache/pulsar/pull/24821)) \[fix]\[test] Fixed nondeterministic JSON ordering in multiple tests

([#24798](https://github.com/apache/pulsar/pull/24798)) \[improve]\[client] PIP-420: Update the schema ID format

([#24825](https://github.com/apache/pulsar/pull/24825)) \[improve]\[broker] Cache last publish timestamp for idle topics to reduce storage reads

([#24815](https://github.com/apache/pulsar/pull/24815)) \[feat]\[monitor] Add ML write latency histogram and entry size histogram as OTel metrics

([#24829](https://github.com/apache/pulsar/pull/24829)) \[fix]\[broker] Allow intermittent error from topic policies service when loading topics

([#24822](https://github.com/apache/pulsar/pull/24822)) \[fix]\[client] Make auto partitions update work for old brokers without PIP-344

([#24810](https://github.com/apache/pulsar/pull/24810)) \[feat]\[monitor] Add publish latency histogram as OTel metrics

([#24823](https://github.com/apache/pulsar/pull/24823)) \[fix]\[test] Fix flaky SingleThreadNonConcurrentFixedRateSchedulerTest.testPeriodicTaskCancellation

([#24824](https://github.com/apache/pulsar/pull/24824)) \[improve]\[ml] Upgrade Oxia client to 0.7.0

([#24790](https://github.com/apache/pulsar/pull/24790)) \[feat]\[client] Implement PIP-234 for sharing thread pools and DNS resolver/cache across multiple Pulsar Client instances

([#24734](https://github.com/apache/pulsar/pull/24734)) \[fix]\[client] Fix PulsarAdmin description check and add test

([#24729](https://github.com/apache/pulsar/pull/24729)) \[improve]\[client] Allow adding custom description to User-Agent header

([#24728](https://github.com/apache/pulsar/pull/24728)) \[fix]\[client] Add description method to ClientBuilder

([#24812](https://github.com/apache/pulsar/pull/24812)) \[fix]\[build] Remove invalid profile in settings.xml that caused gpg signing to fail

([#24811](https://github.com/apache/pulsar/pull/24811)) \[fix]\[build] Fix maven deploy with maven-source-plugin 3.3.1

([#24801](https://github.com/apache/pulsar/pull/24801)) \[improve]\[broker]Improve NamespaceService log that is printed when cluster was removed

([#24785](https://github.com/apache/pulsar/pull/24785)) \[fix]\[broker] Fix incorrect topic loading latency metric and timeout might not be respected

([#24770](https://github.com/apache/pulsar/pull/24770)) \[fix]\[broker] Flaky-test: ExtensibleLoadManagerImplTest.testDisableBroker

([#24784](https://github.com/apache/pulsar/pull/24784)) \[improve]\[client/broker] Add DnsResolverGroup to share DNS cache across multiple PulsarClient instances

([#24780](https://github.com/apache/pulsar/pull/24780)) \[improve]\[broker] Replace isServiceUnitActiveAsync with checkTopicNsOwnership

([#24813](https://github.com/apache/pulsar/pull/24813)) \[fix] Update gRPC to 1.75.0

### AoP

74655b2 Use commons-lang3

### MoP

a57d06db update pulsar and sn dependency version

### KoP

Handle invalid topic format error for metadata request

feat(kafka-admin): Support describing topic configs with `kop.kafka.` prefix

Use the Pulsar format for the Kafka consumer offsets topic

fix(kafka-impl): correct topic partition extraction from Pulsar topic names

Reduce spamming logs from schema registry and topic lookup

\[Ursa] Fix the wrong position comparison in topic replay

492502d4a Bump version to 4.1.0.4

Add producer ID expiration mechanism for classic engine

Fix released produce request buffer could be accessed

Make avro-maven-plugin version consistent with avro

Use new APIs from Oxia 0.7.0

Write transaction log and offset log with partition log

\[schema-registry] add option to disable compatibility mode configuration

Add more info to logs when the connection is closed

Fix direct memory oom with ack=0

\[Ursa] Improve producer state snapshot taking to avoid metadata thread

Refactor schema provider to use Confluent avro schema provider

\[Ursa] Remove oldest producers when the serialized producer state is too large

Support Pulsar and Kafka client communicate with the Kafka schema registry

Support parse googles built in proto files on protobuf schema

Fix idempotent producer for classic engine

Improve topic loading time by skipping Pulsar message deduplication recovery

Add test cases for record schema validation with subject name strategy

### StreamNative Pulsar Plugins

Upgrade commons-lang to commons-lang3

Upgrade Oxia to 0.7.0

Upgrade zk version to 3.9.4 to fix CVE

### Cloud Pulsar Plugins

5fd37f2 Use commons-lang3

### Function Mesh Worker Service

Use sn-operator to deploy pulsar cluster in CI

fix agent cannot update some fields error

Fix ci failure

fix build and license header

### StreamNative Tiered storage

Use lang3 package StringUtils to replace lang package StringUtils

Upgrade aws sdk to 2.32.28 to keep sync with Pulsar

### StreamNative Ursa storage

Respect markDeletedOffsets when reading entries from PersistStorageApi

Fix pulsar expired ledger not deleted bug

Carry the number of messages info for positions in Entry

Fix delta not support timestamp\_ntz

Do iceberg schema compatibility check before updating schema

Use strict match rule to fetch token from UnityCatalogSasTokenProviderTest

Disable flaky AsyncCleanerTest#simpleTest

Fix parquet reader handle union type bug

fix: support nested field partitioning in Iceberg tables

Remove METADATA\_DELETE\_AFTER\_COMMIT\_ENABLED property

Skip remove preserved properties

Fix catalog close bug

Add more log for parse entry

Move the schema evolution logic into the iceberg table

Upgrade Oxia client to 0.7.0

Bump org.apache.commons:commons-lang3 from 3.17.0 to 3.18.0

Add more admin commands

Abstract the schema evolution parts in the encoder

Removed dependency on commons-lang 2.6

## Security Fixes


# V4.1.0.5
Source: https://docs.streamnative.io/release-notes/pulsar/v4.1/v4.1.0.5



# StreamNative Weekly Release Notes v4.1.0.5

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.1.0.5](https://github.com/streamnative/pulsar/releases/tag/v4.1.0.5)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.1.0.5/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.1.0.5/images/sha256-ee5048388a6a24b8fa2638f9ed4b2f1149ff53bd436556fb28e211a0897b68a3)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.1.0.5/images/sha256-b84ca4b360768d1bb4e4239ac361aff0279365ee133001af63534b986e400dcc)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.1.0.5/images/sha256-b84ca4b360768d1bb4e4239ac361aff0279365ee133001af63534b986e400dcc)

## General Changes

### Apache Pulsar

([#24865](https://github.com/apache/pulsar/pull/24865)) \[fix]\[test] Fix flaky SubscriptionSeekTest.testSeekWillNotEncounteredFencedError by counting subscription is fenced only after seek

([#24861](https://github.com/apache/pulsar/pull/24861)) \[fix]\[test] Stabilize SequenceIdWithErrorTest by fencing after first publish to avoid empty-ledger deletion and send timeout

([#24881](https://github.com/apache/pulsar/pull/24881)) \[improve]\[broker]Skip to mark delete if the target position of expira…

([#24880](https://github.com/apache/pulsar/pull/24880)) \[fix]\[broker] Stop to retry to read entries if the replicator has terminated

### KoP

\[schema-registry] Add validation to reject Avro schemas with union and null types

\[Ursa] Fix messages could be returned when the fetch offset exceeds the last offset

Improve logs for schema registry and request handling

Fix flaky-test: SimpleLoadBalanceTest.testBrokerRestart

### Function Mesh Worker Service

Use new way to build sn-operator image

## Security Fixes


# V4.1.0.6
Source: https://docs.streamnative.io/release-notes/pulsar/v4.1/v4.1.0.6



# StreamNative Weekly Release Notes v4.1.0.6

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.1.0.6](https://github.com/streamnative/pulsar/releases/tag/v4.1.0.6)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.1.0.6/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.1.0.6/images/sha256-9afd8df965ebeba220c2d76b30c32efdb4c9d885ba67667c8e627f5bd4dcdef9)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.1.0.6/images/sha256-1b1e242a2466596f69e9c810c20fc81c6642ebfa9b511e30663b941e902932ca)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.1.0.6/images/sha256-1b1e242a2466596f69e9c810c20fc81c6642ebfa9b511e30663b941e902932ca)

## General Changes

### Apache Pulsar

([#24794](https://github.com/apache/pulsar/pull/24794)) \[fix]\[client] Fix thread leak in reloadLookUp method which is used by ServiceUrlProvider

([#24859](https://github.com/apache/pulsar/pull/24859)) \[fix]\[broker] Run ResourceGroup tasks only when tenants/namespaces registered

([#24915](https://github.com/apache/pulsar/pull/24915)) \[fix]\[broker] BacklogMessageAge is not reset when cursor mdPosition is on an open ledger

([#24860](https://github.com/apache/pulsar/pull/24860)) \[fix]\[broker] Fix wrong behaviour when using namespace.allowed\_clusters, such as namespace deletion and namespace policies updating

([#24917](https://github.com/apache/pulsar/pull/24917)) \[improve]\[ci] Move replication tests to new group Broker Group 5 in Pulsar CI

([#24911](https://github.com/apache/pulsar/pull/24911)) \[improve]\[misc] Upgrade Netty to 4.1.128.Final

([#24904](https://github.com/apache/pulsar/pull/24904)) \[fix]\[test] Fix flaky ReplicatorTest.testResumptionAfterBacklogRelaxed

([#24895](https://github.com/apache/pulsar/pull/24895)) \[improve]\[broker] Reduce the broker close time to avoid useless wait for event loop shutdown

([#24885](https://github.com/apache/pulsar/pull/24885)) \[fix]\[broker] Fix totalAvailablePermits not reduced when removing consumer from non-persistent dispatcher

([#24858](https://github.com/apache/pulsar/pull/24858)) \[fix]\[test]fix flaky SimpleProducerConsumerTest.testReceiveAsyncCompletedWhenClosing

([#24896](https://github.com/apache/pulsar/pull/24896)) \[improve]\[io] Upgrade Debezium version to 3.2.4.Final

([#24807](https://github.com/apache/pulsar/pull/24807)) \[fix]\[test] Made ProtobufSchemaTest.testParsingInfoProperty order-independent

([#24848](https://github.com/apache/pulsar/pull/24848)) \[improve]\[client]Add null check for Pulsar client clock configuration

([#24887](https://github.com/apache/pulsar/pull/24887)) \[fix]\[test] BacklogQuotaManagerTest.backlogsAgeMetricsNoPreciseWithoutBacklogQuota handle empty /metrics scrape

([#24826](https://github.com/apache/pulsar/pull/24826)) \[fix]\[broker] Flaky-test: TopicTransactionBufferTest.testMessagePublishInOrder

([#24864](https://github.com/apache/pulsar/pull/24864)) \[fix]\[test] Stabilize PublishRateLimiterOverconsumingTest by aligning measurement and using adjacent 2-sec averages

([#24799](https://github.com/apache/pulsar/pull/24799)) \[improve]\[broker] Part-2 of PIP-434: Use ServerCnxThrottleTracker, instead of modifying channel.readable directly

([#24854](https://github.com/apache/pulsar/pull/24854)) \[fix]\[test] Fix flaky LookupPropertiesTest.testConcurrentLookupProperties

([#24800](https://github.com/apache/pulsar/pull/24800)) \[improve]\[broker] PIP-434: add configurations to broker.conf

([#24423](https://github.com/apache/pulsar/pull/24423)) \[improve]\[broker] Part-1 of PIP-434: Expose Netty channel configuration WRITE\_BUFFER\_WATER\_MARK to pulsar conf and pause receive requests when channel is unwritable

### KoP

Fix Ursa storage in NamespaceBundleOwnershipListener unload behavior

([#1591)](https://github.com/streamnative/ksn/pull/1591))) Revert "Schedule unload group metadata if not owner broker

8053bbb39 Bump version to 4.1.0.6

Fix broken build due to rate limiter change and TransportCnx interface change from Pulsar

Add producer ID expiration mechanism for Ursa engine

Remove "Found owner" logs with low lookup latency (\< 10ms)

Schedule unload group metadata if not owner broker

Don't return non-retriable error in metadata response when lookup fails

\[SchemaRegistry] Add `ALWAYS_INCOMPATIBLE` compatibility mode

Fix OffsetCommit request might not take effect unless reloading from metadata store

Support client side retry for temporary metadata or write failures

Support returning partition count in create topics response

Bump spotbugs version to 4.9.8

Fix topic validation to enforce kafka topic naming conventions

Reduce the protocol handler close time by closing built-in clients asynchronously

Improve group metadata immigration logic by replacing admin client usage with namespace service calls

### StreamNative Pulsar Plugins

Fix detector deadlock when subDetector panics or fails

### pulsarctl

upgrade golang to 1.24.9

### StreamNative Unified RBAC

feat: improve sdk-java enforce performance

## Security Fixes

### Apache Pulsar

([#24923](https://github.com/apache/pulsar/pull/24923)) \[fix]\[sec] Upgrade BouncyCastle FIPS to 2.0.10 to remediate CVE-2025-8916

([#24903](https://github.com/apache/pulsar/pull/24903)) \[fix]\[sec] Upgrade Spring to 6.2.12 to remediate CVE-2025-22233 and CVE-2025-41249

([#24897](https://github.com/apache/pulsar/pull/24897)) \[fix]\[sec] Upgrade Jetty to 9.4.58.v20250814 to address CVE-2025-5115

([#24889](https://github.com/apache/pulsar/pull/24889)) \[fix]\[sec] Bump io.vertx:vertx-web from 4.5.10 to 4.5.22


# V4.1.0.7
Source: https://docs.streamnative.io/release-notes/pulsar/v4.1/v4.1.0.7



# StreamNative Weekly Release Notes v4.1.0.7

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.1.0.7](https://github.com/streamnative/pulsar/releases/tag/v4.1.0.7)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.1.0.7/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.1.0.7/images/sha256-52a093183b6f1721004a1a401ed8191e6eb5604fd2bf9a03678c37514222e6a9)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.1.0.7/images/sha256-9ec9e013b7e03c137f9c3c1269edb27e5900f3578826e4d9402d35484fed45c8)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.1.0.7/images/sha256-9ec9e013b7e03c137f9c3c1269edb27e5900f3578826e4d9402d35484fed45c8)

## General Changes

### Apache Pulsar

([#24976](https://github.com/apache/pulsar/pull/24976)) \[feat]\[meta] upgrade oxia version to 0.7.2

([#24971](https://github.com/apache/pulsar/pull/24971)) \[fix]\[broker]Leaving orphan schemas and topic-level policies after partitioned topic is deleted by GC

([#24805](https://github.com/apache/pulsar/pull/24805)) \[fix]\[test] Made ProtobufNativeSchemaTest.testSchema order-independent

([#24962](https://github.com/apache/pulsar/pull/24962)) \[improve]\[client] Deduplicate getTopicsUnderNamespace in BinaryProtoLookupService

([#24972](https://github.com/apache/pulsar/pull/24972)) \[fix]\[test] Add Delta Tolerance in Double-Precision Assertions to Fix Rounding Flakiness

([#24872](https://github.com/apache/pulsar/pull/24872)) \[fix]\[test] Fixed ResponseBody Check in Test Helper

([#24969](https://github.com/apache/pulsar/pull/24969)) \[fix]\[test] Fixed Nondeterministic Ordering in SchemaInfoTest

([#24965](https://github.com/apache/pulsar/pull/24965)) \[fix]\[client] Fix deduplication for getPartitionedTopicMetadata to include method parameters

([#24945](https://github.com/apache/pulsar/pull/24945)) \[fix]\[broker]Transactional messages can never be sent successfully if concurrently taking transaction buffer snapshot

([#24955](https://github.com/apache/pulsar/pull/24955)) \[fix]\[test] Fix flaky KeySharedSubscriptionBrokerCacheTest.testReplayQueueReadsGettingCached

([#24957](https://github.com/apache/pulsar/pull/24957)) \[fix]\[test] Fix invalid test NonPersistentTopicTest.testProducerRateLimit

([#24833](https://github.com/apache/pulsar/pull/24833)) \[feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace

([#24952](https://github.com/apache/pulsar/pull/24952)) \[improve]\[fn] Use PulsarByteBufAllocator.DEFAULT instead of ByteBufAllocator.DEFAULT

([#24958](https://github.com/apache/pulsar/pull/24958)) \[cleanup]\[broker] Remove unused configuration maxMessageSizeCheckIntervalInSeconds

([#24954](https://github.com/apache/pulsar/pull/24954)) \[fix]\[broker] AvgShedder comparison error

([#24951](https://github.com/apache/pulsar/pull/24951)) \[fix]\[test] Fix flaky NonPersistentTopicTest.testProducerRateLimit

([#24802](https://github.com/apache/pulsar/pull/24802)) \[fix]\[broker] Trigger topic creation event only once for non-existent topic

([#24948](https://github.com/apache/pulsar/pull/24948)) \[improve]\[test] Disable flaky PatternConsumerBackPressureTest until the problem is fixed

([#23551](https://github.com/apache/pulsar/pull/23551)) \[fix]\[txn] fix concurrent error cause txn stuck in TransactionBufferHandlerImpl#endTxn

([#24939](https://github.com/apache/pulsar/pull/24939)) \[fix]\[broker] Avoid recursive update in ConcurrentHashMap during policy cache cleanup

([#24787](https://github.com/apache/pulsar/pull/24787)) \[improve]\[broker] Add tests for using absolute FQDN for advertisedAddress and remove extra dot from brokerId

([#24893](https://github.com/apache/pulsar/pull/24893)) \[feat]\[client] PIP-234: Support shared resources in PulsarAdmin to reduce thread usage

([#24762](https://github.com/apache/pulsar/pull/24762)) \[fix]\[admin] Set local policies overwrites "number of bundles" passed during namespace creation

([#24941](https://github.com/apache/pulsar/pull/24941)) \[fix]\[broker] Fix bug in PersistentMessageExpiryMonitor which blocked further expirations

([#24929](https://github.com/apache/pulsar/pull/24929)) \[fix]\[test] Stabilize testMsgDropStat by reliably triggering non-persistent publisher drop

([#24934](https://github.com/apache/pulsar/pull/24934)) \[fix]\[broker] Fix stack overflow caused by race condition when closing a connection

([#24932](https://github.com/apache/pulsar/pull/24932)) \[fix]\[broker] ExtensibleLoadManager: handle SessionReestablished and Reconnected events to re-register broker metadata

([#24933](https://github.com/apache/pulsar/pull/24933)) \[fix]\[broker] Use `poll` instead `remove` to avoid `NoSuchElementException`

([#24898](https://github.com/apache/pulsar/pull/24898)) \[fix]\[broker] fix getMaxReadPosition in TransactionBufferDisable should return latest

([#24943](https://github.com/apache/pulsar/pull/24943)) \[improve]\[broker] Don't log an error when updatePartitionedTopic is called on a non-partitioned topic

([#24942](https://github.com/apache/pulsar/pull/24942)) \[improve]\[broker] Optimize lookup result warn log

### KoP

Fix incorrect version ID displayed in schema retrieval log

Add log to trace existing schema retrieval

### Function Mesh Worker Service

add kafka managed auth data annotation

bump funciton-mesh v0.26.0

bump function-mesh to 0.26.0

feat: make function api pass sink and source config

Add retry for ci

### StreamNative Tiered storage

Move the unsupported handler to info level

## Security Fixes

### Apache Pulsar

([#24953](https://github.com/apache/pulsar/pull/24953)) \[fix]\[sec] Update Hbase version to 2.6.3-hadoop3 and exclude Avro from hbase-client to remediate CVEs

([#24949](https://github.com/apache/pulsar/pull/24949)) \[fix]\[sec] Added Exclusions for tomcat-embed-core and derby and override mina-core to remediate CVEs

([#24950](https://github.com/apache/pulsar/pull/24950)) \[fix]\[sec] Upgrade hadoop3 version from 3.4.0 to 3.4.1

([#24937](https://github.com/apache/pulsar/pull/24937)) \[fix]\[sec] Override nimbus-jose-jwt to remediate CVE-2023-52428 and CVE-2025-53864

([#24936](https://github.com/apache/pulsar/pull/24936)) \[fix]\[sec] Override commons-beanutils and commons-configuration2 to remediate CVEs

([#24935](https://github.com/apache/pulsar/pull/24935)) \[fix]\[sec] Override kafka-clients in kinesis-kpl-shaded to remediate CVE-2024-31141 and CVE-2025-27817


# V4.1.0.8
Source: https://docs.streamnative.io/release-notes/pulsar/v4.1/v4.1.0.8



# StreamNative Weekly Release Notes v4.1.0.8

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.1.0.8](https://github.com/streamnative/pulsar/releases/tag/v4.1.0.8)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.1.0.8/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.1.0.8/images/sha256-13caf8ac74bfa1b8a1e561cf756b040bfcbe24cd51f1a5776a85c272bf5e9ad7)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.1.0.8/images/sha256-0e806ea9c73517afe8c029721e42908a085edd8f47bc1154105161d5a33f0558)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.1.0.8/images/sha256-0e806ea9c73517afe8c029721e42908a085edd8f47bc1154105161d5a33f0558)

## General Changes

### Apache Pulsar

([#24997](https://github.com/apache/pulsar/pull/24997)) \[fix]\[broker] Fix creation of replicated subscriptions for partitioned topics

([#24983](https://github.com/apache/pulsar/pull/24983)) \[improve] Upgrade Apache Commons library versions

([#24995](https://github.com/apache/pulsar/pull/24995)) \[improve]\[test] Use Oxia project docker container for integration tests

([#24986](https://github.com/apache/pulsar/pull/24986)) \[fix] Handle TLS close\_notify to avoid SslClosedEngineException: SSLEngine closed already

([#24982](https://github.com/apache/pulsar/pull/24982)) \[improve]\[build] Upgrade Testcontainers to 1.21.3

([#24975](https://github.com/apache/pulsar/pull/24975)) \[improve]\[broker]Improve error response of failed to delete topic if it has replicators connected

([#24938](https://github.com/apache/pulsar/pull/24938)) \[fix]\[broker]Wrong backlog: expected 0 but got 1

([#24985](https://github.com/apache/pulsar/pull/24985)) \[improve] Upgrade Log4j2 to 2.25.2 and slf4j to 2.0.17

([#24984](https://github.com/apache/pulsar/pull/24984)) \[improve] Upgrade Caffeine to 3.2.3

([#24871](https://github.com/apache/pulsar/pull/24871)) \[fix]\[test] Fixed Non-Guaranteed Order in PoliciesDataTest.propertyAdmin

([#24981](https://github.com/apache/pulsar/pull/24981)) \[fix]\[build] Remove Confluent and Restlet maven repositories from top level pom.xml

### KoP

4ce40f39a Bump version to 4.1.0.8

Fix ProducerStateManager last mapped offset

Improve logging when OutOfOrderSequenceException happens

Prevent out-of-order messages caused by asynchronous authorization

Fix snapshot might not be taken when using system topic for producer state

Fix flaky testCommitOffsetsForMultiPartitions

Use `KopTopicTransactionBufferProvider` by default when transaction coordinator is enabled

Fix producer state snapshot

Add kop transaction buffer provider to disable transaction buffer recover for Kafka system topics

feat(kafka): Replace random UUID with Kafka Uuid for topic identification

Align Kafka version to 3.9.1 and replace removed kafka.admin.ConsumerGroupCommand in tests with Kafka Admin client

Support cluster-level schema validation

\[Ursa] Allow client to retry send when the producer state recovery fails

Schedule unload group metadata if not owner broker

### pulsarctl

Fix CVE CVE-2025-63811

### Function Mesh Worker Service

Fix ci

Support unified-rbac for all components

## Security Fixes

### Apache Pulsar

([#24987](https://github.com/apache/pulsar/pull/24987)) \[fix]\[sec] Bump github.com/dvsekhvalnov/jose2go from 1.6.0 to 1.7.0 in /pulsar-function-go


# V4.1.0.9
Source: https://docs.streamnative.io/release-notes/pulsar/v4.1/v4.1.0.9



# StreamNative Weekly Release Notes v4.1.0.9

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.1.0.9](https://github.com/streamnative/pulsar/releases/tag/v4.1.0.9)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.1.0.9/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.1.0.9/images/sha256-7e62f580c7ff8bc7c4edb5680a5bff223c4f604d822118563f448588638b2cf7)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.1.0.9/images/sha256-a9f2531c5f7a696c4a04ebaf714030b972099d329279cb13357a058557614c11)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.1.0.9/images/sha256-a9f2531c5f7a696c4a04ebaf714030b972099d329279cb13357a058557614c11)

## General Changes

### Apache Pulsar

([#24994)](https://github.com/apache/pulsar/pull/24994))) Revert "\[improve]\[monitor] Upgrade OpenTelemetry to 1.56.0, Otel instrumentation to 2.21.0 and Otel semconv to 1.37.0

([#25022](https://github.com/apache/pulsar/pull/25022)) \[fix] Upgrade gson to 2.13.2

([#25018](https://github.com/apache/pulsar/pull/25018)) \[improve]\[broker]Remove the warn log that frequently prints

([#25016](https://github.com/apache/pulsar/pull/25016)) \[fix]\[broker]Fix memory leak when using a customized ManagedLedger implementation

([#25015](https://github.com/apache/pulsar/pull/25015)) \[fix]\[client] Fix AutoProduceBytesSchema.clone() method

([#25014](https://github.com/apache/pulsar/pull/25014)) \[fix]\[client] Fix thread-safety of AutoProduceBytesSchema

([#25013](https://github.com/apache/pulsar/pull/25013)) \[improve]\[client] Test no exception could be thrown for invalid epoch in message

([#25011](https://github.com/apache/pulsar/pull/25011)) \[improve] Eliminate unnecessary duplicate schema lookups for partitioned topics in client and geo-replication

([#25004](https://github.com/apache/pulsar/pull/25004)) \[fix]\[broker] Add schema version in rest produce api

([#25012](https://github.com/apache/pulsar/pull/25012)) \[fix]\[broker] Fix issue with schemaValidationEnforced in geo-replication

([#25008](https://github.com/apache/pulsar/pull/25008)) \[fix]\[client] Fix double recycling of the message in isValidConsumerEpoch method

([#25007](https://github.com/apache/pulsar/pull/25007)) \[fix]\[client] PIP-84: Skip processing a message in the message listener if the consumer epoch is no longer valid

([#25006](https://github.com/apache/pulsar/pull/25006)) \[fix]\[client] Skip processing messages in the listener when the consumer has been closed

([#24994](https://github.com/apache/pulsar/pull/24994)) \[improve]\[monitor] Upgrade OpenTelemetry to 1.56.0, Otel instrumentation to 2.21.0 and Otel semconv to 1.37.0

### KoP

([#1644)](https://github.com/streamnative/ksn/pull/1644))) Revert "Fix breaking changes of latest 4.2.0-SNAPSHOT

095fc4875 Bump version to 4.1.0.9

Fix breaking changes of latest 4.2.0-SNAPSHOT

feat(kafka-config): support for kop\_kafka\_\* style configuration keys

Return UNKNOWN\_SERVER\_ERROR when topic fails to delete due to metadata store error

Fix flaky test `KafkaRbacCompatibilityAuthorizationTest`

Use metadata store to store KSN producer state snapshot by default

### StreamNative Pulsar Plugins

Update jose2go for CVE-2025-63811

### Cloud Pulsar Plugins

support

### StreamNative Unified RBAC

fixes: the invalid token exception not have been caught

## Security Fixes

### Apache Pulsar

([#25024](https://github.com/apache/pulsar/pull/25024)) \[fix]\[sec] Eliminate commons-collections dependency


# V4.1.3.1
Source: https://docs.streamnative.io/release-notes/pulsar/v4.1/v4.1.3.1



# StreamNative Weekly Release Notes v4.1.3.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.1.3.1](https://github.com/streamnative/pulsar/releases/tag/v4.1.3.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.1.3.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.1.3.1/images/sha256-390047b8c922594631de007d0aeace9f232117c30b7f36930b52be21c7ad0883)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.1.3.1/images/sha256-6ec2ee8552da73f89b7e3056a3faa812759346351804a5e954f4df507da5ef55)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.1.3.1/images/sha256-6ec2ee8552da73f89b7e3056a3faa812759346351804a5e954f4df507da5ef55)

## General Changes

### Apache Pulsar

([#25269](https://github.com/apache/pulsar/pull/25269)) \[improve]\[broker] Optimize AsyncTokenBucket overflow solution further to reduce fallback to BigInteger

([#25262](https://github.com/apache/pulsar/pull/25262)) \[fix]\[broker] Guard AsyncTokenBucket against long overflow

([#25255](https://github.com/apache/pulsar/pull/25255)) \[fix]\[broker] Use compatible Avro name validator in JsonSchemaCompatibilityCheck

([#25193](https://github.com/apache/pulsar/pull/25193)) \[fix]\[broker] Use compatible Avro name validator to allow '\$' in schema record names

([#25254](https://github.com/apache/pulsar/pull/25254)) \[fix]\[client] Reduce logging in OAuth auth to fix parsing of Pulsar cli command output

([#25253](https://github.com/apache/pulsar/pull/25253)) \[improve] Upgrade RoaringBitmap to 1.6.9 version

([#25251](https://github.com/apache/pulsar/pull/25251)) \[improve]\[fn] Upgrade Pulsar Python client version to 3.10.0

([#25246](https://github.com/apache/pulsar/pull/25246)) \[fix]\[meta] Metadata cache refresh might not take effect

([#25247](https://github.com/apache/pulsar/pull/25247)) \[fix]\[test] Fix ResourceQuotaCalculatorImplTest#testNeedToReportLocalUsage

([#25241](https://github.com/apache/pulsar/pull/25241)) \[fix]\[test] fix testBatchMetadataStoreMetrics.

([#25232](https://github.com/apache/pulsar/pull/25232)) \[improve] Upgrade Netty to 4.1.131.Final

([#25187)](https://github.com/apache/pulsar/pull/25187))) Reapply "\[improve]\[meta] PIP-453: Improve the metadata store threading model

([#25187)](https://github.com/apache/pulsar/pull/25187))) Revert "\[improve]\[meta] PIP-453: Improve the metadata store threading model

([#25231](https://github.com/apache/pulsar/pull/25231)) \[fix]\[broker] Fix transactionMetadataFuture completeExceptionally with null value

([#25229](https://github.com/apache/pulsar/pull/25229)) \[fix]\[client] Send all chunkMessageIds to broker for redelivery

([#25221](https://github.com/apache/pulsar/pull/25221)) \[improve]\[broker] Give the detail error msg when authenticate failed with AuthenticationException

([#25227](https://github.com/apache/pulsar/pull/25227)) \[fix]\[test] Fix Mockito stubbing race in TopicListServiceTest

([#25228](https://github.com/apache/pulsar/pull/25228)) \[fix]\[broker] Fix incomplete futures in topic property update/delete methods

([#25224](https://github.com/apache/pulsar/pull/25224)) \[improve]\[broker] Add idle timeout support for http

([#25052](https://github.com/apache/pulsar/pull/25052)) \[improve]\[client] Make authorization server metadata path configurable in AuthenticationOAuth2

([#24944](https://github.com/apache/pulsar/pull/24944)) \[feat]\[client] oauth2 trustcerts file and timeouts

([#25185](https://github.com/apache/pulsar/pull/25185)) \[improve]\[broker] Add strictAuthMethod to require explicit authentication method

([#25223](https://github.com/apache/pulsar/pull/25223)) \[fix]\[broker] Fix httpProxyTimeout config

([#25200](https://github.com/apache/pulsar/pull/25200)) \[improve]\[broker] Change log level from warn to debug when cursor mark-deleted position ledger doesn't exist

([#25195](https://github.com/apache/pulsar/pull/25195)) \[feat]\[io] implement pip-297 for jdbc sinks

([#25127](https://github.com/apache/pulsar/pull/25127)) \[improve]\[admin] Add client side looping to analyze-backlog in Topics to avoid potential HTTP call timeout

([#25188](https://github.com/apache/pulsar/pull/25188)) \[fix]\[broker] Prevent missed topic changes in topic watchers and schedule periodic refresh with patternAutoDiscoveryPeriod interval

([#25207](https://github.com/apache/pulsar/pull/25207)) \[fix]\[client] Fix producer synchronous retry handling in failPendingMessages method

([#25199](https://github.com/apache/pulsar/pull/25199)) \[fix]\[broker]Fix ledgerHandle failed to read by using new BK API

([#25165](https://github.com/apache/pulsar/pull/25165)) \[fix]\[broker] Fix ManagedCursorImpl.asyncDelete() method may lose previous async mark delete properties in race condition

([#25216](https://github.com/apache/pulsar/pull/25216)) \[fix]\[test]Fix flaky ExtensibleLoadManagerImplTest\_testGetMetrics

([#25211](https://github.com/apache/pulsar/pull/25211)) \[improve]\[proxy] Add regression tests for package upload with 'Expect: 100-continue'

([#24994](https://github.com/apache/pulsar/pull/24994)) \[improve]\[monitor] Upgrade OpenTelemetry to 1.56.0, Otel instrumentation to 2.21.0 and Otel semconv to 1.37.0

([#25187](https://github.com/apache/pulsar/pull/25187)) \[improve]\[meta] PIP-453: Improve the metadata store threading model

([#25208](https://github.com/apache/pulsar/pull/25208)) \[fix]\[client] Fix race condition between isDuplicate() and flushAsync() method in PersistentAcknowledgmentsGroupingTracker due to incorrect use Netty Recycler

([#25209](https://github.com/apache/pulsar/pull/25209)) \[fix] \[test] Upgrade docker-java to 3.7.0

([#25179](https://github.com/apache/pulsar/pull/25179)) \[fix]\[proxy] Close client connection immediately when credentials expire and forwardAuthorizationCredentials is disabled

([#25197](https://github.com/apache/pulsar/pull/25197)) \[fix]\[misc] Allow JWT tokens in OpenID auth without nbf claim

([#25186](https://github.com/apache/pulsar/pull/25186)) \[fix]\[test] Bump org.assertj:assertj-core from 3.27.5 to 3.27.7

([#25182](https://github.com/apache/pulsar/pull/25182)) \[improve]\[misc] Upgrade snappy version to 1.1.10.8

([#25178](https://github.com/apache/pulsar/pull/25178)) \[fix]\[client] ControlledClusterFailover avoid unnecessary reconnection.

([#25172](https://github.com/apache/pulsar/pull/25172)) \[improve]\[client]Reduce unnecessary getPartitionedTopicMetadata requests when using retry and DLQ topics.

([#25177](https://github.com/apache/pulsar/pull/25177)) \[fix]\[ml] Fix NoSuchElementException in EntryCountEstimator caused by a race condition

([#25166](https://github.com/apache/pulsar/pull/25166)) \[improve]\[broker] Upgrade bookkeeper to 4.17.3

([#25132](https://github.com/apache/pulsar/pull/25132)) \[improve]\[broker] Ensure metadata session state visibility and improve Unstable observability for ServiceUnitStateChannelImpl

([#25070](https://github.com/apache/pulsar/pull/25070)) \[improve]\[broker] PIP-442: Add memory limits for topic list watcher (part 2)

([#25157](https://github.com/apache/pulsar/pull/25157)) \[fix]\[fn] Fix graceful Pulsar Function shutdown so that consumers and producers are closed

([#25151](https://github.com/apache/pulsar/pull/25151)) \[fix]\[broker] Fence reset cursor by timestamp to avoid concurrent timestamp-based position lookups

([#25148](https://github.com/apache/pulsar/pull/25148)) \[fix]\[ml] Retry offload reads when OffloadReadHandleClosedException is encountered

([#25149](https://github.com/apache/pulsar/pull/25149)) \[fix]\[admin] Fix offload policy incompatible issue.

([#25142](https://github.com/apache/pulsar/pull/25142)) \[fix]\[proxy] Fix memory leaks in ParserProxyHandler

([#25140](https://github.com/apache/pulsar/pull/25140)) \[fix]\[fn] complete flushAsync before closeAsync in ProducerCache and wait for completion in closing the cache

([#25031](https://github.com/apache/pulsar/pull/25031)) \[fix]\[broker] Avoid split non-existent bundle

([#25136](https://github.com/apache/pulsar/pull/25136)) \[fix]\[broker] Fix regex matching of namespace name which might contain a regex char

([#25110](https://github.com/apache/pulsar/pull/25110)) \[fix]\[broker] Fix markDeletedPosition race condition in ManagedLedgerImpl.maybeUpdateCursorBeforeTrimmingConsumedLedger() method

([#25125](https://github.com/apache/pulsar/pull/25125)) \[fix]\[test] Wait for txn.abort() to complete to avoid AdminApiTransactionTest.testAnalyzeSubscriptionBacklogWithTransactionMarker() flaky test

([#25114](https://github.com/apache/pulsar/pull/25114)) \[fix]\[broker]Topic deleting failed after removed local cluster from namespace policies

([#25130](https://github.com/apache/pulsar/pull/25130)) \[improve]\[broker] Change the log level from error to info when throwing NotAllowedException

([#25048](https://github.com/apache/pulsar/pull/25048)) \[improve]\[broker] Enhance logging for adding schema failures in ServerCnx

([#25121](https://github.com/apache/pulsar/pull/25121)) \[fix]\[broker] Fix MultiRolesTokenAuthorizationProvider error when subscription prefix doesn't match.

([#25119](https://github.com/apache/pulsar/pull/25119)) \[fix]\[broker] Fix compaction horizon might be reset to an old position when phase two is interrupted

([#25104](https://github.com/apache/pulsar/pull/25104)) \[improve]\[broker] Fix thread safety issue in ManagedCursorImpl.removeProperty

([#25091](https://github.com/apache/pulsar/pull/25091)) \[improve]\[admin] Add counter for marker messages in PersistentTopics.analyzeSubscriptionBacklog() rest api

([#25089](https://github.com/apache/pulsar/pull/25089)) \[fix]\[ml] Fix cursor backlog size to account for individual acks

([#25077](https://github.com/apache/pulsar/pull/25077)) \[fix]\[broker] Fix chunked message loss when no consumers are available

([#25101](https://github.com/apache/pulsar/pull/25101)) \[fix]\[test] Fix ManagedCursorTest and NonDurableCursorTest flaky tests

([#25106](https://github.com/apache/pulsar/pull/25106)) \[fix]\[client]Producer stuck or geo-replication stuck due to wrong value of message.numMessagesInBatch

([#25105](https://github.com/apache/pulsar/pull/25105)) \[fix]\[broker]pulsar\_ml\_reads\_inflight\_bytes and pulsar\_ml\_reads\_available\_inflight\_bytes are 0 at the same time

([#25087](https://github.com/apache/pulsar/pull/25087)) \[fix]\[broker] Fix cursor position persistence in ledger trimming

([#25085](https://github.com/apache/pulsar/pull/25085)) \[improve]\[io] Replace Qpid in tests with RabbitMQ in Testcontainers and upgrade RabbitMQ client version

([#25084](https://github.com/apache/pulsar/pull/25084)) \[fix]\[build] Activate jdk21 and jdk24 profiles on Java 25

([#25073](https://github.com/apache/pulsar/pull/25073)) \[fix]\[broker]Infinitely failed to delete topic if the first time failed and enabled transaction

([#25047](https://github.com/apache/pulsar/pull/25047)) \[fix]\[broker]Fix incorrect backlog if use multiple acknowledge types on the same subscription

([#24980](https://github.com/apache/pulsar/pull/24980)) \[fix]\[broker] fix prepareInitPoliciesCacheAsync in SystemTopicBasedTopicPoliciesService

([#24658](https://github.com/apache/pulsar/pull/24658)) \[improve]\[broker] Optimize Reader creation in TopicPoliciesService

([#25053](https://github.com/apache/pulsar/pull/25053)) \[improve]\[broker] Use atomic counter for ongoing transaction count

([#25069](https://github.com/apache/pulsar/pull/25069)) \[fix]\[client] Fix invalid parameter type passed to Map.get in TopicsImpl.getListAsync method

([#25066](https://github.com/apache/pulsar/pull/25066)) \[fix]\[broker] PIP-442: Fix race condition in async semaphore permit updates that causes memory limits to become ineffective

([#25044](https://github.com/apache/pulsar/pull/25044)) \[improve]\[broker] Improve replicated subscription snapshot cache so that subscriptions can be replicated when mark delete position update is not frequent

([#25067](https://github.com/apache/pulsar/pull/25067)) \[fix]\[broker] Force EnsemblePolicies to resolve network location after rackInfoMap is updated due to changes in /ledgers/available znode

([#25050](https://github.com/apache/pulsar/pull/25050)) \[fix]\[admin] Refactor bookie affinity group sync operations to async in rest api

([#25059](https://github.com/apache/pulsar/pull/25059)) \[fix]\[broker] Fix various error-prone detected errors mainly in logging and String.format parameters

([#25054](https://github.com/apache/pulsar/pull/25054)) \[improve]\[build] Upgrade errorprone to 2.45.0 version

([#25056](https://github.com/apache/pulsar/pull/25056)) \[fix]\[cli] Fix output of --print-metadata in cli consume

([#25051](https://github.com/apache/pulsar/pull/25051)) \[fix]\[cli] Fix some pulsar-admin topicPolicies commands exiting before async operations complete

([#16651](https://github.com/apache/pulsar/pull/16651)) \[improve]\[broker] Fix replicated subscriptions race condition with mark delete update and snapshot completion

([#25027](https://github.com/apache/pulsar/pull/25027)) \[improve]\[misc] Add log4j-layout-template-json to server distribution to enable e.g. ECS template support in log4j configurations for Pulsar server components.

([#25032](https://github.com/apache/pulsar/pull/25032)) \[fix]\[test] Replace LZ4FastDecompressor with LZ4SafeDecompressor in test

([#25034](https://github.com/apache/pulsar/pull/25034)) \[improve]\[misc]introduce log4j Console appender ConsoleJson

([#25039](https://github.com/apache/pulsar/pull/25039)) \[fix]\[broker] Fix potential NPE in InMemTransactionBuffer.appendBufferToTxn by returning a valid Position

([#25026](https://github.com/apache/pulsar/pull/25026)) \[improve]\[broker]Add test for getting partitioned topic metadata with PulsarAdmin client

([#25029](https://github.com/apache/pulsar/pull/25029)) \[improve]\[io] Upgrade Debezium version to 3.2.5.Final

([#25036](https://github.com/apache/pulsar/pull/25036)) \[improve]\[client] Add null checks for MessageAcknowledger methods to prevent NullPointerException

([#25037](https://github.com/apache/pulsar/pull/25037)) \[fix]\[broker]Incorrect backlog that is larger than expected

([#24994)](https://github.com/apache/pulsar/pull/24994))) Revert "\[improve]\[monitor] Upgrade OpenTelemetry to 1.56.0, Otel instrumentation to 2.21.0 and Otel semconv to 1.37.0

([#25022](https://github.com/apache/pulsar/pull/25022)) \[fix] Upgrade gson to 2.13.2

([#25018](https://github.com/apache/pulsar/pull/25018)) \[improve]\[broker]Remove the warn log that frequently prints

([#25016](https://github.com/apache/pulsar/pull/25016)) \[fix]\[broker]Fix memory leak when using a customized ManagedLedger implementation

([#25015](https://github.com/apache/pulsar/pull/25015)) \[fix]\[client] Fix AutoProduceBytesSchema.clone() method

([#25014](https://github.com/apache/pulsar/pull/25014)) \[fix]\[client] Fix thread-safety of AutoProduceBytesSchema

([#25013](https://github.com/apache/pulsar/pull/25013)) \[improve]\[client] Test no exception could be thrown for invalid epoch in message

([#25011](https://github.com/apache/pulsar/pull/25011)) \[improve] Eliminate unnecessary duplicate schema lookups for partitioned topics in client and geo-replication

([#25004](https://github.com/apache/pulsar/pull/25004)) \[fix]\[broker] Add schema version in rest produce api

([#25012](https://github.com/apache/pulsar/pull/25012)) \[fix]\[broker] Fix issue with schemaValidationEnforced in geo-replication

([#25008](https://github.com/apache/pulsar/pull/25008)) \[fix]\[client] Fix double recycling of the message in isValidConsumerEpoch method

([#25007](https://github.com/apache/pulsar/pull/25007)) \[fix]\[client] PIP-84: Skip processing a message in the message listener if the consumer epoch is no longer valid

([#25006](https://github.com/apache/pulsar/pull/25006)) \[fix]\[client] Skip processing messages in the listener when the consumer has been closed

([#24994](https://github.com/apache/pulsar/pull/24994)) \[improve]\[monitor] Upgrade OpenTelemetry to 1.56.0, Otel instrumentation to 2.21.0 and Otel semconv to 1.37.0

([#24997](https://github.com/apache/pulsar/pull/24997)) \[fix]\[broker] Fix creation of replicated subscriptions for partitioned topics

([#24983](https://github.com/apache/pulsar/pull/24983)) \[improve] Upgrade Apache Commons library versions

([#24995](https://github.com/apache/pulsar/pull/24995)) \[improve]\[test] Use Oxia project docker container for integration tests

([#24986](https://github.com/apache/pulsar/pull/24986)) \[fix] Handle TLS close\_notify to avoid SslClosedEngineException: SSLEngine closed already

([#24982](https://github.com/apache/pulsar/pull/24982)) \[improve]\[build] Upgrade Testcontainers to 1.21.3

([#24975](https://github.com/apache/pulsar/pull/24975)) \[improve]\[broker]Improve error response of failed to delete topic if it has replicators connected

([#24938](https://github.com/apache/pulsar/pull/24938)) \[fix]\[broker]Wrong backlog: expected 0 but got 1

([#24985](https://github.com/apache/pulsar/pull/24985)) \[improve] Upgrade Log4j2 to 2.25.2 and slf4j to 2.0.17

([#24984](https://github.com/apache/pulsar/pull/24984)) \[improve] Upgrade Caffeine to 3.2.3

([#24871](https://github.com/apache/pulsar/pull/24871)) \[fix]\[test] Fixed Non-Guaranteed Order in PoliciesDataTest.propertyAdmin

([#24981](https://github.com/apache/pulsar/pull/24981)) \[fix]\[build] Remove Confluent and Restlet maven repositories from top level pom.xml

([#24976](https://github.com/apache/pulsar/pull/24976)) \[feat]\[meta] upgrade oxia version to 0.7.2

## Security Fixes

### Apache Pulsar

([#25256](https://github.com/apache/pulsar/pull/25256)) \[fix]\[sec] Upgrade aircompressor to 2.0.3 to resolve CVE-2025-67721

([#25250](https://github.com/apache/pulsar/pull/25250)) \[fix]\[sec] Upgrade Python protobuf version to 6.33.5 to address CVE-2026-0994

([#25095](https://github.com/apache/pulsar/pull/25095)) \[fix]\[sec] Upgrade jose4j to 0.9.6 to address CVE-2024-29371

([#25206](https://github.com/apache/pulsar/pull/25206)) \[fix]\[sec] Upgrade OpenSearch to 2.19.4 to remediate CVE-2025-9624

([#25198](https://github.com/apache/pulsar/pull/25198)) \[fix]\[sec] Exclude org.lz4:lz4-java and standardize on at.yawk.lz4-java to remediate CVE-2025-12183 and CVE-2025-66566

([#25175](https://github.com/apache/pulsar/pull/25175)) \[fix]\[sec] Bump org.apache.solr:solr-core from 9.8.0 to 9.10.1 in /pulsar-io/solr

([#25152](https://github.com/apache/pulsar/pull/25152)) \[fix]\[sec] Upgrade vertx to address CVE-2026-1002

([#25102](https://github.com/apache/pulsar/pull/25102)) \[fix]\[sec] Upgrade log4j to 2.25.3 to address CVE-2025-68161

([#25095](https://github.com/apache/pulsar/pull/25095)) \[fix]\[sec] Upgrade jose4j to 0.9.6 to address CVE-2024-29371

([#25078](https://github.com/apache/pulsar/pull/25078)) \[fix]\[sec] Upgrade Netty to 4.1.130.Final

([#25045](https://github.com/apache/pulsar/pull/25045)) \[fix]\[sec] Bump at.yawk.lz4:lz4-java from 1.9.0 to 1.10.1 in /pulsar-common

([#25024](https://github.com/apache/pulsar/pull/25024)) \[fix]\[sec] Eliminate commons-collections dependency

([#24987](https://github.com/apache/pulsar/pull/24987)) \[fix]\[sec] Bump github.com/dvsekhvalnov/jose2go from 1.6.0 to 1.7.0 in /pulsar-function-go


# V4.1.3.2
Source: https://docs.streamnative.io/release-notes/pulsar/v4.1/v4.1.3.2



# StreamNative Weekly Release Notes v4.1.3.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.1.3.2](https://github.com/streamnative/pulsar/releases/tag/v4.1.3.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.1.3.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.1.3.2/images/sha256-a79cd993a326e0112378b2029091e3aa49b00654bbd8a68b31ae883a8b82ad6c)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.1.3.2/images/sha256-994b762f5fa3504547723b977115f9e45f30350c99e30f28bb974c18f44ce68c)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.1.3.2/images/sha256-994b762f5fa3504547723b977115f9e45f30350c99e30f28bb974c18f44ce68c)

## General Changes

### KoP

Fix the jackson-dataformat-yaml dependency not found

Fix mvn deploy failure for oauth-client module

### StreamNative Pulsar Plugins

8342425fc fix

7c4daca65 Fix oidc test

7ce694187 Fix oidc test

Fix enum type cause Avro deserialize error

### Function Mesh Worker Service

Update dynamic auth

## Security Fixes

### Apache Pulsar

([#25264](https://github.com/apache/pulsar/pull/25264)) \[fix]\[sec] Upgrade Jackson version to 2.18.6


# V4.1.3.3
Source: https://docs.streamnative.io/release-notes/pulsar/v4.1/v4.1.3.3



# StreamNative Weekly Release Notes v4.1.3.3

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.1.3.3](https://github.com/streamnative/pulsar/releases/tag/v4.1.3.3)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.1.3.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.1.3.3/images/sha256-5b5dd26b8fd5c21f8d2243481852d94a5c713bfe5e038f333ac1aeac70a4f523)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.1.3.3/images/sha256-4c47bc76f3079301c0261eac6f876a1e3cb8371d5727a0b513ece126eb961e8f)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.1.3.3/images/sha256-4c47bc76f3079301c0261eac6f876a1e3cb8371d5727a0b513ece126eb961e8f)

## General Changes

### Apache Pulsar

([#25296](https://github.com/apache/pulsar/pull/25296)) \[fix]\[offload] Close all resources in BlobStoreBackedReadHandleImplV2.closeAsync

([#25276](https://github.com/apache/pulsar/pull/25276)) \[fix]\[broker] Support namespace unsubscribe when bundles are unloaded

### KoP

Improve performance for finding position by offset

### StreamNative Pulsar Plugins

Use `CLOUDSTORAGE_S3_BUCKET` for AWS cloud storage tests

Remove deprecated checkCluster method using TopicName.getCluster()

\[imporve]\[audit-log]expands producer maxPendingMessages to avoid the error 'Producer send queue is full'

\[feature] A new detector for loading topics

\[fix]\[audit-log] Log ProducerQueueIsFullError at INFO level to prevent log flooding

fix: override log4j to 2.25.3 for metadata tool

### pulsarctl

test: tolerate namespace not found errors

bump go version to 1.25.8 to fix CVE-2026-25679 and CVE-2026-27142

feat(topic): add subscription dispatch rate commands

### StreamNative Unified RBAC

a4666a0 Bump version to 1.9.3

fix ci

b25340b Bump version to 1.9.2

feat: Add workspace related permissions

7217191 Bump version to 1.9.1

f67a04f Bump version to 1.9.0

7e951d3 Bump version to 1.8.4

2e934a1 Bump version to 1.8.3

55dc467 Bump version to 1.8.2

df7efec Bump version to 1.8.1

79549de Bump version to 1.8.0

feat: upgrade sdk-go version to v0.15.0

feat: support instances permissions mapping

ceb3418 Bump version to 1.7.4

fixes the schedule release workflow

5ef2946 fixes: configure maven

34c62ff fixes: use the pulsar image directly

fca5ff8 fixes: fix the wrong image

feat: support features and featuregates permissions mapping

## Security Fixes

### Apache Pulsar

([#25303](https://github.com/apache/pulsar/pull/25303)) \[fix]\[sec] Bump org.apache.zookeeper:zookeeper from 3.9.4 to 3.9.5


# V4.1.3.4
Source: https://docs.streamnative.io/release-notes/pulsar/v4.1/v4.1.3.4



# StreamNative Weekly Release Notes v4.1.3.4

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.1.3.4](https://github.com/streamnative/pulsar/releases/tag/v4.1.3.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.1.3.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.1.3.4/images/sha256-b9c2cc29aaff9fb2c3432b66788bcb47697594be5671fd5f8e6a059b8a95eddf)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.1.3.4/images/sha256-185605589c58a017c973e93dd180c9517a67ec89dc74ac3433f51561ebbd1b64)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.1.3.4/images/sha256-185605589c58a017c973e93dd180c9517a67ec89dc74ac3433f51561ebbd1b64)

## General Changes

### Apache Pulsar

([#25293](https://github.com/apache/pulsar/pull/25293)) \[improve]\[broker]Reduce the lock range of SimpleCache to enhance performance

([#25371](https://github.com/apache/pulsar/pull/25371)) \[fix]\[broker] Fix IllegalArgumentException in BucketDelayedDeliveryTracker.addMessage

([#25352](https://github.com/apache/pulsar/pull/25352)) \[fix]\[broker] Fix race condition in ServerCnx producer/consumer async callbacks

([#25312](https://github.com/apache/pulsar/pull/25312)) \[fix]\[broker]system topic was created with different partitions acrossing clusters after enabled namespace-level replication

([#25325](https://github.com/apache/pulsar/pull/25325)) \[fix]\[io]\[kca] kafka headers silently dropped

([#25317](https://github.com/apache/pulsar/pull/25317)) \[fix]\[client] Fail messages immediately in ProducerImpl when in terminal state

([#25316](https://github.com/apache/pulsar/pull/25316)) \[fix] Fix flaky OneWayReplicatorTest.testTopicPoliciesReplicationRule

([#25314](https://github.com/apache/pulsar/pull/25314)) \[fix]\[test] Fix flaky PulsarDebeziumOracleSourceTest

([#25266](https://github.com/apache/pulsar/pull/25266)) \[fix]\[broker] Handle missing replicator during snapshot request processing

([#25346](https://github.com/apache/pulsar/pull/25346)) \[fix]\[broker] Fix concurrency bug in BucketDelayedDeliveryTracker

### MoP

d4a069f3 Ignore some flaky test

### KoP

\[branch-4.1] Fix compatibility issue for ManagedLedgerInfo

fix: respect max bytes limit for both requests and partitions

Scheduled to add first offset to the ledger info

Fix "LastConfirmedEntry is xxx when reading" read failures after ledger rollover

### StreamNative Pulsar Plugins

222eabf7d Fix hashicorp image issue

fix: exclude stream-storage-server from pulsar-metadata-tool

fix: upgrade spring-beans to 6.2.12 and spring-ldap-core to 3.2.16 to fix CVEs in broker-auth-ldap

aa20817d2 Update bom version to 4.1.0-SNAPSHOT, the 4.0.0-SNAPSHOT will keep update when bom branch-4.1 have any change

Remove zookeeper version defiine

5d00fcc46 Upgrade bookie-rackinfo netty version

fix: upgrade Go from 1.25.6 to 1.25.8 to patch CVE-2026-27139 in stdlib

Fix vault docker version issue

### pulsarctl

Update Trivy GitHub Action to v0.35.0

Gate Docker login and snstage image usage on streamnativebot actor

Add platform and compute teams as CODEOWNERS

### Function Mesh Worker Service

e9fa1acc fix: use 4.1.3.3 base image in CI

4a919936 feat: Support volume & volumeMounts in CustomRuntimeOptions

Support unified RBAC for registry service

feat: expose clusterRef field to ConnectionConfig

fix: do not set static bootstrapServers when in registry mode

fix: fix build error

feat: add a new endpoint to validate connection

### StreamNative Unified RBAC

fix: include Pulsar 4.1 in pulsar40 compat source set

## Security Fixes

### Apache Pulsar

([#25399](https://github.com/apache/pulsar/pull/25399)) \[fix]\[sec] Upgrade to Netty 4.1.132.Final to address CVEs


# V4.1.3.5
Source: https://docs.streamnative.io/release-notes/pulsar/v4.1/v4.1.3.5



# StreamNative Weekly Release Notes v4.1.3.5

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.1.3.5](https://github.com/streamnative/pulsar/releases/tag/v4.1.3.5)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.1.3.5/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.1.3.5/images/sha256-d08bd555d73bf231a8ac87e4beaa00dad34670ec82d08e29d5c4e1d9534f7c38)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.1.3.5/images/sha256-d32ee1db72eb02870199ea3b961270f1765c4926df09fe9e553feccac738e178)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.1.3.5/images/sha256-d32ee1db72eb02870199ea3b961270f1765c4926df09fe9e553feccac738e178)

## General Changes

### Apache Pulsar

([#25400](https://github.com/apache/pulsar/pull/25400)) \[fix]\[client] Fix thread-safety and refactor MessageCryptoBc key management

([#25379](https://github.com/apache/pulsar/pull/25379)) \[fix]\[broker] Fix ExtensibleLoadManagerImpl stuck Assigning bundle state after broker restart

([#25437](https://github.com/apache/pulsar/pull/25437)) \[fix]\[broker]Producer with AUTO\_PRODUCE schema failed to reconnect, which caused by schema incompatible

### KoP

Improve offset index update logic

Only try creating missed partition when the partition does not exist

Improve logic for offset finding by timestamp

614da424a \[branch-4.1] Bump version to 4.1.3.4

([#1859)](https://github.com/streamnative/ksn/pull/1859))) Revert "Scheduled to add first offset to the ledger info

([#1872)](https://github.com/streamnative/ksn/pull/1872))) Revert "\[branch-4.1] Fix compatibility issue for ManagedLedgerInfo

### StreamNative Pulsar Plugins

aebb871f4 Fix hashicorp image issue

### pulsarctl

Update version from v4.0.6.1 to v4.1.3.4

### Function Mesh Worker Service

fix: fix AgentFunction missing auth for package service

de5ee705 fix ci

fix(runtime): preserve connection fields during updates

feat(registry): add support for short-form package URLs

fix: make pulsar package service always use internal auth

Implement package service

### StreamNative Unified RBAC

fix: remove Cloud Integration workflow

## Security Fixes


# V4.1.3.6
Source: https://docs.streamnative.io/release-notes/pulsar/v4.1/v4.1.3.6



# StreamNative Weekly Release Notes v4.1.3.6

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.1.3.6](https://github.com/streamnative/pulsar/releases/tag/v4.1.3.6)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.1.3.6/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.1.3.6/images/sha256-05b178c941952ad7b46c0646c6ed1cd190b672bf7f53623ef0c6b698504522f3)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.1.3.6/images/sha256-cad0a5b20291222e01f5e62b51fa55e5397a6d44340db02f8aa5fc6307634d59)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.1.3.6/images/sha256-cad0a5b20291222e01f5e62b51fa55e5397a6d44340db02f8aa5fc6307634d59)

## General Changes

### pulsarctl

Bump go version to 1.25.9 to fix CVE-2026-32280

### Cloud Pulsar Plugins

\[ApiKeys] Include role in auth failure log during new/refresh auth state

### Function Mesh Worker Service

feat: add more cases for registry CI

## Security Fixes


# V4.2.0.1
Source: https://docs.streamnative.io/release-notes/pulsar/v4.2/v4.2.0.1



# StreamNative Weekly Release Notes v4.2.0.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.2.0.1](https://github.com/streamnative/pulsar/releases/tag/v4.2.0.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.2.0.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.2.0.1/images/sha256-224897c1dbcd3f356a006f58dc4bcc03dfc32265a7f7fec840a647a73782052b)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.2.0.1/images/sha256-b0f95e76ef8c225990f3a77599e10e58441409405d69d4f52eeb60f1e778d5ca)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.2.0.1/images/sha256-b0f95e76ef8c225990f3a77599e10e58441409405d69d4f52eeb60f1e778d5ca)

## General Changes

## Security Fixes


# V4.2.0.2
Source: https://docs.streamnative.io/release-notes/pulsar/v4.2/v4.2.0.2



# StreamNative Weekly Release Notes v4.2.0.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.2.0.2](https://github.com/streamnative/pulsar/releases/tag/v4.2.0.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.2.0.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.2.0.2/images/sha256-ee7a04b1cd6568e7f139efccd8fa44809049805d6e1c19279ffd773a0d45be5c)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.2.0.2/images/sha256-6966a304aa5e98855f493b8054ba18e50879c7da1d90606c4e47eb4b1f7c60fd)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.2.0.2/images/sha256-6966a304aa5e98855f493b8054ba18e50879c7da1d90606c4e47eb4b1f7c60fd)

## General Changes

### StreamNative Unified RBAC

feat: upgrade sdk-go version to v0.16.0

220bc57 Bump version to 1.13.1-rc1

feat: RC-based release workflow

1bbe365 fix: silence TypeScript moduleResolution deprecation warning

f67f8a6 Bump version to 1.13.0

refactor: migrate to Gradle version catalog, restructure project, and fix NAR packaging

2816c22 Bump version to 1.12.0

fix: fix cv image

## Security Fixes


# V4.2.0.3
Source: https://docs.streamnative.io/release-notes/pulsar/v4.2/v4.2.0.3



# StreamNative Weekly Release Notes v4.2.0.3

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.2.0.3](https://github.com/streamnative/pulsar/releases/tag/v4.2.0.3)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.2.0.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.2.0.3/images/sha256-2e6ebade6aed60e4593d2ee2994d8a39f35f6e9c9dcf82d5e82c1b65a001a546)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.2.0.3/images/sha256-534919ecf46f0db656702f05cf47be50b4f4c2eb02bc568fb6fa281b434e9633)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.2.0.3/images/sha256-534919ecf46f0db656702f05cf47be50b4f4c2eb02bc568fb6fa281b434e9633)

## General Changes

### KoP

Prevent concurrent metadata requests in each connection

Improve the performance of metadata request processing

Support listing non-partitioned topics

Speed up maven build by adjusting the repository order

Improve offset index update logic

Only try creating missed partition when the partition does not exist

### pulsarctl

Bump go version to 1.25.9 to fix CVE-2026-32280

### Cloud Pulsar Plugins

\[ApiKeys] Include role in auth failure log during new/refresh auth state

### Function Mesh Worker Service

e1285860 fix ci

fix(auth): preserve service account when updateAuthData is false

## Security Fixes


# V4.2.0.5
Source: https://docs.streamnative.io/release-notes/pulsar/v4.2/v4.2.0.5



# StreamNative Weekly Release Notes v4.2.0.5

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.2.0.5](https://github.com/streamnative/pulsar/releases/tag/v4.2.0.5)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.2.0.5/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.2.0.5/images/sha256-b9f9fb09cf38b7ad690e5e7e665b590fb6b54dd67ad22ddf3c996fe7c61685df)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.2.0.5/images/sha256-b1719e2a6f7c9a8aca0817c705f0f87b2f06aabaf7af901e47c5ab5a0686f12e)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.2.0.5/images/sha256-b1719e2a6f7c9a8aca0817c705f0f87b2f06aabaf7af901e47c5ab5a0686f12e)

## General Changes

### Apache Pulsar

([#25563](https://github.com/apache/pulsar/pull/25563)) \[fix]\[test] Extend SameAuthParamsLookupAutoClusterFailoverTest phase timeouts

([#25562](https://github.com/apache/pulsar/pull/25562)) \[fix]\[test] Relax BrokerRegistryIntegrationTest broker-close threshold

([#25557](https://github.com/apache/pulsar/pull/25557)) \[fix]\[broker] pulsar admin stats internal with metadata command

([#25463](https://github.com/apache/pulsar/pull/25463)) \[fix]\[test] Fix flaky BrokerRegistryIntegrationTest port binding race

([#25560](https://github.com/apache/pulsar/pull/25560)) \[fix]\[test] Recreate EventLoop in PublishRateLimiterTest setup

([#25502](https://github.com/apache/pulsar/pull/25502)) \[fix]\[broker] Unthrottle producers immediately when publish rate limiting is disabled

([#25558](https://github.com/apache/pulsar/pull/25558)) \[fix]\[broker] Lower log level of DrainingHashesTracker not-found entry to DEBUG

([#25500](https://github.com/apache/pulsar/pull/25500)) \[fix]\[test] Fix flaky ExtensibleLoadManagerTest.startBroker timeout

([#25509](https://github.com/apache/pulsar/pull/25509)) \[fix]\[test] Fix flaky ExtensibleLoadManagerImpl client reconnection tests: PulsarClientException\$AlreadyClosedException: Client already closed

([#25427](https://github.com/apache/pulsar/pull/25427)) \[fix]\[test] Fix flaky testLoadBalancerServiceUnitTableViewSyncer

([#25497](https://github.com/apache/pulsar/pull/25497)) \[fix]\[test] Fix flaky ServerCnxTest.testCreateProducerTimeoutThenCreateSameNamedProducerShouldFail

([#25460](https://github.com/apache/pulsar/pull/25460)) \[fix]\[broker] Prevent timed-out producer creation from racing with retry

([#25551](https://github.com/apache/pulsar/pull/25551)) \[fix]\[broker]Namespaces can be created with may empty replication\_clusters policy

([#25272](https://github.com/apache/pulsar/pull/25272)) \[fix]\[broker] Fix backlog clearing for unloaded namespace bundles

([#24463](https://github.com/apache/pulsar/pull/24463)) \[improve]\[broker] Improve the performance of TopicName constructor

([#25367](https://github.com/apache/pulsar/pull/25367)) \[improve]\[common] Optimize TopicName.get() to reduce lock contention on cache lookup

### MoP

daddd520 fix checkstyle

c3843589 Fix mqtt client not have the consumer metric

### KoP

5d608443c Merge branch 'branch-4.2' into branch-4.2.0.5

657c15f01 \[branch-4.2] Bump version to 4.2.0.5 and fix testUnValidTopicToUnload

8b2b58b62 Merge branch 'branch-4.2' into branch-4.2.0.5

9cae31b70 fix dependencies might not be synced to Google mirror

Include client id as the suffix of producer name in topic stats

### StreamNative Pulsar Plugins

271cdc326 test apache maven central repo

Add keyword to filter the result

fix: backport permission/ACL audit logging to branch-4.2

## Security Fixes


# V4.2.0.6
Source: https://docs.streamnative.io/release-notes/pulsar/v4.2/v4.2.0.6



# StreamNative Weekly Release Notes v4.2.0.6

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.2.0.6](https://github.com/streamnative/pulsar/releases/tag/v4.2.0.6)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.2.0.6/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.2.0.6/images/sha256-f0d190ec3b8f58e76d537607e3dc53769a9bf8d8cbd7a10615bbb58740ab69fe)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.2.0.6/images/sha256-84a970a2960c09b077580a3b75caed95b1450885c4c3e253bb0f73c2aab6697c)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.2.0.6/images/sha256-84a970a2960c09b077580a3b75caed95b1450885c4c3e253bb0f73c2aab6697c)

## General Changes

### Apache Pulsar

([#25578](https://github.com/apache/pulsar/pull/25578)) \[fix]\[client] Stabilize scaleReceiverQueueHint against concurrent enqueue/take

([#25518](https://github.com/apache/pulsar/pull/25518)) \[improve]\[broker] Use full bundle name for namespace bundle destination affinity in ModularLoadManagerImpl

([#25566](https://github.com/apache/pulsar/pull/25566)) \[fix]\[test] Flaky SameAuthParamsLookupAutoClusterFailoverTest

([#25561](https://github.com/apache/pulsar/pull/25561)) \[fix]\[test] Fix flaky OffloadPrefixTest.testPositionOnEdgeOfLedger race with ledger rollover

### StreamNative Pulsar Plugins

Add apache maven central repo as the fallback repo

### pulsarctl

Fix flaky tests

### Function Mesh Worker Service

feat: Support setting extraDependenciesDir via CustomRuntimeOptions

## Security Fixes

### Apache Pulsar

([#25569](https://github.com/apache/pulsar/pull/25569)) \[fix]\[sec] Upgrade BouncyCastle to 1.84 (CVE-2026-5588, CVE-2026-0636)


# V4.2.0.7
Source: https://docs.streamnative.io/release-notes/pulsar/v4.2/v4.2.0.7



# StreamNative Weekly Release Notes v4.2.0.7

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.2.0.7](https://github.com/streamnative/pulsar/releases/tag/v4.2.0.7)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.2.0.7/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.2.0.7/images/sha256-19c9a33900fdbaf2e36635730edb04e9abca2dd7a56c84d56f053168fc3466f5)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.2.0.7/images/sha256-c4c1996c9e0424758532843cf4ba99df1122529a260a32b29bd28b6e498e0140)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.2.0.7/images/sha256-c4c1996c9e0424758532843cf4ba99df1122529a260a32b29bd28b6e498e0140)

## General Changes

### Apache Pulsar

([#25679](https://github.com/apache/pulsar/pull/25679)) \[fix]\[test] Fix flaky OneWayReplicatorDeduplicationTest.testDeduplication

([#25677](https://github.com/apache/pulsar/pull/25677)) \[improve]\[test] Set diskUsageThreshold to 0.999 for tests to effectively disable the check

([#25641](https://github.com/apache/pulsar/pull/25641)) \[fix]\[test] Make NamespacesTest.cleanupAfterMethod tolerant of transient infra failures

([#25640](https://github.com/apache/pulsar/pull/25640)) \[fix]\[test] Fix flaky testGetExcludedBookiesWithIsolationGroups

([#25638](https://github.com/apache/pulsar/pull/25638)) \[fix]\[test] Reduce flakiness in testLoadBalancerServiceUnitTableViewSyncer

([#25596](https://github.com/apache/pulsar/pull/25596)) \[fix]\[test] Fix flaky ExtensibleLoadManagerImplTest.testLoadBalancerServiceUnitTableViewSyncer

([#25759](https://github.com/apache/pulsar/pull/25759)) \[fix]\[client] Apply Avro logical type conversions when decoding schema without classloader

([#25752](https://github.com/apache/pulsar/pull/25752)) \[improve]\[misc] Upgrade Jetty to 12.1.9

([#25538](https://github.com/apache/pulsar/pull/25538)) \[improve]\[client] Implement tls\_client\_auth for AuthenticationOAuth2

([#25736](https://github.com/apache/pulsar/pull/25736)) \[fix]\[broker] Merge broker offload extra configurations

([#25626](https://github.com/apache/pulsar/pull/25626)) \[improve]\[broker] optimize namespaceBundle validation to fix single-thread 100% CPU during unloading entire namespaces

([#25624](https://github.com/apache/pulsar/pull/25624)) \[fix]\[broker] Close pending acks cleanup gap in BacklogQuotaManager

([#25592](https://github.com/apache/pulsar/pull/25592)) \[fix]\[broker] Move pending acks cleanup to selected mark-delete callbacks

([#25589](https://github.com/apache/pulsar/pull/25589)) \[fix]\[broker] Fix race in pending acks removal in redeliverUnacknowledgedMessages

([#25579](https://github.com/apache/pulsar/pull/25579)) \[fix]\[broker] Wait for orphan schema ledger cleanup before retry

([#25514](https://github.com/apache/pulsar/pull/25514)) \[fix]\[broker] Clean up orphan ledger on concurrent initial schema creation in BookkeeperSchemaStorage

([#25681](https://github.com/apache/pulsar/pull/25681)) \[fix]\[broker] Correct two race conditions in the tracker code and logic bug in InMemoryDelayedDeliveryTracker that failed with NoSuchElementException

([#25684](https://github.com/apache/pulsar/pull/25684)) \[fix]\[broker] Skip backlog-quota eviction on fenced/closing topics

([#25730](https://github.com/apache/pulsar/pull/25730)) ([#25739](https://github.com/apache/pulsar/pull/25739)) \[fix]\[client] Make ClientBuilder serializable

([#25581](https://github.com/apache/pulsar/pull/25581)) \[fix]\[broker] Decrement unacked counter when removeAllUpTo removes pending acks

([#25126](https://github.com/apache/pulsar/pull/25126)) \[improve]\[cli] Add client side looping in "pulsar-admin topics analyze-backlog" cli to avoid potential HTTP call timeout

([#25725](https://github.com/apache/pulsar/pull/25725)) \[fix]\[client]Broker-side producer handle leak if closes a producer which state is regitering schema

([#25728](https://github.com/apache/pulsar/pull/25728)) \[fix]\[broker]\[branch-4.2] URL-encode sub-name in Txn pending-ack topic #25727

([#25583](https://github.com/apache/pulsar/pull/25583)) \[fix]\[broker]\[fix]\[broker]Replication stats is empty when the cluster is the target cluster of a one-way replication

([#25625](https://github.com/apache/pulsar/pull/25625)) \[fix]\[broker]Replication is stuck because failed to read entries

([#25663](https://github.com/apache/pulsar/pull/25663)) \[improve]\[misc] Upgrade Caffeine to 3.2.4

([#25644](https://github.com/apache/pulsar/pull/25644)) \[fix]\[broker] ConcurrentLongHashMap throw ArrayIndexOutOfBoundsException

([#25572](https://github.com/apache/pulsar/pull/25572)) \[fix]\[broker] Race condition causes perpetual backlog on internal topics

### KoP

Implement lazy recovery for producer state management for classic engine

4c5015c44 \[branch-4.2] Remove proxy from build.sh

18f8469c8 Remove pulsar-kafka-proxy dependency from pom.xml

Fix fetch request duplicate key issue

Fix rdkafka requests might be forward to non-leader brokers and never recovered

Skip permission check for internal stats consumer

Stabilize generated swagger definitions

Remove proxy module and related tests

Fix possible message loss from idempotent producers during ledger rollover

### StreamNative Pulsar Plugins

fix: upgrade Netty to 4.1.133.Final in bookie-rackinfo to fix CVEs for sn-platform-slim:4.2.0.6

Fix to allow message id seek for non-partitioned topics

### pulsarctl

Bump go version to 1.25.10 to fix CVE

Add JDK path

### Function Mesh Worker Service

5fbc54fb fix ci

feat: Support pin agent version when create session

feat: support filter based on metadata and use CRD for sessions

feat: Align with Anthropic api

fix(registry-service): mirror agent rename onto binding display-name

feat: add default metadata and add metadata to labels

feat: Support sandbox agent

### StreamNative Unified RBAC

refactor: remove Pulsar servlet compatibility logic on main

63f1b65 Bump version to 1.13.2-rc10

41b673e Bump version to 1.13.2-rc9

b2f01df Bump version to 1.13.2-rc8

feat: add more agent related permissions

6dfb0ea Bump version to 1.13.2-rc7

c2fd5c0 Bump version to 1.13.2-rc6

\[codex] Fix maintenance notification action RBAC mappings

0ad4ea1 Bump version to 1.13.2-rc5

feat: add MaintenanceNotification permissions to unified RBAC

9c144cf Bump version to 1.13.2-rc4

ba6b888 Bump version to 1.13.2-rc3

feat: add test cases for workspaces and update sdk-apiserver

bde7d6e Bump version to 1.13.2-rc2

fix: update workspaces' api group to compute.streamnative.io

66bbb7a Bump version to 1.13.2-rc1

fix: support cloud integration with specified image

fix: remove Cloud Integration workflow

fix: switch default Pulsar to snstage/pulsar:4.0.9.6 and simplify integration tests

fix: exclude integration tests from PR CI

fix: remove duplicate CI runs on branch-\* pushes

fix: make pulsar-broker Maven group configurable

0c6a183 Bump version to 1.13.1

9aa9527 Bump version to 1.13.1-rc5

fix: remove duplicate npm version in Publish JS step

60db2c5 Bump version to 1.13.1-rc4

fix: bump sdk-js package.json version in release step

82eb61a Bump version to 1.13.1-rc3

feat: validate before release, fix pipeline ordering

00b7cc6 Bump version to 1.13.1-rc2

refactor: independent versioning, sdk-java-admin split, shade cel

## Security Fixes

### Apache Pulsar

([#25744](https://github.com/apache/pulsar/pull/25744)) \[fix]\[sec] Upgrade thrift to 0.23.0 to address CVE-2026-43869

([#25745](https://github.com/apache/pulsar/pull/25745)) \[fix]\[sec] Upgrade vertx to 4.5.27 to address CVE-2026-6860

([#25737](https://github.com/apache/pulsar/pull/25737)) \[fix]\[sec] Upgrade vert.x to 4.5.25 to address CVE-2026-6860

([#25670](https://github.com/apache/pulsar/pull/25670)) \[fix]\[sec] Upgrade Netty to 4.1.133.Final to address CVEs


# V4.2.0.8
Source: https://docs.streamnative.io/release-notes/pulsar/v4.2/v4.2.0.8



# StreamNative Weekly Release Notes v4.2.0.8

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.2.0.8](https://github.com/streamnative/pulsar/releases/tag/v4.2.0.8)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.2.0.8/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.2.0.8/images/sha256-bf3f26cd80a137787883fe363f88f1382d4eed2aca8b8929058d6f72e42be28d)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.2.0.8/images/sha256-51fa72581eef1e53d77b66d15632155edf685d8641c530d9cd8e30b173397a9d)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.2.0.8/images/sha256-51fa72581eef1e53d77b66d15632155edf685d8641c530d9cd8e30b173397a9d)

## General Changes

### Apache Pulsar

([#25796](https://github.com/apache/pulsar/pull/25796)) \[fix]\[broker] Fix ManagedLedgerImpl.advanceCursorsIfNecessary() method may lose non-durable cursor properties in race condition

([#25781](https://github.com/apache/pulsar/pull/25781)) \[fix]\[broker] Use effective offload policies for extra configs

([#25767](https://github.com/apache/pulsar/pull/25767)) \[improve]\[broker] Prevent stale replicator pending reads after termination

([#25790](https://github.com/apache/pulsar/pull/25790)) \[refactor]\[fn] Use Map instead of TreeMap for connector/function API types

([#25785](https://github.com/apache/pulsar/pull/25785)) \[improve]\[build] Upgrade org.apache.kerby:kerb-simplekdc from 1.1.1 to 2.1.1

([#25773](https://github.com/apache/pulsar/pull/25773)) \[improve]\[fn] make built-in connector reload incremental

([#25777](https://github.com/apache/pulsar/pull/25777)) \[fix]\[broker] Fix PulsarService.closeAsync where Condition.signalAll was called without holding a lock

([#25770](https://github.com/apache/pulsar/pull/25770)) \[fix]\[proxy] Close channel on connection failure

### StreamNative Pulsar Plugins

Add rest v2 consume timeout to help return the current data

### pulsar-tiered-storage

fix(read): terminate V1 block-index scan when requested range already satisfied

### Function Mesh Worker Service

feat: update agents api to match Claude managed agent api

feat: align api updates from Claude

## Security Fixes

### Apache Pulsar

([#25818](https://github.com/apache/pulsar/pull/25818)) \[fix]\[sec] Bump org.asynchttpclient:async-http-client from 2.14.5 to 2.15.0


# V4.2.0.9
Source: https://docs.streamnative.io/release-notes/pulsar/v4.2/v4.2.0.9



# StreamNative Weekly Release Notes v4.2.0.9

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.2.0.9](https://github.com/streamnative/pulsar/releases/tag/v4.2.0.9)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.2.0.9/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.2.0.9/images/sha256-7b3e28e15e1980f3c9a2d763dec4e02adee565ad72ddf6f5fb4b405ba3dfd591)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.2.0.9/images/sha256-afbbf07e4d3655cef10b5ed40017ce8dc35120f8fc41b564d50a811b2a2205e1)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.2.0.9/images/sha256-afbbf07e4d3655cef10b5ed40017ce8dc35120f8fc41b564d50a811b2a2205e1)

## General Changes

### Apache Pulsar

([#25805](https://github.com/apache/pulsar/pull/25805)) \[fix]\[client] Fix failed to close consumer because of the error: param memorySize is a negative value

([#25854](https://github.com/apache/pulsar/pull/25854)) \[improve]\[client] In cases where there is a risk of message loss, adjust the log level to error

([#25855](https://github.com/apache/pulsar/pull/25855)) \[improve]\[build] Remove kotlin-stdlib override; upgrade okhttp3 5.3.2 and okio 3.17.0

([#25852](https://github.com/apache/pulsar/pull/25852)) \[fix]\[test] Fix flaky ResendRequestTest.testSharedSingleAckedPartitionedTopic() test

([#25828](https://github.com/apache/pulsar/pull/25828)) \[fix]\[test] Add timeout to initial receives in ResendRequestTest.testSharedSingleAckedPartitionedTopic

([#25840](https://github.com/apache/pulsar/pull/25840)) \[fix]\[fn] Fix functions update issue where artifact is provided as a http url

([#25819](https://github.com/apache/pulsar/pull/25819)) \[improve]\[fn] Avoid gRPC timeout when getting status of a dead process runtime

### KoP

Initialize PID with carried producer ID

Fix destination broker for NotOwnedBundleHandler

feat: integrate group lag to Pulsar subscription's backlog in stats

Fix leader epoch capability advertisement and unknown epoch handling

### Cloud Pulsar Plugins

\[improve] support token with whitespace

### Function Mesh Worker Service

ec7a2733 fix ci

feat: remove "organization", "instance" label from metrics

b2474f3d fix ci

feat: expose agent session metrics

Implement agent trigger

## Security Fixes


# V4.2.1.1
Source: https://docs.streamnative.io/release-notes/pulsar/v4.2/v4.2.1.1



# StreamNative Weekly Release Notes v4.2.1.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.2.1.1](https://github.com/streamnative/pulsar/releases/tag/v4.2.1.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.2.1.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.2.1.1/images/sha256-61d727962dc64e2b356f24836f53f1cb161a4beadd4ee7672991c17d5213ad5d)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.2.1.1/images/sha256-3d653f070fc6cadf9cb985a3a53dd4e702763ab00bb687c15fae940a6bbce815)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.2.1.1/images/sha256-3d653f070fc6cadf9cb985a3a53dd4e702763ab00bb687c15fae940a6bbce815)

## General Changes

### Apache Pulsar

([#25793](https://github.com/apache/pulsar/pull/25793)) \[improve]\[offload] Coalesce automatic offload triggers to reduce retry loops and ledger scans

([#25899](https://github.com/apache/pulsar/pull/25899)) \[fix]\[client] Prevent duplicate ServiceUrlProvider initialization

([#25919](https://github.com/apache/pulsar/pull/25919)) \[fix]\[proxy] Avoid intermittent 502 when admin proxy follows a broker redirect for a request with a body

([#25916](https://github.com/apache/pulsar/pull/25916)) \[fix]\[client] Clean up unacked messages when unsubscribing a topic with ack timeout backoff

([#25868](https://github.com/apache/pulsar/pull/25868)) \[improve]\[fn] make built-in functions reload incremental

([#25892](https://github.com/apache/pulsar/pull/25892)) \[fix]\[test] Fix flaky SameAuthParamsLookupAutoClusterFailoverTest.testAutoClusterFailover() test

([#25910](https://github.com/apache/pulsar/pull/25910)) \[fix]\[meta] Fix ZooKeeper session reconnect race condition in PulsarZooKeeperClient.clientCreator

([#25913](https://github.com/apache/pulsar/pull/25913)) \[fix]\[meta] Fix PulsarZooKeeperClient async addWatch callback retry behavior

([#25426](https://github.com/apache/pulsar/pull/25426)) \[fix]\[test] Fix flaky testMsgDropStat in NonPersistentTopicTest

([#25826](https://github.com/apache/pulsar/pull/25826)) \[fix]\[client] Reset higher-index states on recovery in SameAuthParamsLookupAutoClusterFailover

([#25620](https://github.com/apache/pulsar/pull/25620)) \[fix]\[broker] Fix stuck chunks in SharedConsumerAssignor permit tracking

([#25594](https://github.com/apache/pulsar/pull/25594)) \[fix]\[broker] Fix precision loss in DataSketchesSummaryLogger by replacing LongAdder with DoubleAdder for sum accumulation

([#25525](https://github.com/apache/pulsar/pull/25525)) \[improve]\[client] Best-effort retry for individual/batch-index acks on send failure when ackReceiptEnabled=false

([#25817](https://github.com/apache/pulsar/pull/25817)) \[fix]\[broker] Fix non-batched null-value messages not removed during topic compaction

([#25825](https://github.com/apache/pulsar/pull/25825)) \[fix]\[bk] Fix NPE in IsolatedBookieEnsemblePlacementPolicy when policy class does not match

([#25803](https://github.com/apache/pulsar/pull/25803)) \[fix]\[broker] Fix PersistentMessageExpiryMonitor findEntryComplete() method may lose mark-delete properties in race condition

([#25865](https://github.com/apache/pulsar/pull/25865))  Return 400 for invalid reader messageId query parameter

([#25862](https://github.com/apache/pulsar/pull/25862)) \[fix]\[broker] Fix compaction cursor reset may lose mark-delete properties

([#25889](https://github.com/apache/pulsar/pull/25889)) \[fix]\[test] Fix flaky PulsarFunctionTlsTest.testFunctionsCreation() test

([#25864](https://github.com/apache/pulsar/pull/25864)) \[fix]\[test] Fix flaky ProducerCleanupTest timer cleanup

([#25867](https://github.com/apache/pulsar/pull/25867)) \[fix]\[fn] Fix Go function runtime to continue after user exceptions and add neg-ack tests

([#25866](https://github.com/apache/pulsar/pull/25866)) \[fix]\[test] Stabilize WebService rate limiting test

### MoP

Change error log to warn

Fix not release entry issue

### KoP

\[fix] Improve error log for OauthValidatorCallbackHandler

abc19ceb8 Increase the unload time for NotOwnedBundleHandler

### Function Mesh Worker Service

feat: support Kafka functions in registry service

c397c788 fix build

registry-service: Support agent trigger client pagination

feat: support orca managed agents

## Security Fixes

### Apache Pulsar

([#25918](https://github.com/apache/pulsar/pull/25918)) \[fix]\[sec] Upgrade Netty to 4.1.135.Final to address several CVEs


# V4.2.1.10
Source: https://docs.streamnative.io/release-notes/pulsar/v4.2/v4.2.1.10



# StreamNative Weekly Release Notes v4.2.1.10

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.2.1.10](https://github.com/streamnative/pulsar/releases/tag/v4.2.1.10)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.2.1.10/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.2.1.10/images/sha256-7e04d8af23111f38839283dd72f44539e70f22ae03be79aa52ff36ffbdbecdfa)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.2.1.10/images/sha256-9f94405d448746c3ecdd669eb4005331e203c7d77eeb0ef8806f19c2914ddf75)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.2.1.10/images/sha256-9f94405d448746c3ecdd669eb4005331e203c7d77eeb0ef8806f19c2914ddf75)

## General Changes

### Apache Pulsar

([#26242](https://github.com/apache/pulsar/pull/26242)) \[fix]\[broker] Fix delayed-delivery bucket merge failures when delayedDeliveryMaxNumBuckets is 1-3

([#25857](https://github.com/apache/pulsar/pull/25857)) \[fix]\[client] Avoid exception in ConsumerImpl hasMessageAvailable before first receive

([#26247](https://github.com/apache/pulsar/pull/26247)) \[fix]\[ml] Tolerate concurrent creation of the managed ledger z-node

([#26203](https://github.com/apache/pulsar/pull/26203)) \[fix]\[broker] Log exception in PulsarMetadataEventSynchronizer failure path

([#26141](https://github.com/apache/pulsar/pull/26141)) \[fix]\[test] Await reader reconnect after seek() to fix flaky TopicReaderTest assertions

([#26121](https://github.com/apache/pulsar/pull/26121)) \[improve]\[fn] Standardize log4j2 Root logger configuration to use system property

([#26219](https://github.com/apache/pulsar/pull/26219)) \[improve]\[broker]\[branch-4.2] Upgrade bookkeeper to 4.17.4

### MoP

04d1aa56 fix

### KoP

\[fix] Remove the NotOwnedBundleHandler

### StreamNative Pulsar Plugins

3cd25b00e Cleanup docker disk usage

### Function Mesh Worker Service

9aad6581 fix bump version error

registry-service: handle missing connector config definitions

registry-service: Align list pagination with Claude

## Security Fixes

### Apache Pulsar

([#26250](https://github.com/apache/pulsar/pull/26250)) \[fix]\[sec] Upgrade lz4-java to 1.11.1 to address CVE-2026-59949

([#26235](https://github.com/apache/pulsar/pull/26235)) \[fix]\[sec] Upgrade grpc in pulsar-function-go to 1.82.1 to fix GHSA-hrxh-6v49-42gf

([#26231](https://github.com/apache/pulsar/pull/26231)) \[fix]\[sec] Bump google.golang.org/grpc from 1.79.3 to 1.82.1 in /pulsar-function-go/examples

([#26270](https://github.com/apache/pulsar/pull/26270)) \[fix]\[sec]\[branch-4.2] Upgrade Spring to 7.0.8


# V4.2.1.4
Source: https://docs.streamnative.io/release-notes/pulsar/v4.2/v4.2.1.4



# StreamNative Weekly Release Notes v4.2.1.4

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.2.1.4](https://github.com/streamnative/pulsar/releases/tag/v4.2.1.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.2.1.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.2.1.4/images/sha256-a64cab7dc00502f79ed2e15ea1759ff772cd89bdcb9bc926ca93cdb1257cd951)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.2.1.4/images/sha256-debc0f2ce08f02406225c3849826311f9dbd1f1d1dcaa41fa8ba5ace1c3dc05c)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.2.1.4/images/sha256-debc0f2ce08f02406225c3849826311f9dbd1f1d1dcaa41fa8ba5ace1c3dc05c)

## General Changes

## Security Fixes


# V4.2.1.5
Source: https://docs.streamnative.io/release-notes/pulsar/v4.2/v4.2.1.5



# StreamNative Weekly Release Notes v4.2.1.5

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.2.1.5](https://github.com/streamnative/pulsar/releases/tag/v4.2.1.5)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.2.1.5/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.2.1.5/images/sha256-eb83fe30c870fc4e1713de81b0cfda98243da3bd947b92469bca3782c7186049)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.2.1.5/images/sha256-05637b79cbe732aca79b0fa8916a65f26c6c72b5ceb04535e49bbb443b95aad5)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.2.1.5/images/sha256-05637b79cbe732aca79b0fa8916a65f26c6c72b5ceb04535e49bbb443b95aad5)

## General Changes

### Apache Pulsar

([#26079](https://github.com/apache/pulsar/pull/26079)) \[feat]\[broker] Expose managed ledger properties via topic internal stats

([#26075](https://github.com/apache/pulsar/pull/26075)) \[fix]\[broker] Avoid attaching a consumer to a migrated non-persistent topic on subscribe

([#26065](https://github.com/apache/pulsar/pull/26065)) \[fix]\[meta] Run ledger-underreplication notification callbacks off the metadata-store listener thread

([#26064](https://github.com/apache/pulsar/pull/26064)) \[fix]\[client] Run the failover health probe off the Netty event-loop thread

([#25675](https://github.com/apache/pulsar/pull/25675)) \[fix]\[test] Make SameAuthParamsLookupAutoClusterFailoverTest less timing-sensitive

([#26038](https://github.com/apache/pulsar/pull/26038)) \[improve]\[test]Add test: test/testTopicPartitionCannotBeCreatedAfterTopicDeleted

([#26002](https://github.com/apache/pulsar/pull/26002)) \[fix]\[broker] Fix geo-replication stuck after a failed publish to the remote cluster

([#26059](https://github.com/apache/pulsar/pull/26059)) \[fix] functions: Run worker leader-election off the consumer event-listener thread

([#26054](https://github.com/apache/pulsar/pull/26054)) \[fix]\[broker] Avoid blocking the bundle-throughput lookup on per-bundle metadata reads

([#26053](https://github.com/apache/pulsar/pull/26053)) \[fix]\[broker] Avoid blocking the dispatcher close path on delayed-delivery tracker close

([#26052](https://github.com/apache/pulsar/pull/26052)) \[fix]\[proxy] Avoid blocking the proxy IO thread on a cold broker cache

([#26051](https://github.com/apache/pulsar/pull/26051)) \[fix]\[broker] Avoid blocking metadata read on the IO thread when redirecting migrated producers/consumers

([#26044](https://github.com/apache/pulsar/pull/26044)) \[fix]\[broker] Prevent topic policy initialization race with a buffering listener wrapper

([#26049](https://github.com/apache/pulsar/pull/26049)) \[fix]\[test] Fix flaky testPrepareInitPoliciesCacheAsyncThrowExceptionAfterCreateReader

([#26040](https://github.com/apache/pulsar/pull/26040)) \[fix]\[broker] Run the message expiry check off the topic policy update path

([#26042](https://github.com/apache/pulsar/pull/26042)) \[fix]\[broker] Run topic policy notifications on the topic-ordered executor

([#26046](https://github.com/apache/pulsar/pull/26046)) \[fix]\[fn] Make exclusiveLeaderProducer volatile in FunctionMetaDataManager

([#26033](https://github.com/apache/pulsar/pull/26033)) \[improve]\[fn] Upgrade pulsar-client-python to 3.12.0

([#26031](https://github.com/apache/pulsar/pull/26031)) \[fix]\[broker] Fail fast for load balancer misconfigurations instead of falling back to SimpleLoadManagerImpl

([#26025](https://github.com/apache/pulsar/pull/26025)) \[fix]\[broker] Don't let a stuck or aborted topic policies cache init make a namespace's topics unloadable

([#26026](https://github.com/apache/pulsar/pull/26026)) \[fix]\[broker] Fix forced topic/namespace deletion still hanging when the compaction reader reconnect stalls

([#26016](https://github.com/apache/pulsar/pull/26016)) \[fix]\[broker] Fix forced topic/namespace deletion hanging or failing when compaction is in progress

([#26015](https://github.com/apache/pulsar/pull/26015)) \[fix]\[broker] Prevent subscribe rate limit from stalling compaction and blocking forced deletion

([#26012](https://github.com/apache/pulsar/pull/26012)) \[fix]\[broker] Fix delayed messages stalling with isDelayedDeliveryDeliverAtTimeStrict=true

([#26045](https://github.com/apache/pulsar/pull/26045)) \[fix]\[client] Prevent client shutdown from leaking event loop threads when DNS resolver close fails

([#26000](https://github.com/apache/pulsar/pull/26000)) \[fix]\[meta] Keep the leader value in the election cycle and make leader reads authoritative

([#25998](https://github.com/apache/pulsar/pull/25998)) \[fix]\[broker] Fix compacted read could be stuck forever or message loss due to cursor mark delete

### KoP

Fix the internal Kafka client could be configured with TLS but no certificates

### pulsarctl

feat: add set-replication-clusters command for topics

feat(cmd): add custom runtime options injection

### Function Mesh Worker Service

31b58845 stablize CI

registry-service: Split agent session token metrics

### Pulsar Tiered Storage v2

refactor(offload): key shared object-storage clients by destination only

improve(gcs): raise default read-executor pool to min(cpu\*10, 60)

fix(tests): add utility to ensure test namespace readiness

test(e2e): add scenario 1d probe for V1 block-index crash (+ drive-by: unify IT broker image)

feat(mertics): add read ahead cache prometheus metrics

refactor(offload): exclude deletes from offload concurrency limiter

refactor(offload): broker-wide offload concurrency limiter, split from client cache

feat(read): bound in-flight cloud-read bytes with backpressure

feat(read): share ReadAheadCache broker-wide

improve(storage): add s3 and gcs upload tuning config

## Security Fixes

### Apache Pulsar

([#26068](https://github.com/apache/pulsar/pull/26068)) \[fix]\[sec] Upgrade jline to 4.2.1 and picocli to 4.7.7, drop unused jline2


# V4.2.1.6
Source: https://docs.streamnative.io/release-notes/pulsar/v4.2/v4.2.1.6



# StreamNative Weekly Release Notes v4.2.1.6

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.2.1.6](https://github.com/streamnative/pulsar/releases/tag/v4.2.1.6)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.2.1.6/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.2.1.6/images/sha256-bd7bea26846a6ed4a12fd9a63432d8ca5ab6a6870d8769d415101ec8eabd3b86)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.2.1.6/images/sha256-e1f819d92e4da9ebbb6026e97bcb75ecbdeb49c00e89d1c327f4854fff85f449)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.2.1.6/images/sha256-e1f819d92e4da9ebbb6026e97bcb75ecbdeb49c00e89d1c327f4854fff85f449)

## General Changes

### Apache Pulsar

([#26136](https://github.com/apache/pulsar/pull/26136)) \[fix]\[fn] Reorder Function Worker shutdown to stop scheduler before runtime manager

([#26139](https://github.com/apache/pulsar/pull/26139)) \[improve]\[fn] Upgrade pulsar-client-python to 3.13.0

([#26134](https://github.com/apache/pulsar/pull/26134)) \[improve]\[broker] Load topic policies on non-persistent topic load and gate the policy replay

([#26123](https://github.com/apache/pulsar/pull/26123)) \[fix]\[test] Run makeReadEntryProbFail's errorOrNot on a caller-provided executor

([#26083](https://github.com/apache/pulsar/pull/26083)) \[fix]\[test] Fix flaky PersistentTopicsTest setup caused by concurrent Mockito stubbing

([#26122](https://github.com/apache/pulsar/pull/26122)) \[fix]\[test] Fix flaky AuditorBookieTest.testBookieClusterRestart

([#26132](https://github.com/apache/pulsar/pull/26132)) \[fix]\[broker] Don't let a closing topic-policies reader abort a concurrent cache-init reload

([#26106](https://github.com/apache/pulsar/pull/26106)) \[fix]\[broker] Fix replication stall when a cursor rewind skips an in-flight read

([#25645](https://github.com/apache/pulsar/pull/25645)) \[fix]\[test] Fix flaky SchemaServiceTest.testSchemaRegistryMetrics

([#26110](https://github.com/apache/pulsar/pull/26110)) \[fix]\[broker] Forward topic policy updates after init failures

([#26005](https://github.com/apache/pulsar/pull/26005)) \[fix]\[broker] Fix replicator getting stuck under rate limiter throttling and honor readBatchSize/maxReadSizeBytes on the default read path

([#26055](https://github.com/apache/pulsar/pull/26055)) \[improve]\[broker] Improve dispatch performance by summing entry bytes with a loop

([#26080](https://github.com/apache/pulsar/pull/26080)) \[fix]\[broker] Guard BucketDelayedDeliveryTracker.nextDeliveryTime against empty queues

([#25984](https://github.com/apache/pulsar/pull/25984)) \[improve]\[broker] Trim orphaned bucket snapshots when ledgers are deleted

([#25915](https://github.com/apache/pulsar/pull/25915)) \[fix]\[broker]Do not trigger topic GC if replication is still active

### MoP

cc9fb9cd test: isolate mock ZooKeeper sessions per broker

### KoP

Support handling non-partitioned topics for DescribeTopicPartitions and configs

488bc0e87 \[branch-4.2] Remove explicit Netty dependencies because they are not included in sn-bom

\[feature] Refactor JSON schema compatibility checker

\[branch-4.2] Use sn-bom to manage all dependency versions

29ac2b707 \[branch-4.2] Bump TestNG version to 7.12.0

Improve earliest offset query performance by recording start offset in managed ledger properties

\[fix] lock-ordering deadlock between partitionLock and group lock in storeOffsetMessageAsync

### StreamNative Pulsar Plugins

ff575d44c Upgrade the bk version

fix: upgrade golang.org/x/net to v0.55.0 to fix multiple CVEs

### Function Mesh Worker Service

feat: support session-local agent tool/mcp\_servers overrides on session update

feat: Align managed agent endpoints

feat: use agent id for agent trigger and support agent version

Support cron agent triggers in worker service

feat: Align managed agent vault API validation

\[codex] Support Claude memory and file APIs

### Pulsar Tiered Storage v2

feat(offloader): configurable GCS/S3 upload tuning

feat(gcs): configurable read/upload buffering (fix Old-Gen churn) + parallel composite upload

fix(azure): let streamnative-bom manage azure-identity (matches what we ship)

## Security Fixes

### Apache Pulsar

([#26142](https://github.com/apache/pulsar/pull/26142)) \[fix]\[sec] Bump github.com/go-jose/go-jose/v4 from 4.1.3 to 4.1.4 in /pulsar-function-go

([#26140](https://github.com/apache/pulsar/pull/26140)) \[fix]\[sec] Upgrade pulsar-client-go to v0.20.0 in pulsar-function-go, also address CVEs

([#26099](https://github.com/apache/pulsar/pull/26099)) \[fix]\[sec]\[branch-4.2] Upgrade Jackson version to 2.18.8


# V4.2.1.7
Source: https://docs.streamnative.io/release-notes/pulsar/v4.2/v4.2.1.7



# StreamNative Weekly Release Notes v4.2.1.7

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.2.1.7](https://github.com/streamnative/pulsar/releases/tag/v4.2.1.7)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.2.1.7/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.2.1.7/images/sha256-9f5a0044fdc8a5775e79d6d2bd9f999cebe99f48a3576a4464f7469ceea3bfda)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.2.1.7/images/sha256-f06b022c7cceda4b9f7c64736e9fceaa23911f1101b43073d43cddc62bf6da77)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.2.1.7/images/sha256-f06b022c7cceda4b9f7c64736e9fceaa23911f1101b43073d43cddc62bf6da77)

## General Changes

### Apache Pulsar

([#26158](https://github.com/apache/pulsar/pull/26158)) \[fix]\[metadata] Fix orphaned UR parent nodes not cleaned up with Oxia metadata backend

([#26150](https://github.com/apache/pulsar/pull/26150)) \[improve]\[meta] Support tuning Oxia MetadataStoreConfig through metadata-store URIs

### KoP

Fix OffsetFetch duplicate partition handling

Implement DeleteRecords request correctly

Skip Pulsar message deduplication snapshot for Kafka topics

### Function Mesh Worker Service

5ede611b fix build error

fix(registry): add 'type' field to EnvironmentPackages

81ac6a3a fix(registry): fix QueryParam import for ListAgentsParams

feat: test registry agents api with orca managed agent

registry-service: Type environment config unions

### Pulsar Tiered Storage v2

feat(metrics): unify tiered\_storage\_\* prefix, add offload waiting/max + cache usage/eviction

## Security Fixes


# V4.2.1.8
Source: https://docs.streamnative.io/release-notes/pulsar/v4.2/v4.2.1.8



# StreamNative Weekly Release Notes v4.2.1.8 (rapid)

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.2.1.8](https://github.com/streamnative/pulsar/releases/tag/v4.2.1.8)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.2.1.8/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.2.1.8/images/sha256-ef4cab40204783f0bf999e38a0219c76cf46e5caf498c4d6a6ac07593730a23a)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.2.1.8/images/sha256-01af67ebb3bea27812103f614ae9ae8f48df9b7c64c38c26a95441c396ebacf2)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.2.1.8/images/sha256-01af67ebb3bea27812103f614ae9ae8f48df9b7c64c38c26a95441c396ebacf2)

## General Changes

### Apache Pulsar

([#26188](https://github.com/apache/pulsar/pull/26188)) \[fix]\[broker] Prevent early replay of non-strict delayed messages

([#26182](https://github.com/apache/pulsar/pull/26182)) \[improve]\[monitor]\[branch-4.2] Upgrade OpenTelemetry libraries

### AoP

\[fix] Fix flaky test MultiBundlesTest

### StreamNative Pulsar Plugins

Fix OIDC auth metrics recording and add pool match warning log

### Function Mesh Worker Service

ci: Support managed-agent sandbox isolation

f11947da fix ci

fix: remove debug logs

registry-service: Add snServiceAccount registry config mapping

feat(agents): add agent binding update functionality

ci: Add agent trigger e2e coverage

registry-service: Support session-scoped files

### Pulsar Tiered Storage v2

feat(s3): support S3 Object Lock offload buckets

fix(read): skip prefetch after a large (cache-bypassing) read

## Security Fixes

### Apache Pulsar

([#26194](https://github.com/apache/pulsar/pull/26194)) \[fix]\[sec]\[branch-4.2] Upgrade Hadoop to 3.5.0

([#26186](https://github.com/apache/pulsar/pull/26186)) \[fix]\[sec]\[branch-4.2] Upgrade Jackson version to 2.18.9

([#26168](https://github.com/apache/pulsar/pull/26168)) \[fix]\[sec]\[branch-4.2] Upgrade Netty to 4.1.136.Final


# V4.2.1.9
Source: https://docs.streamnative.io/release-notes/pulsar/v4.2/v4.2.1.9



# StreamNative Weekly Release Notes v4.2.1.9

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.2.1.9](https://github.com/streamnative/pulsar/releases/tag/v4.2.1.9)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.2.1.9/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.2.1.9/images/sha256-998b1fa04adcee7bcf740a17b86d69049a598fa687ab1e212905a9a3163e4f16)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.2.1.9/images/sha256-f6f389b951dfbf9f2f84a2184a9a643f3c633c27564e0cbee17e9de4f11ecb0b)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.2.1.9/images/sha256-f6f389b951dfbf9f2f84a2184a9a643f3c633c27564e0cbee17e9de4f11ecb0b)

## General Changes

### Apache Pulsar

([#26089](https://github.com/apache/pulsar/pull/26089)) \[fix]\[broker] Release entry on GetLastMessageId when parseMessageMetadata throws

([#26146](https://github.com/apache/pulsar/pull/26146)) \[fix]\[broker] Prevent stale service unit callbacks from dropping active lookup and cleanup jobs

([#26243](https://github.com/apache/pulsar/pull/26243)) \[fix]\[broker] Fix TableViewLoadDataStoreImpl close deadlock that stalls broker shutdown

([#26001](https://github.com/apache/pulsar/pull/26001)) \[fix]\[client] Fix unAckedMessageTracker cleanup on multi-topics batch ack

([#26191](https://github.com/apache/pulsar/pull/26191)) \[fix]\[fn] Forward source message properties in Python runtime

([#26237](https://github.com/apache/pulsar/pull/26237)) \[fix]\[broker] Fix silently dropped acknowledgement failures in PulsarMetadataEventSynchronizer

([#26240](https://github.com/apache/pulsar/pull/26240)) \[fix]\[broker] Fix delayed message index data loss when trimming overlapping bucket snapshots

([#26245](https://github.com/apache/pulsar/pull/26245)) \[fix]\[broker] Fix incorrect listener URLs returned by ModularLoadManager lookups

([#26119](https://github.com/apache/pulsar/pull/26119)) \[fix]\[broker] Prevent completing replicated snapshot before marker publish

([#26145](https://github.com/apache/pulsar/pull/26145)) \[fix]\[broker] Prevent stale topic unload cleanup from removing active cache entries

([#26199](https://github.com/apache/pulsar/pull/26199)) \[fix]\[meta] Complete handleMetadataEvent future exceptionally when the initial get fails

([#26201](https://github.com/apache/pulsar/pull/26201)) \[fix]\[meta] Record get op stats on the correct completion branch in AbstractMetadataStore

([#26236](https://github.com/apache/pulsar/pull/26236)) \[fix]\[broker] Fix Key\_Shared delivery stall when look-ahead triggers at the end of the topic

([#26218](https://github.com/apache/pulsar/pull/26218)) \[fix]\[meta] Fix RocksdbMetadataStore instanceId not advancing across restarts

([#26232](https://github.com/apache/pulsar/pull/26232)) \[improve]\[offload] Support credentials from offload policies for S3 and Aliyun OSS drivers

([#26228](https://github.com/apache/pulsar/pull/26228)) \[fix]\[ml] Preserve ledger entries/size when transformLedgerInfo callback completes after a concurrent close

([#26174](https://github.com/apache/pulsar/pull/26174)) \[fix]\[broker] Prevent stale read completions from stranding Failover subscriptions

([#26233](https://github.com/apache/pulsar/pull/26233)) \[improve]\[misc] Upgrade Jetty to 12.1.11

([#26234](https://github.com/apache/pulsar/pull/26234)) \[fix]\[broker] Trigger max read position callback for messages published during transaction buffer recovery

([#26230](https://github.com/apache/pulsar/pull/26230)) \[fix]\[broker] Check deliverAt before containsMessage in bucket addMessage

([#26227](https://github.com/apache/pulsar/pull/26227)) \[fix]\[ml] Preserve ledger properties when closing ledger

([#26225](https://github.com/apache/pulsar/pull/26225)) \[improve]\[build] Upgrade docker base image Alpine to 3.24

([#26226](https://github.com/apache/pulsar/pull/26226)) \[improve]\[build] Upgrade slog to 0.10.0

([#26223](https://github.com/apache/pulsar/pull/26223)) \[fix]\[broker]\[branch-4.2] Fix admin API HTTP 400 FAIL\_ON\_TRAILING\_TOKENS when a broker interceptor is loaded

([#26198](https://github.com/apache/pulsar/pull/26198)) \[fix]\[test] Fix flaky test `testCompactionPriority `

([#26217](https://github.com/apache/pulsar/pull/26217)) \[fix]\[fn] Return inputSpecs consumerProperties in function GET info

([#26200](https://github.com/apache/pulsar/pull/26200)) \[fix]\[meta] Fix NPE in shouldIgnoreEvent when MetadataEvent options is null

([#26179](https://github.com/apache/pulsar/pull/26179)) \[fix]\[broker] Prevent partition expansion from inheriting delayed-delivery bucket state

([#26149](https://github.com/apache/pulsar/pull/26149)) \[improve]\[broker] Skip system cursor when check inactive cursor.

([#26143](https://github.com/apache/pulsar/pull/26143)) \[fix]\[client] Fix lookup permit double-release, waiting queue starvation and timeout-response races in ClientCnx

([#26193](https://github.com/apache/pulsar/pull/26193)) \[improve]\[meta] Upgrade Oxia client to 0.9.4

([#26163](https://github.com/apache/pulsar/pull/26163)) \[improve]\[broker] Trace the asynchronous tasks in logs when loading topics

([#26171](https://github.com/apache/pulsar/pull/26171)) \[fix]\[broker] Fix bucket delayed message index metrics reset on scrape

([#26160](https://github.com/apache/pulsar/pull/26160)) \[fix]\[broker] Fix BucketDelayedDeliveryTracker recovery after LightProto migration

([#26159](https://github.com/apache/pulsar/pull/26159)) \[fix]\[broker] Read subscription properties directly from cursor

([#26043](https://github.com/apache/pulsar/pull/26043)) \[fix]\[client] Fix UnAckedMessageRedeliveryTracker to skip cancelled timeouts

([#26135](https://github.com/apache/pulsar/pull/26135)) \[fix]\[client] Sync ackSet in client with broker to stop acked messages reaching the DLQ

([#26169](https://github.com/apache/pulsar/pull/26169)) \[fix]\[ci] Upgrade sandboxed-trivy-action to approved sha

([#25480](https://github.com/apache/pulsar/pull/25480)) \[improve]\[ci] Replace trivy-action with sandboxed-trivy-action

([#26221](https://github.com/apache/pulsar/pull/26221)) \[fix]\[test]\[branch-4.2] Fix ManagedCursorTest compilation

([#26184](https://github.com/apache/pulsar/pull/26184)) \[fix]\[broker] Fix `getEstimatedSizeSinceMarkDeletePosition` throw `IllegalArgumentException`

([#26196](https://github.com/apache/pulsar/pull/26196)) \[fix]\[client] Preserve null values in pulsar-admin schema output

### AoP

\[branch-4.2] Remove OpenTelemetry dependencies

### MoP

368b5e63 fix

04e487db fix

### KoP

Introduce size and time based snapshottable metadata for producer state

Skip metadata lookup for topics with errors in async lookup

Fix pending txn offsets never removed due to deleted TXN markers after compaction

### pulsarctl

Upgrade go version to avoid cve

### Cloud Pulsar Plugins

c344ea49 fix(tests): remove pinned opentelemetry spi version

Check auth provider name in the authorization stage

### Function Mesh Worker Service

Align with orca-managed-agents

registry-service: Support PostgreSQL managed-agent storage

registry-service: Expose session cache token metrics

ci: Seed managed-agents API key fingerprint

### Pulsar Tiered Storage v2

test(read): reproduce empty-range crash after oversized entries

fix(read): support v1 entries larger than read buffer

tune(read): lower default in-flight read budget to 15% of direct memory

## Security Fixes


# Use Kafka Tools With StreamNative Cloud
Source: https://docs.streamnative.io/tools/cli/other-tools/use-kafka-tools-with-streamnative-cloud



Apache Kafka provides a suite of command-line interface (CLI) tools that can be accessed from the `bin/` directory after [downloading](https://kafka.apache.org/downloads) and extracting the Kafka distribution. These tools offer a range of capabilities, including starting and stopping Kafka, managing topics, and handling partitions. To learn how to use each tool, simply run it with no argument or use the `--help` argument for detailed instructions.

You can use these tools by creating a configuration file that contains basic connectivity details such as the bootstrap server and a [API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview). You can use this file with any Kafka tool that accepts a configuration file. Some of the tools that provide a configuration option, and the option to specify the configuration file are listsed in the table that follows:

| Kafka Tool                  | Config property option |
| --------------------------- | ---------------------- |
| `kafka-configs.sh`          | `--command-config`     |
| `kafka-console-consumer.sh` | `--consumer.config`    |
| `kafka-console-producer.sh` | `--producer.config`    |
| `kafka-consumer-groups.sh`  | `--command-config`     |

## Create a configuration file

1. Create a file named `cloud.properties` and save it in a secure location. You will populate this file with credentials to access your StreamNative Cloud account so you **must** keep in a safe place. Add the following content to the file:

   ```properties theme={null}
   bootstrap.servers=<broker-endpoint>
   security.protocol=SASL_SSL
   sasl.mechanism=PLAIN
   sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="unused" password="token:<api-key>";
   ```

2. Next, you will populate the file with your StreamNative Cloud cluster information.

   1. Sign in to your StreamNative Cloud account.

   2. In the Cloud Console, navigate to the cluster you want to connect to.

   3. In the **Cluster Dashboard** page, select the **Details** tab.

   4. Copy the **Kafka Service URL (TCP)** value and paste it into the `bootstrap.servers` property in the `cloud.properties` file.

   5. Follow the instructions in [Create an API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#create-an-api-key) to create an API key and paste the **API Key** value into the `cloud.properties` file.

## Use the tool

After you have set up the configuration file that references your cluster, you can use it with some of the Kafka tools. You will also need the bootstrap server of the cluster when you run the tool.

The following example demonstrates how to write messages to a topic named `test` in your StreamNative Cloud cluster using the `kafka-console-producer.sh` tool. The command requires both the configuration file and bootstrap server to be specified as options. Before running this command:

1. Ensure the topic `test` exists in your cluster
2. Replace `<bootstrap-server>` with your cluster's actual bootstrap server address
3. Update the path to point to your `cloud.properties` file location

```bash theme={null}
./bin/kafka-console-producer.sh --producer.config /path/to/secure/location/cloud.properties --bootstrap-server <bootstrap-server> --topic test
```

Now enter some messages at the prompt (>):

```bash theme={null}
>hello world
>another message
>3rd message
```

You can also use the `kafka-console-consumer.sh` tool to read messages from the topic.

```bash theme={null}
./bin/kafka-console-consumer.sh --consumer.config /path/to/secure/location/cloud.properties --bootstrap-server <bootstrap-server> --from-beginning --group test-group --topic test
```

You should see the following output:

```bash theme={null}
hello world
another message
3rd message
```


# Use kcat With StreamNative Cloud
Source: https://docs.streamnative.io/tools/cli/other-tools/use-kcat-with-streamnative-cloud



[`kcat`](https://github.com/edenhill/kcat) is a popular CLI tool that you can use to test and debug your StreamNative Cloud clusters using the Kafka protocol. You can use kcat to produce, consume, and list topic and partition information for Kafka topics. Described as "netcat for Kafka", it is a swiss-army knife of tools for inspecting and creating data in Kafka.

It is similar to Kafka Console Producer (`kafka-console-producer.sh`) and Kafka Console Consumer (`kafka-console-consumer.sh`), but it offers more features and is easier to use.

## Create a configuration file

Any librdkafka [configuration](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md) property can be set on the command line using `-X key=value`, or in a configuration file specified by `-F <config-file>`.

1. Create a file named `cloud.properties` and save it in a secure location. You will populate this file with credentials to access your StreamNative Cloud account so you **must** keep in a safe place. Add the following content to the file:

<Warning title="Warning">
  Due to length limitations, StreamNative's API Key should be specified in the command line using the `-X` option rather than in the configuration file.
</Warning>

```properties theme={null}
bootstrap.servers=<broker-endpoint>
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.username=unused
```

2. Next, you will populate the file with your StreamNative Cloud cluster information.

   1. Sign in to your StreamNative Cloud account.

   2. In the Cloud Console, navigate to the cluster you want to connect to.

   3. In the **Cluster Dashboard** page, select the **Details** tab.

   4. Copy the **Kafka Service URL (TCP)** value and paste it into the `bootstrap.servers` property in the `cloud.properties` file.

   5. Follow the instructions in [Create an API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#create-an-api-key) to create an API key and paste the **API Key** value into the `cloud.properties` file.

## Use the tool

After you have set up the configuration file that references your cluster and noted the API Key, you can use `kcat` to produce and consume messages. You will also need the bootstrap server of the cluster when you run the tool.

The following examples demonstrate how to use `kcat` to produce and consume messages.

### Produce messages

Run the following command to produce messages to a topic `test_kcat`. Before running this command:

1. Replace `<bootstrap-server>` with your cluster's actual bootstrap server address
2. Update the path to point to your `cloud.properties` file location
3. Replace `<api-key>` with your actual API key

```bash theme={null}
kcat -b <bootstrap-server> -t test_kcat -F /path/to/secure/location/cloud.properties -X "sasl.password=token:<api-key>" -P -K:
```

The `-K:` flag indicates that messages should be formatted as `key:value`. Enter messages in this format at the prompt:

```bash theme={null}
1:apple
2:orange
3:pear
4:grape
```

After that, you type `Ctrl-D` to send the messages to the topic.

### Consume messages

Now, you can consume messages from the topic `test_kcat` using the following command. Before running this command:

1. Replace `<bootstrap-server>` with your cluster's actual bootstrap server address
2. Update the path to point to your `cloud.properties` file location
3. Replace `<api-key>` with your actual API key

```bash theme={null}
kcat -b <bootstrap-server> -t test_kcat -F /path/to/secure/location/cloud.properties -X "sasl.password=token:<api-key>" -C -f 'Key: %k, Value: %s\n'
```

You should see the following output:

```bash theme={null}
Key: 1, Value: apple
Key: 2, Value: orange
Key: 3, Value: pear
Key: 4, Value: grape
```


# Use Pulsar Tools With StreamNative Cloud
Source: https://docs.streamnative.io/tools/cli/other-tools/use-pulsar-tools-with-streamnative-cloud



Apache Pulsar provides a suite of [command-line interface (CLI) tools](https://pulsar.apache.org/docs/reference-cli-tools/) that can be accessed from the `bin/` directory after [downloading](https://pulsar.apache.org/download/) and extracting the Pulsar distribution. These tools offer a range of capabilities, including starting and stopping Pulsar, managing tenants, namespace, & topics, and doing other operations such as benchmarking. To learn how to use each tool, simply run it with no argument or use the `--help` argument for detailed instructions.

Most of the Pulsar client CLI tools use a configuration file `client.conf` stored in the `conf/` directory to connect to a Pulsar cluster. You can edit this file to specify the required information to connect to your StreamNative Cloud cluster.

## Collect your cluster information

You need to gather the following information for your StreamNative Cloud cluster:

* **webServiceUrl**: The Pulsar Web Service URL used for accessing the Pulsar admin interface
* **brokerServiceUrl**: The Pulsar Broker Service URL used for connecting to the Pulsar brokers
* **API Key**: The API Key for your StreamNative Cloud account

1. Sign in to your StreamNative Cloud account.

2. In the Cloud Console, navigate to the cluster you want to connect to.

3. In the **Cluster Dashboard** page, select the **Details** tab.

4. Copy the **HTTP Service URL (TLS)** value and paste it into the `webServiceUrl` property in the `client.conf` file.

5. Copy the **Broker Service URL (TLS)** value and paste it into the `brokerServiceUrl` property in the `client.conf` file.

6. Follow the instructions in [Create an API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#create-an-api-key) to create an API key and copy the **API Key** value.

7. Add the following value `token:<api-key>` to the `authParams` property in the `client.conf` file. `<api-key>` is the value you copied in the previous step.

8. Update the `authPlugin` property in the `client.conf` file to `org.apache.pulsar.client.impl.auth.AuthenticationToken`.

## Use the tools

### Run admin commands

You can use the `pulsar-admin` tool to run admin operations on your StreamNative Cloud cluster. For example, you can use the following command to list all the available tenants in your cluster:

<Warning title="Warning">
  Please make sure the service account has the necessary permissions to run the command.
</Warning>

```bash theme={null}
./bin/pulsar-admin tenants list
```

### Produce messages

You can use the `pulsar-client` tool to produce messages to a Pulsar topic. For example, you can use the following command to produce 10 messages to a topic named `my-topic`:

```bash theme={null}
bin/pulsar-client produce -m "test" -n 10 my_topic
```

You should see the following output at the end of the command:

```bash theme={null}
10 messages successfully produced
```

### Consume messages

You can use the `pulsar-client` tool to consume messages from a Pulsar topic. For example, you can use the following command to consume 10 messages from a topic named `my-topic`:

```bash theme={null}
bin/pulsar-client consume -n 10 -p Earliest -s my_sub my_topic
```

You should see a similar output to the following:

```bash theme={null}
----- got message -----
publishTime:[1732941461552], eventTime:[0], key:[null], properties:[], content:test
sidebarTitle: Use Pulsar Tools
----- got message -----
publishTime:[1732941461636], eventTime:[0], key:[null], properties:[], content:test
----- got message -----
publishTime:[1732941461704], eventTime:[0], key:[null], properties:[], content:test
sidebarTitle: Use Pulsar Tools
----- got message -----
publishTime:[1732941461774], eventTime:[0], key:[null], properties:[], content:test
----- got message -----
publishTime:[1732941461843], eventTime:[0], key:[null], properties:[], content:test
sidebarTitle: Use Pulsar Tools
----- got message -----
publishTime:[1732941461913], eventTime:[0], key:[null], properties:[], content:test
----- got message -----
publishTime:[1732941461982], eventTime:[0], key:[null], properties:[], content:test
sidebarTitle: Use Pulsar Tools
----- got message -----
publishTime:[1732941462051], eventTime:[0], key:[null], properties:[], content:test
----- got message -----
publishTime:[1732941462121], eventTime:[0], key:[null], properties:[], content:test
sidebarTitle: Use Pulsar Tools
----- got message -----
publishTime:[1732941462189], eventTime:[0], key:[null], properties:[], content:test
```


# pulsarctl Command References
Source: https://docs.streamnative.io/tools/cli/pulsarctl/pulsarctl-command-references



## Available releases

The following table lists the available releases of `pulsarctl` and their reference documentation.

| Version    | Reference                                                                                   |
| ---------- | ------------------------------------------------------------------------------------------- |
| latest     | [Command reference](https://doc-references.streamnative.io/pulsarctl/latest/index.html)     |
| v2.10.3.3  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.10.3.3/index.html)  |
| v2.10.3.1  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.10.3.1/index.html)  |
| v2.10.2.4  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.10.2.4/index.html)  |
| v2.10.2.3  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.10.2.3/index.html)  |
| v2.10.2.2  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.10.2.2/index.html)  |
| v2.10.2.1  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.10.2.1/index.html)  |
| v2.10.1.12 | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.10.1.12/index.html) |
| v2.10.1.11 | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.10.1.11/index.html) |
| v2.10.1.10 | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.10.1.10/index.html) |
| v2.10.1.9  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.10.1.9/index.html)  |
| v2.10.1.8  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.10.1.8/index.html)  |
| v2.10.1.7  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.10.1.7/index.html)  |
| v2.10.1.6  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.10.1.6/index.html)  |
| v2.10.1.5  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.10.1.5/index.html)  |
| v2.10.1.4  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.10.1.4/index.html)  |
| v2.10.1.3  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.10.1.3/index.html)  |
| v2.10.1.2  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.10.1.2/index.html)  |
| v2.10.1.1  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.10.1.1/index.html)  |
| v2.10.0.7  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.10.0.7/index.html)  |
| v2.10.0.6  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.10.0.6/index.html)  |
| v2.10.0.5  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.10.0.5/index.html)  |
| v2.10.0.4  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.10.0.4/index.html)  |
| v2.10.0.3  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.10.0.3/index.html)  |
| v2.10.0.2  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.10.0.2/index.html)  |
| v2.10.0.1  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.10.0.1/index.html)  |
| v2.9.4.3   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.4.3/index.html)   |
| v2.9.4.2   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.4.2/index.html)   |
| v2.9.4.1   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.4.1/index.html)   |
| v2.9.3.21  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.3.21/index.html)  |
| v2.9.3.20  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.3.20/index.html)  |
| v2.9.3.19  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.3.19/index.html)  |
| v2.9.3.18  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.3.18/index.html)  |
| v2.9.3.17  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.3.17/index.html)  |
| v2.9.3.16  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.3.16/index.html)  |
| v2.9.3.15  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.3.15/index.html)  |
| v2.9.3.14  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.3.14/index.html)  |
| v2.9.3.13  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.3.13/index.html)  |
| v2.9.3.12  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.3.12/index.html)  |
| v2.9.3.11  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.3.11/index.html)  |
| v2.9.3.10  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.3.10/index.html)  |
| v2.9.3.9   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.3.9/index.html)   |
| v2.9.3.8   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.3.8/index.html)   |
| v2.9.3.7   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.3.7/index.html)   |
| v2.9.3.6   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.3.6/index.html)   |
| v2.9.3.5   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.3.5/index.html)   |
| v2.9.3.4   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.3.4/index.html)   |
| v2.9.3.3   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.3.3/index.html)   |
| v2.9.3.2   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.3.2/index.html)   |
| v2.9.3.1   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.3.1/index.html)   |
| v2.9.2.24  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.2.24/index.html)  |
| v2.9.2.23  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.2.23/index.html)  |
| v2.9.2.22  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.2.22/index.html)  |
| v2.9.2.21  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.2.21/index.html)  |
| v2.9.2.20  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.2.20/index.html)  |
| v2.9.2.19  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.2.19/index.html)  |
| v2.9.2.18  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.2.18/index.html)  |
| v2.9.2.17  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.2.17/index.html)  |
| v2.9.2.16  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.2.16/index.html)  |
| v2.9.2.15  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.2.15/index.html)  |
| v2.9.2.14  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.2.14/index.html)  |
| v2.9.2.13  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.2.13/index.html)  |
| v2.9.2.12  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.2.12/index.html)  |
| v2.9.2.11  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.2.11/index.html)  |
| v2.9.2.9   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.2.9/index.html)   |
| v2.9.2.8   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.2.8/index.html)   |
| v2.9.2.7   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.2.7/index.html)   |
| v2.9.2.6   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.2.6/index.html)   |
| v2.9.2.5   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.2.5/index.html)   |
| v2.9.2.4   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.2.4/index.html)   |
| v2.9.2.3   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.2.3/index.html)   |
| v2.9.2.2   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.2.2/index.html)   |
| v2.9.2.1   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.2.1/index.html)   |
| v2.9.1.5   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.1.5/index.html)   |
| v2.9.1.4   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.1.4/index.html)   |
| v2.9.1.3   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.1.3/index.html)   |
| v2.9.1.2   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.1.2/index.html)   |
| v2.9.1.1   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.9.1.1/index.html)   |
| v2.8.4.2   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.4.2/index.html)   |
| v2.8.4.1   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.4.1/index.html)   |
| v2.8.3.8   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.3.8/index.html)   |
| v2.8.3.7   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.3.7/index.html)   |
| v2.8.3.6   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.3.6/index.html)   |
| v2.8.3.5   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.3.5/index.html)   |
| v2.8.3.4   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.3.4/index.html)   |
| v2.8.3.3   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.3.3/index.html)   |
| v2.8.3.2   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.3.2/index.html)   |
| v2.8.3.1   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.3.1/index.html)   |
| v2.8.2.16  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.2.16/index.html)  |
| v2.8.2.15  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.2.15/index.html)  |
| v2.8.2.14  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.2.14/index.html)  |
| v2.8.2.13  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.2.13/index.html)  |
| v2.8.2.12  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.2.12/index.html)  |
| v2.8.2.11  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.2.11/index.html)  |
| v2.8.2.10  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.2.10/index.html)  |
| v2.8.2.9   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.2.9/index.html)   |
| v2.8.2.8   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.2.8/index.html)   |
| v2.8.2.7   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.2.7/index.html)   |
| v2.8.2.6   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.2.6/index.html)   |
| v2.8.2.5   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.2.5/index.html)   |
| v2.8.2.4   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.2.4/index.html)   |
| v2.8.2.2   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.2.2/index.html)   |
| v2.8.2.1   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.2.1/index.html)   |
| v2.8.2.0   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.2.0/index.html)   |
| v2.8.1.30  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.30/index.html)  |
| v2.8.1.29  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.29/index.html)  |
| v2.8.1.28  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.28/index.html)  |
| v2.8.1.26  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.26/index.html)  |
| v2.8.1.25  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.25/index.html)  |
| v2.8.1.24  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.24/index.html)  |
| v2.8.1.23  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.23/index.html)  |
| v2.8.1.22  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.22/index.html)  |
| v2.8.1.21  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.21/index.html)  |
| v2.8.1.20  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.20/index.html)  |
| v2.8.1.19  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.19/index.html)  |
| v2.8.1.18  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.18/index.html)  |
| v2.8.1.17  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.17/index.html)  |
| v2.8.1.16  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.16/index.html)  |
| v2.8.1.15  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.15/index.html)  |
| v2.8.1.14  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.14/index.html)  |
| v2.8.1.13  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.13/index.html)  |
| v2.8.1.12  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.12/index.html)  |
| v2.8.1.11  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.11/index.html)  |
| v2.8.1.10  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.10/index.html)  |
| v2.8.1.9   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.9/index.html)   |
| v2.8.1.8   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.8/index.html)   |
| v2.8.1.7   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.7/index.html)   |
| v2.8.1.6   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.6/index.html)   |
| v2.8.1.5   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.5/index.html)   |
| v2.8.1.4   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.4/index.html)   |
| v2.8.1.3   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.3/index.html)   |
| v2.8.1.2   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.2/index.html)   |
| v2.8.1.1   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.1/index.html)   |
| v2.8.1.0   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.1.0/index.html)   |
| v2.8.0.16  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.0.16/index.html)  |
| v2.8.0.15  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.0.15/index.html)  |
| v2.8.0.14  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.0.14/index.html)  |
| v2.8.0.13  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.0.13/index.html)  |
| v2.8.0.12  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.0.12/index.html)  |
| v2.8.0.11  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.0.11/index.html)  |
| v2.8.0.10  | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.0.10/index.html)  |
| v2.8.0.9   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.0.9/index.html)   |
| v2.8.0.8   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.0.8/index.html)   |
| v2.8.0.7   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.0.7/index.html)   |
| v2.8.0.6   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.0.6/index.html)   |
| v2.8.0.5   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.0.5/index.html)   |
| v2.8.0.4   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.0.4/index.html)   |
| v2.8.0.3   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.0.3/index.html)   |
| v2.8.0.2   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.0.2/index.html)   |
| v2.8.0.1   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.8.0.1/index.html)   |
| v2.7.4.8   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.4.8/index.html)   |
| v2.7.4.7   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.4.7/index.html)   |
| v2.7.4.6   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.4.6/index.html)   |
| v2.7.4.5   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.4.5/index.html)   |
| v2.7.4.3   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.4.3/index.html)   |
| v2.7.4.2   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.4.2/index.html)   |
| v2.7.3.9   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.3.9/index.html)   |
| v2.7.3.8   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.3.8/index.html)   |
| v2.7.3.5   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.3.5/index.html)   |
| v2.7.3.4   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.3.4/index.html)   |
| v2.7.3.2   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.3.2/index.html)   |
| v2.7.3.1   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.3.1/index.html)   |
| v2.7.2.9   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.2.9/index.html)   |
| v2.7.2.8   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.2.8/index.html)   |
| v2.7.2.7   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.2.7/index.html)   |
| v2.7.2.6   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.2.6/index.html)   |
| v2.7.2.5   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.2.5/index.html)   |
| v2.7.2.4   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.2.4/index.html)   |
| v2.7.2.3   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.2.3/index.html)   |
| v2.7.2.2   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.2.2/index.html)   |
| v2.7.2.1   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.2.1/index.html)   |
| v2.7.1.5   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.1.5/index.html)   |
| v2.7.1.4   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.1.4/index.html)   |
| v2.7.1.3   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.1.3/index.html)   |
| v2.7.1.2   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.1.2/index.html)   |
| v2.7.1.1   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.1.1/index.html)   |
| v2.7.1.0   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.1.0/index.html)   |
| v2.7.0.8   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.0.8/index.html)   |
| v2.7.0.7   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.0.7/index.html)   |
| v2.7.0.6   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.0.6/index.html)   |
| v2.7.0.5   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.0.5/index.html)   |
| v2.7.0.4   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.0.4/index.html)   |
| v2.7.0.3   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.0.3/index.html)   |
| v2.7.0.2   | [Command reference](https://doc-references.streamnative.io/pulsarctl/v2.7.0.2/index.html)   |
| v0.5.0     | [Command reference](https://doc-references.streamnative.io/pulsarctl/v0.5.0/index.html)     |
| v0.4.2     | [Command reference](https://doc-references.streamnative.io/pulsarctl/v0.4.2/index.html)     |
| v0.4.1     | [Command reference](https://doc-references.streamnative.io/pulsarctl/v0.4.1/index.html)     |
| v0.4.0     | [Command reference](https://doc-references.streamnative.io/pulsarctl/v0.4.0/index.html)     |
| v0.3.0     | [Command reference](https://doc-references.streamnative.io/pulsarctl/v0.3.0/index.html)     |
| v0.2.0     | [Command reference](https://doc-references.streamnative.io/pulsarctl/v0.2.0/index.html)     |
| v0.1.0     | [Command reference](https://doc-references.streamnative.io/pulsarctl/v0.1.0/index.html)     |


# Configure Access to Multiple Clusters (snctl)
Source: https://docs.streamnative.io/tools/cli/pulsarctl/pulsarctl-configure-access-multiple-clusters



This page shows how to configure access to multiple Pulsar clusters by using configuration files. After your clusters, authentication information, and contexts are defined in one or more configuration files, you can quickly switch between clusters by using the `pulsarctl context use` command.

<Note title="Note">
  A file that is used to configure access to a cluster is sometimes called a *pulsarconfig* file. This is a generic way of referring to configuration files. It does not mean that there is a file named `pulsarconfig`.
</Note>

<Warning title="Warning">
  Only use pulsarconfig files from trusted sources. Using a specially-crafted pulsarconfig file could result in malicious code execution or file exposure. If you must use an untrusted pulsarconfig file, inspect it carefully first, much as you would a shell script.
</Warning>

## Before you begin

You need to have a Pulsar cluster, and the `pulsarctl` command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one fully-managed Pulsar cluster on [StreamNative Cloud](/cloud/get-started/quickstart-console), [spin a self-managed StreamNative Private Cloud cluster](/private-cloud/v2/quick-start/private-cloud-quickstart), or [run a standalone Pulsar cluster](https://pulsar.apache.org/docs/3.1.x/getting-started-docker/) locally.

To check that `pulsarctl` is installed, run `pulsarctl --version`.

## Define contexts and authentication info

Suppose you have two clusters, one for development work and one for production work. The `development` cluster is self-managed in your own datacenter, using Token authentication, while the `production` cluster is fully-managed on StreamNative Cloud, using OAuth2 authentication.

Now, you can use `pulsarctl context set` to define your clusters and the corresponding authentication information.

First, you can create a `development` context to access your development cluster running at `https://1.2.3.4` using token stored in file `/path/to/token`.

```bash theme={null}
pulsarctl context set development \
  --admin-service-url https://1.2.3.4 \
  --token-file /path/to/token
```

Secondly, you can create a `production` context to access your production cluster running at `https://5.6.7.8` on StreamNative Cloud using the OAuth2 private key file `/path/to/credentials.json`.

<Note title="Note">
  Instead of manually configuring a context to access a StreamNative Cloud cluster, you can also use `snctl x update-pulsar-config` to [add the cluster to the *pulsarconfig* file](/tools/cli/snctl/snctl-overview#add-context-to-pulsarctl).

  Please note that `x` in `snctl x` is a sub command for a group of experimental commands.
</Note>

```bash theme={null}
pulsarctl context set production \
  --admin-service-url https://5.6.7.8 \
  --issuer-endpoint https://auth.streamnative.cloud \
  --key-file /path/to/credentials.json \
  --audience urn:sn:pulsar:myorg:production
```

Then you can use `pulsarctl context get` to retrieve the list of available contexts configured for `pulsarctl`. You should be able to see similar output as below.

```bash theme={null}
+---------+-------------+-----------------------------------------------------------------------------------+-----------------------+
| CURRENT |    NAME     |                                BROKER SERVICE URL                                 |  BOOKIE SERVICE URL   |
+---------+-------------+-----------------------------------------------------------------------------------+-----------------------+
| *       | production  | https://5.6.7.8                                                                   | http://localhost:8080 |
|         | development | https://1.2.3.4                                                                   | http://localhost:8080 |
+---------+-------------+-----------------------------------------------------------------------------------+-----------------------+
```

You are able to find the *pulsarconfig* file located at `${HOME}/.config/pulsar/config`. Run the following command to check the context of *pulsarconfig* file.

```bash theme={null}
cat ${HOME}/.config/pulsar/config
```

You should be able to see similar content as below:

```bash theme={null}
auth-info:
  development:
    locationoforigin: /Users/john.doe/.config/pulsar/config
    tls_trust_certs_file_path: ""
    tls_allow_insecure_connection: false
    token: ""
    tokenFile: /path/to/token
    issuer_endpoint: ""
    client_id: ""
    audience: ""
    scope: ""
    key_file: ""
  production:
    locationoforigin: /Users/john.doe/.config/pulsar/config
    tls_trust_certs_file_path: ""
    tls_allow_insecure_connection: false
    token: ""
    tokenFile: ""
    issuer_endpoint: https://auth.streamnative.cloud
    client_id: ""
    audience: urn:sn:pulsar:myorg:production
    scope: ""
    key_file: /path/to/credentials.json
contexts:
  development:
    admin-service-url: https://1.2.3.4
    bookie-service-url: http://localhost:8080
  production:
    admin-service-url: https://5.6.7.8
    bookie-service-url: http://localhost:8080
current-context: production
```

## Set the current context

When you define the context in `${HOME}/.config/pulsar/config`, you can quickly switch between clusters by using the following command (suppose you want to use the development cluster):

```bash theme={null}
pulsarctl context use development
```

Now, you are using the context of `development` cluster. And you can validate the current context by using `pulsarctl context current` command.

If you don't know the current available list of contexts, you can use the following command:

```bash theme={null}
pulsarctl context get
```

You will see a similar output as follows:

```bash theme={null}
+---------+-------------+-----------------------------------------------------------------------------------+-----------------------+
| CURRENT |    NAME     |                                BROKER SERVICE URL                                 |  BOOKIE SERVICE URL   |
+---------+-------------+-----------------------------------------------------------------------------------+-----------------------+
| *       | production  | https://5.6.7.8                                                                   | http://localhost:8080 |
|         | development | https://1.2.3.4                                                                   | http://localhost:8080 |
+---------+-------------+-----------------------------------------------------------------------------------+-----------------------+
```

## Rename the context

You can modify the context name by using the following command:

```bash theme={null}
pulsarctl context rename <old_name> <new_name>
```

## Delete the context

If the cluster information is invalid or is not used anymore, you want to delete it. You can use the following command to delete a context:

```bash theme={null}
pulsarctl context delete development
```


# Get Started with Pulsar CLI (pulsarctl)
Source: https://docs.streamnative.io/tools/cli/pulsarctl/pulsarctl-overview



The Pulsar command-line tool, `pulsarctl`, enables developers to administer resources in a Pulsar cluster. The tool is [open-sourced](https://github.com/streamnative/pulsarctl) under [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0).

## Prerequisties

Before moving on to the subsequent steps, ensure you review the following requirements.

### Operating systems

The `pulsarctl` is compatible with the following operating systems and architectures only:

* macOS with 64-bit Intel chips (Darwin AMD64)
* macOS with Apple chips (Darwin ARM64)
* Windows with 64-bit Intel or AMD chips (Microsoft Windows AMD64)
* Linux with 64-bit Intel or AMD chips (Linux AMD64)
* Linux with 64-bit ARM chips (Linux ARM64)

### Network access

When the `pulsarctl` interacts with a Pulsar cluster, it requires network access to its admin service url (i.e., `https://pulsar-cluster-domain-name`).

## Install pulsarctl

This section describes how to install `pulsarctl` on Linux, MAC, and Windows Operating System (OS).

<Tabs>
  <Tab title="Linux">
    1. Use this command to install `pulsarctl` on the Linux operation system.

       ```bash theme={null}
       sudo bash -c "$(curl -fsSL https://raw.githubusercontent.com/streamnative/pulsarctl/master/install.sh)"
       ```

    2. Check whether `pulsarctl` is installed successfully.

       ```bash theme={null}
       pulsarctl --version
       ```
  </Tab>

  <Tab title="Mac">
    You can use the `curl` command or use Homebrew to install `pulsarctl` on a Mac.

    #### Install pulsarctl with curl command

    <span />

    1. Use this command to install `pulsarctl` on the Mac operation system.

       ```bash theme={null}
       sudo bash -c "$(curl -fsSL https://raw.githubusercontent.com/streamnative/pulsarctl/master/install.sh)"
       ```

    2. Check whether `pulsarctl` is installed successfully.

       ```bash theme={null}
       pulsarctl --version
       ```

    #### Install pulsarctl with Homebrew

    <span />

    1. Add the repository.

       ```bash theme={null}
       brew tap streamnative/streamnative
       ```

    2. Install pulsarctl.

       ```bash theme={null}
       brew install pulsarctl
       ```

    3. Check whether `pulsarctl` is installed successfully.

       ```bash theme={null}
       pulsarctl --version
       ```
  </Tab>

  <Tab title="Windows">
    To install `pulsarctl` on the Windows operation system, follow these steps:

    1. Download the latest release package from [here](https://github.com/streamnative/pulsarctl/releases).

    2. Extract the pulsarctl `.tar.gz` package using Windows Explorer.

    3. Add the `pulsarctl` directory to your `PATH`.

    4. Check whether `pulsarctl` is installed successfully.

       ```bash theme={null}
       pulsarctl --version
       ```
  </Tab>
</Tabs>

## Configure pulsarctl

This section describes how to configure `pulsarctl`.

### Configure pulsarctl for a StreamNative Cloud cluster

You can use `snctl` to configure a context for `pulsarctl` to be used in a StreamNative Cloud cluster.

1. [Initialize](/tools/cli/snctl/snctl-overview#configure-snctl) `snctl` configuration.
2. [Sign in](/tools/cli/snctl/snctl-overview#sign-in-to-an-organization) to an organization.
3. Use `snctl x update-pulsar-config --cluster-name` to [add a given cluster as a context](/tools/cli/snctl/snctl-overview#add-context-to-pulsarctl) to `pulsarctl`.
   ```bash theme={null}
   snctl x update-config --cluster-name <cluster-name>
   ```
4. Verify that the current context has been changed to `<cluster-name>`.
   ```bash theme={null}
   pulsarctl context current
   ```
5. After verifying that the cluster has been added to `pulsarctl` contexts, you can use `pulsarctl` to interact with the target cluster.

### Configure pulsarctl for a Pulsar cluster

You can configure pulsarctl for a Pulsar cluster with different authentication mechanisms.

<Tabs>
  <Tab title="OAuth2">
    You can use the following command to configure pulsarctl for a Pulsar cluster that is configured with OAuth2 authentication.

    ```bash theme={null}
    pulsarctl context set <CONTEXT_NAME> \
        --admin-service-url=<PULSAR_ADMIN_SERVICE_URL> \
        --issuer-endpoint=<OAUTH 2.0 ISSUER ENDPOINT> \
        --key-file=/path/to/credentials.json \
        --audience=<AUDIENCE>
    ```

    Notes: Please replace the following variables before using the command.

    * `<CONTEXT_NAME>`: The name is used for identifying the cluster.
    * `<PULSAR_ADMIN_SERVICE_URL>`: The admin service url of the cluster to connect.
    * `<OAUTH 2.0 ISSUER ENDPOINT>`: The OAuth 2.0 issuer endpoint for the client to connect to.
    * `/path/to/credentials.json`: The private key credentials file to use.
    * `<AUDIENCE>`: The audience to use for OAuth 2.0 authentication.
  </Tab>

  <Tab title="API Key">
    You can use the following commands to configure pulsarctl for a Pulsar cluster that is configured with Token authentication.

    ```bash theme={null}
    pulsarctl context set <CONTEXT_NAME> \
        --admin-service-url=<PULSAR_ADMIN_SERVICE_URL> \
        --token=<API_Key>
    ```

    or

    ```bash theme={null}
    pulsarctl context set <CONTEXT_NAME> \
        --admin-service-url=<PULSAR_ADMIN_SERVICE_URL> \
        --token-file=/path/to/apikey
    ```

    Notes: Please replace the following variables before using the command.

    * `<CONTEXT_NAME>`: The name is used for identifying the cluster.
    * `<PULSAR_ADMIN_SERVICE_URL>`: The admin service url of the cluster to connect.
    * `<API_KEY>`: The API Key to use.
    * `/path/to/apikey`: The file that contains the API Key.
  </Tab>

  <Tab title="mTLS">
    You can use the following commands to configure pulsarctl for a Pulsar cluster that is configured with mTLS authentication.

    ```bash theme={null}
    pulsarctl context set <CONTEXT_NAME> \
        --admin-service-url=<PULSAR_ADMIN_SERVICE_URL> \
        --tls-cert-file=/path/to/tls_cert_file \
        --tls-key-file=/path/to/tls_key_file
    ```

    Notes: Please replace the following variables before using the command.

    * `<CONTEXT_NAME>`: The name is used for identifying the cluster.
    * `<PULSAR_ADMIN_SERVICE_URL>`: The admin service url of the cluster to connect.
    * `/path/to/tls_cert_file`: The TLS cert file to use.
    * `/path/to/tls_key_file`: The TLS key file to use.
  </Tab>
</Tabs>


# Extend the Pulsar CLI with Plugins
Source: https://docs.streamnative.io/tools/cli/pulsarctl/pulsarctl-plugins



Pulsar CLI (`pulsarctl`) plugins enable you to extend the capabilities of the Pulsar CLI to interact with Pulsar resources. You can create simple and complex scripting workflows using the CLI and plugins.

To use plugins, you must have [`pulsarctl` installed](/tools/cli/pulsarctl/pulsarctl-overview##install-pulsarctl).

## Write a plugin

You can write a plugin in any programming or scripting language that allows you to write terminal commands.

### Plugin file name

A plugin’s command name is determined by its filename. The following rules apply:

* A plugin filename must begin with `pulsarctl-`.
* Subcommands in a plugin’s command are separated by dashes (`-`) in its filename. For example, a plugin named `streamnative-this-command` would define the command `streamnative this command`.
* To have a plugin command containing dashes (`-`) or underscores (`_`), use an underscore (`_`) in the plugin filenames in place of a dash (`-`). For example, you can invoke a plugin whose filename is `streamnative-that_command` by running the following commmand:
  ```bash theme={null}
  streamnative that_command
  ```
* On Linux and macOS, any file extension is supported as long as the file is executable.

### Naming limitations

The following limitations apply to naming plugins. If these rules are violated, the `plugin list` command output will have a warning message that the offending plugin will be ignored.

* A plugin can’t override an existing command. Therefore, a plugin whose name exactly matches a native CLI command’s name will be ignored.
* Two or more plugins can’t have the same name.
  The first one found on your `$PATH` is used. The other plugins discovered with the same name are ignored.

### Plugin flags and arguments

If the user invokes a plugin and passes in additional arguments and/or flags, it is the plugin’s responsibility to validate and parse them, as the CLI will pass in arguments and flags as-is.

For example, when running `pulsarctl example arg1 --flag=val arg2`, the `pulsarctl` will:

1. Look for the plugin with the longest possible name, `pulsarctl-example-arg1`.
2. Treat the last dash-separated value as an argument and try to find the next longest possible name, `pulsarctl-example` since the `pulsarctl-example-arg1` plugin is not found.
3. Repeat the search process until either a plugin is found or there are no more dash-separated values besides `pulsarctl-` meaning that no plugins matching the command have been found.
4. Invoke the found plugin and pass all arguments and flags after the plugin’s name (`arg1 --flag=val arg2`) as arguments to the plugin process, since `pulsarctl-example` exists.

## Install a Plugin

To install and use a plugin:

1. Make the plugin file executable:

   ```bash theme={null}
   sudo chmod +x <plugin file>
   ```

2. Place the plugin file on your PATH.

3. Execute the plugin.

   Note that plugin executables inherit the environment settings from the `pulsarctl`.

## Discover plugins

Plugins are user-created and may or may not be included with the Pulsar CLI. Use the `plugin list` command to search your PATH for plugin executables. This command lists plugin names in the order in which they are discovered.

## Plugin repository

You can find contributed plugins for use with the `pulsarctl` in [StreamNative’s GitHub repository](https://github.com/streamnative/pulsarctl/tree/master/plugins). You can also contribute a plugin for others to leverage as well. To do so, follow the steps to [add a plugin](https://github.com/streamnative/pulsarctl/blob/master/CONTRIBUTING.md).

## Example plugin

Here is an example plugin written in bash script to print a message. The plugin is saved in a file named `pulsarctl-foo`.

```bash theme={null}
#!/bin/bash

if [[ $1 == "args" ]]
then
    echo "I am the args of the pulsarctl-foo"
    exit 0
fi

echo "I am a plugin named pulsarctl-foo"
```

To use the above plugin, simply make the file `pulsarctl-foo` executable:

```bash theme={null}
chmod +x ./pulsarctl-foo
```

and place it anywhere in your `PATH`:

```bash theme={null}
mv pulsarctl-foo /usr/local/bin
```

You may now invoke your plugin as a kubectl command:

```bash theme={null}
pulsarctl foo
```

You will see the output as follows:

```
I am a plugin named pulsarctl-foo
```

All args and flags are passed as-is to the executable:

```bash theme={null}
pulsarctl foo args
```

You will see the output as follows:

```
I am the args of the pulsarctl-foo
```


# pulsarctl Quick Reference
Source: https://docs.streamnative.io/tools/cli/pulsarctl/pulsarctl-quick-reference



This quick reference covers `pulsarctl` syntax, describes the command operations, and provides common examples. For details about each command, including all the supported flags and subcommands, see the [pulsarctl reference documentation](/tools/cli/pulsarctl/pulsarctl-command-references).

For installation instructions, see [Installing pulsarctl](/tools/cli/pulsarctl/pulsarctl-overview#install-pulsarctl).

## Syntax

Use the following syntax to run `pulsarctl` command from your terminal window:

```
pulsarctl [resource] [command] [name] [flags]
```

where `resource`, `command`, `name`, and `flags` are:

* `resource`: Specifies the [resource type](#resource-type). Resource types are case-insensitive and you can specify the singular, plural, or abbreviated forms. For example, the following commands produce the same output:

  ```bash theme={null}
  pulsarctl topics get mytopic
  pulsarctl topic get mytopic
  ```

* `command`: Specifies the operation that you want to perform one one or more resources, for example `create`, `get`, `list`, `delete`.

* `name`: Specifies the name of the resource. Names are case-sensitive.

* `flags`: Specifies optional flags. For example, you can use the `-o` or `--output` flags to specify the output format of a `get` result.

<Warning title="Caution">
  Flags that you specify from the command line override default values and any corresponding environment variables.
</Warning>

If you need help, run `pulsarctl help` from the terminal window.

### Resource types (`resource`)

The following table includes a list of all the supported resource types and their descriptions.

The following table includes the descriptions for `resource`.

| NAME                | Description                                                                                                                                                                                                                                      |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| bookkeeper          | Operations about BookKeeper. In order to interact with the bookkeeper cluster, you need to specify `--bookie-service-url` when creating a context and make sure you are able to connect to the bookkeeper cluster via the specified service url. |
| broker-stats        | Operations to collect broker statistics                                                                                                                                                                                                          |
| brokers             | Operations about broker(s)                                                                                                                                                                                                                       |
| clusters            | Operations about Pulsar cluster(s)                                                                                                                                                                                                               |
| topics              | Operations about Pulsar topics                                                                                                                                                                                                                   |
| completion          | Generates shell completion scripts                                                                                                                                                                                                               |
| context             | Interface for setting and managing Pulsar Context(s)                                                                                                                                                                                             |
| functions           | Interface for managing Pulsar Functions (lightweight, Lambda-style compute processes that work with Pulsar)                                                                                                                                      |
| function-worker     | Operations to collect function-worker statistics                                                                                                                                                                                                 |
| namespaces          | Operations about namespaces                                                                                                                                                                                                                      |
| ns-isolation-policy | Operations about namespace isolation policy                                                                                                                                                                                                      |
| oauth2              | Operations about oauth2                                                                                                                                                                                                                          |
| package             | Operations about packages                                                                                                                                                                                                                        |
| plugin              | Operations about plugins                                                                                                                                                                                                                         |
| resource-quotas     | Operations about resource quotas                                                                                                                                                                                                                 |
| schemas             | Operations related to Schemas associated with Pulsar topics                                                                                                                                                                                      |
| sinks               | Interface for managing Pulsar IO sinks (egress data from Pulsar)                                                                                                                                                                                 |
| sources             | Interface for managing Pulsar IO Sources (ingress data into Pulsar)                                                                                                                                                                              |
| status              | Check service(broker or proxy) status                                                                                                                                                                                                            |
| subscriptions       | Operations about subscription(s)                                                                                                                                                                                                                 |
| tenants             | Operations about tenant(s)                                                                                                                                                                                                                       |
| token               | Operations of token                                                                                                                                                                                                                              |
| topics              | Operations about topic(s)                                                                                                                                                                                                                        |

### Operations (`command`)

Specifies the operation to be performed on one or more resources. This argument is required. Common operations include `create`, `get`, `delete`, `update`, and `list`.

To learn more about command operations, see the [pulsarctl reference](/tools/cli/pulsarctl/pulsarctl-command-references) documentation.

### Resource Name (`name`)

Specifies the name of the `resource`. This argument is required.

<Note title="Note">
  `name` is case-sensitive.
</Note>

For example, `pulsarctl topics list public/default`, where `pulsar/default` is the namespace name.

### Flags (`flags`)

Specifies the flags. This argument is optional.

For example, you can use the `-s` or `--admin-service-url` flags to specify the address and port of the admin web service URL that pulsarctl connects to.

<Note title="Note">
  Flags that you specify from the command line override the default values and corresponding environment variables.
</Note>

<Tip title="Tip">
  * If you need help, run `pulsarctl help` from the terminal window.
  * For more information about pulsarctl, see \[pulsarctl]\(link to pulsarctl website).
</Tip>

## Output options

Use the following sections for information about how you can format the output of certain commands. For details about which commands support the various output options, see the [pulsarctl reference](/tools/cli/pulsarctl/pulsarctl-command-references) documentation.

### Formatting output

The default output format for all `pulsarctl` commands is the human readable plain-text format. To output details to your terminal window in a specific format, you can add either the `-o` or `--output` flags to a supported `pulsarctl` command.

### Syntax

```bash theme={null}
pulsarctl [resource] [command] [name] -o <output_format>
```

Depending on the `pulsarctl` operation, the following output formats are supported:

| Output format | Description                      |
| ------------- | -------------------------------- |
| -o json       | Output a JSON formatted result.  |
| -o yaml       | Output a YAML formatted result.  |
| -o text       | Output a humand-readable result. |

## Enabling shell autocompletion

pulsarctl provides autocompletion support for Bash, Zsh, and Fish, which can save you a lot of typing.

### Zsh

The pulsarctl completion script for Zsh can be generated with the command `pulsarctl completion zsh`. Sourcing the completion script in your shell enables pulsarctl autocompletion.

To configure your zsh shell, run:

```bash theme={null}
mkdir -p ~/.zsh/completion/
pulsarctl completion zsh > ~/.zsh/completion/_pulsarctl
```

Include the directory in your \$fpath, for example by adding in \~/.zshrc:

```bash theme={null}
fpath=($fpath ~/.zsh/completion)
source ~/.zshrc
```

You may have to force rebuild zcompdump:

```bash theme={null}
rm -f ~/.zcompdump; compinit
```

### Bash

#### Introduction

The pulsarctl completion script for Bash can be generated with `pulsarctl completion bash`. Sourcing this script in your shell enables pulsarctl completion.

However, the pulsarctl completion script depends on `bash-completion` which you thus have to previously install.

> Warning: there are two versions of bash-completion, v1 and v2. V1 is for Bash 3.2 (which is the default on macOS), and v2 is for Bash 4.1+. The pulsarctl completion script doesn’t work correctly with bash-completion v1 and Bash 3.2. It requires bash-completion v2 and Bash 4.1+. Thus, to be able to correctly use pulsarctl completion on macOS, you have to install and use Bash 4.1+ (instructions). The following instructions assume that you use Bash 4.1+ (that is, any Bash version of 4.1 or newer).

#### Install bash-completion

> Note: As mentioned, these instructions assume you use Bash 4.1+, which means you will install bash-completion v2 (in contrast to Bash 3.2 and bash-completion v1, in which case pulsarctl completion won’t work).

You can test if you have bash-completion v2 already installed with `brew list | grep bash`. If not, you can install it with Homebrew:

```bash theme={null}
brew install bash-completion@2
```

As stated in the output of this command, add the following to your \~/.bashrc file:

```bash theme={null}
export BASH_COMPLETION_COMPAT_DIR="/usr/local/etc/bash_completion.d"
[[ -r "/usr/local/etc/profile.d/bash_completion.sh" ]] && . "/usr/local/etc/profile.d/bash_completion.sh"
```

#### Enable pulsarctl autocompletion

You now have to ensure that the pulsarctl completion script gets sourced in all your shell sessions. There are multiple ways to achieve this:

* First, you can use `bash` into the bash shell.

> Note: If you are using the bash shell, you can ignore it

* Add the completion script to the `/usr/local/etc/bash_completion.d` directory:

```bash theme={null}
pulsarctl completion bash >/usr/local/etc/bash_completion.d/pulsarctl.bash
```

* Source the completion script in your `~/.bashrc` file:

```bash theme={null}
echo 'source /usr/local/etc/bash_completion.d/pulsarctl.bash' >> ~/.bashrc
source ~/.bashrc
```

You can also use a shorthand alias for `pulsarctl` that also works with completion:

```bash theme={null}
alias p=pulsarctl
complete -o default -F \_\_start_pulsarctl p
```

### Fish

To load completions once in your current session run:

```bash theme={null}
pulsarctl completion fish | source
```

To load completions for each session, run:

```bash theme={null}
pulsarctl completion fish > ~/.config/fish/completions/pulsarctl.fish
```


# Tutorials: Manage Pulsar resources using pulsarctl
Source: https://docs.streamnative.io/tools/cli/pulsarctl/pulsarctl-tutorial



A **tenant** is an administrative unit for allocating capacity and enforcing an authentication or authorization scheme. After creating a cluster, you can create one or more tenants for the organization.

A **namespace** is a logical grouping of topics. After creating a tenant, you can create one or more namespaces for the tenant.

A **topic** is a named channel used to deliver messages published by producers to consumers. After creating a namespace, you can create one or more topics for the namespace.

## Work with tenants

You can create, update, and delete tenants using the pulsarctl CLI tool. For a full list of supported operations on tenants, see [pulsarctl command reference](https://doc-references.streamnative.io/pulsarctl/latest/index.html#-em-update-em--32).

Before using the pulsarctl CLI tool, you need to connect to a Pulsar cluster. For details, see [connect to Pulsar cluster on StreamNative Cloud](/tools/cli/pulsarctl/pulsarctl-overview) and [connect to Pulsar cluster on StreamNative Platform](/tools/cli/pulsarctl/pulsarctl-overview).

### Create a tenant

After creating a Pulsar cluster, you can create tenants for the Pulsar cluster. When creating a tenant, you need to use `-cluster` or `-c` option to specify the target cluster for the tenant.

This example shows how to create a tenant named `example-tenant` for the `example-cluster` cluster with the admin role. If you do not configure the admin role when creating the tenant, you cannot perform follow-up operations on the target tenant, such as updating or deleting tenants.

**Input**

```bash theme={null}
pulsarctl tenants create example-tenant -r bot@matrix.auth.streamnative -c example-cluster
```

**Output**

```bash theme={null}
Create tenant example-tenant successfully
```

### Manage a tenant

This section describes how to manage tenants through the pulsarctl CLI tool.

#### Update a tenant

When you want to add more admin roles for a tenant, you can use the `pulsarctl tenants update` command to update the target tenant.

This example shows how to update the admin role for `example-tenant`.

**Input**

```bash theme={null}
pulsarctl tenants update --admin-roles (bot) --admin-roles (bot1) example-tenant -c example-cluster
```

**Output**

```bash theme={null}
Update tenant example-tenant successfully
```

#### Delete a tenant

When you want to remove a tenant from a Pulsar cluster, you can delete it. If a tenant is associated with any resources, you cannot delete the tenant. In this case, you must delete its associated resources first.

<Note title="Note">
  You cannot delete a tenant if there are resources associated with the tenant.
</Note>

This example shows how to delete `example-tenant`.

**Input**

```bash theme={null}
pulsarctl tenants delete example-tenant
```

**Output**

```bash theme={null}
Delete tenant example-tenant successfully
```

## Work with namespaces

You can create and manage namespaces using the pulsarctl CLI tool. For a full list of supported operations on namespaces, see the [pulsarctl command reference](https://doc-references.streamnative.io/pulsarctl/latest/index.html#-em-update-em--32).

<Note title="Note">
  This section uses a tenant named `example-tenant` as an example. For details about how to create a tenant, see [work with tenants](/cloud/manage-data-streams/tenant).
</Note>

### Create a namespace

After creating and authorizing a tenant, you can create and manage namespaces and topics.

This example shows how to create a namespace named `example-ns` for `example-tenant`.

**Input**

```bash theme={null}
pulsarctl namespaces create example-tenant/example-ns
```

**Output**

```bash theme={null}
Created example-tenant/example-ns successfully
```

### Manage a namespace

This section describes how to manage namespaces using the pulsarctl CLI tool.

#### Clear namespace backlog

Pulsar stores all unacknowledged messages in backlogs until they are processed and acknowledged. You can clear backlogs of messages for a specific namespace to release more backlog quota for the namespace.

This example shows how to clear the backlog for all topics of `example-tenant/example-ns`.

**Input**

```bash theme={null}
pulsarctl namespaces clear-backlog example-tenant/example-ns
```

**Output**

```bash theme={null}
Are you sure you want to clear the backlog? (Y or N)

y

Successfully clear backlog for all topics of the example-tenant/example-ns
```

#### Unload a namespace

This example shows how to unload `example-tenant/example-ns` from the current serving broker.

**Input**

```bash theme={null}
pulsarctl namespaces unload example-tenant/example-ns
```

**Output**

```bash theme={null}
Unload namespace example-tenant/example-ns successfully
```

### Delete a namespace

<Note title="Note">
  You cannot delete a namespace if there are resources associated with the namespace.
</Note>

This example shows how to delete `example-tenant/example-ns`.

**Input**

```bash theme={null}
pulsarctl namespaces delete example-tenant/example-ns
```

**Output**

```bash theme={null}
Deleted example-tenant/example-ns successfully
```

## Work with topics

This section describes how to create and manage topics using the pulsarctl CLI tool. For a full list of supported operations on topics, see [pulsarctl command reference](https://doc-references.streamnative.io/pulsarctl/latest/index.html#-em-update-em--32).

Before using the pulsarctl CLI tool to create and manage topics, you need to create a [tenant](/cloud/manage-data-streams/tenant#create-a-tenant) and a [namespace](/cloud/manage-data-streams/namespace#create-a-namespace).

### Create a topic

You can use the `pulsarctl topics create TOPIC_NAME` command to create a topic.

* If you want to create a non-partitioned topic, you need to set the number of partitions to `0`.

  This example shows how to create a non-partitioned topic in the `example-tenant/example-ns` namespace.

  **Input**

  ```bash theme={null}
  pulsarctl topics create example-tenant/example-ns/topic-test 0
  ```

  **Output**

  ```bash theme={null}
  Create topic persistent://example-tenant/example-ns/topic-test with 0 partitions successfully
  ```

* If you want to create a partitioned topic, you need to set the number of partitions to a specific number.

  This example shows how to create a topic with 5 partitions in the `example-tenant/example-ns` namespace.

  **Input**

  ```bash theme={null}
  pulsarctl topics create example-tenant/example-ns/test-topic 5
  ```

  **Output**

  ```bash theme={null}
  Create topic persistent://example-tenant/example-ns/test-topic with 5 partitions successfully
  ```

### Manage a topic

This section describes how to manage topics using the pulsarctl CLI tool.

#### Get topic status

You can use the `pulsarctl topics get TOPIC_NAME` command to get information about a specific topic.

* This example shows how to list all topics available for the `example-tenant/example-ns` namespace.

  **Input**

  ```bash theme={null}
  pulsarctl topics list example-tenant/example-ns/
  ```

  **Output**

  ```bash theme={null}
  +--------------------------------------------------------------------------------+-----------------------+
  |                 TOPIC NAME                                                          | PARTITIONED ? |
  +--------------------------------------------------------------------------------+-----------------------+
  | persistent://example-tenant/example-ns/topic-test                  | N                         |
  | persistent://example-tenant/example-ns/test-topic                  | Y                         |
  | persistent://example-tenant/example-ns/test-topic-partition-0 | N                         |
  | persistent://example-tenant/example-ns/test-topic-partition-1 | N                         |
  | persistent://example-tenant/example-ns/test-topic-partition-2 | N                         |
  | persistent://example-tenant/example-ns/test-topic-partition-3 | N                         |
  | persistent://example-tenant/example-ns/test-topic-partition-4 | N                         |
  +--------------------------------------------------------------------------------+-----------------------+
  ```

* This example shows how to get detailed information about the `topic-test` topic.

  **Input**

  ```bash theme={null}
  pulsarctl topics get topic-test
  ```

  **Output**

  ```shell theme={null}
  {
  "partitions": 0
  }
  ```

#### Delete a topic

You can use the `pulsarctl topics delete TOPIC_NAME` command to delete a partitioned topic. To delete a non-partitioned topic, you need to set the `--non-partitioned` parameter.

* This example shows how to delete the `test-topic` partitioned topic.

  **Input**

  ```bash theme={null}
  pulsarctl topics delete topic-test
  ```

  **Output**

  ```bash theme={null}
  Delete topic persistent://example-tenant/example-ns/test-topic  successfully
  ```

* This example shows how to delete the `topic-test` non-partitioned topic.

  **Input**

  ```bash theme={null}
  pulsarctl topics delete --non-partitioned topic-test
  ```

  **Output**

  ```bash theme={null}
  Delete topic persistent://example-tenant/example-ns/topic-test  successfully
  ```

## Related topics

* Learn about all the available [Pulsar CLI Tools on StreamNative Cloud](/tools/cli/streamnative-cli-overview).
* Learn about all the available [Pulsar CLI Tools on StreamNative Platform](/tools/cli/streamnative-cli-overview).


# V3.0.10.9
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.10.9



## StreamNative Weekly Release Notes v3.0.10.9

#### General Changes

### Apache Pulsar

([#24960](https://github.com/apache/pulsar/pull/24960)) \[cleanup]\[broker]\[branch-3.0] Remove no-op configurations caused by cherry-picking

([#23551](https://github.com/apache/pulsar/pull/23551)) \[fix]\[txn] fix concurrent error cause txn stuck in TransactionBufferHandlerImpl#endTxn

([#24939](https://github.com/apache/pulsar/pull/24939)) \[fix]\[broker] Avoid recursive update in ConcurrentHashMap during policy cache cleanup

([#24762](https://github.com/apache/pulsar/pull/24762)) \[fix]\[admin] Set local policies overwrites "number of bundles" passed during namespace creation

([#24929](https://github.com/apache/pulsar/pull/24929)) \[fix]\[test] Stabilize testMsgDropStat by reliably triggering non-persistent publisher drop

([#24934](https://github.com/apache/pulsar/pull/24934)) \[fix]\[broker] Fix stack overflow caused by race condition when closing a connection

([#24898](https://github.com/apache/pulsar/pull/24898)) \[fix]\[broker] fix getMaxReadPosition in TransactionBufferDisable should return latest

([#24794](https://github.com/apache/pulsar/pull/24794)) \[fix]\[client] Fix thread leak in reloadLookUp method which is used by ServiceUrlProvider

([#24859](https://github.com/apache/pulsar/pull/24859)) \[fix]\[broker] Run ResourceGroup tasks only when tenants/namespaces registered

([#24915](https://github.com/apache/pulsar/pull/24915)) \[fix]\[broker] BacklogMessageAge is not reset when cursor mdPosition is on an open ledger

([#24917](https://github.com/apache/pulsar/pull/24917)) \[improve]\[ci] Move replication tests to new group Broker Group 5 in Pulsar CI

([#24911](https://github.com/apache/pulsar/pull/24911)) \[improve]\[misc] Upgrade Netty to 4.1.128.Final

([#24904](https://github.com/apache/pulsar/pull/24904)) \[fix]\[test] Fix flaky ReplicatorTest.testResumptionAfterBacklogRelaxed

([#24885](https://github.com/apache/pulsar/pull/24885)) \[fix]\[broker] Fix totalAvailablePermits not reduced when removing consumer from non-persistent dispatcher

([#24848](https://github.com/apache/pulsar/pull/24848)) \[improve]\[client]Add null check for Pulsar client clock configuration

([#24880](https://github.com/apache/pulsar/pull/24880)) \[fix]\[broker] Stop to retry to read entries if the replicator has terminated

([#24865](https://github.com/apache/pulsar/pull/24865)) \[fix]\[test] Fix flaky SubscriptionSeekTest.testSeekWillNotEncounteredFencedError by counting subscription is fenced only after seek

([#24861](https://github.com/apache/pulsar/pull/24861)) \[fix]\[test] Stabilize SequenceIdWithErrorTest by fencing after first publish to avoid empty-ledger deletion and send timeout

([#24852](https://github.com/apache/pulsar/pull/24852)) \[fix]\[ml] Fix `getNumberOfEntries` may point to deleted ledger

([#24841](https://github.com/apache/pulsar/pull/24841)) \[improve]\[ci] Upgrade GitHub Actions workflows to use ubuntu-24.04

([#24830](https://github.com/apache/pulsar/pull/24830)) \[fix]\[client] Fix getPendingQueueSize for PartitionedTopicProducerStatsRecorderImpl: avoid NPE and implement aggregation

([#24832](https://github.com/apache/pulsar/pull/24832)) \[fix] Fix mixed lookup/partition metadata requests causing reliability issues and incorrect responses

([#24822](https://github.com/apache/pulsar/pull/24822)) \[fix]\[client] Make auto partitions update work for old brokers without PIP-344

([#24812](https://github.com/apache/pulsar/pull/24812)) \[fix]\[build] Remove invalid profile in settings.xml that caused gpg signing to fail

([#24770](https://github.com/apache/pulsar/pull/24770)) \[fix]\[broker] Flaky-test: ExtensibleLoadManagerImplTest.testDisableBroker

([#24779](https://github.com/apache/pulsar/pull/24779)) Bump org.apache.zookeeper:zookeeper from 3.9.3 to 3.9.4

([#24772](https://github.com/apache/pulsar/pull/24772)) \[fix]\[misc] Fix compareTo contract violation for NamespaceBundleStats, TimeAverageMessageData and ResourceUnitRanking

([#24769](https://github.com/apache/pulsar/pull/24769)) \[fix]\[test] Flaky-test: BrokerServiceTest.testShutDownWithMaxConcurrentUnload

([#24767](https://github.com/apache/pulsar/pull/24767)) \[fix]\[ci] Fix CI for Java 25 including upgrade of Gradle Develocity Maven extension

([#24763](https://github.com/apache/pulsar/pull/24763)) \[improve]\[build] Upgrade Lombok to 1.18.42 to fully support JDK25

([#23634](https://github.com/apache/pulsar/pull/23634)) \[improve]\[broker] If there is a deadlock in the service, the probe should return a failure because the service may be unavailable

([#24738](https://github.com/apache/pulsar/pull/24738)) \[fix]\[broker] First entry will be skipped if opening NonDurableCursor while trimmed ledger is adding first entry.

([#24753](https://github.com/apache/pulsar/pull/24753)) \[fix]\[ml]Fix EOFException after enabled topics offloading

([#24741](https://github.com/apache/pulsar/pull/24741)) \[fix]\[broker] Prevent unexpected recycle failure in dispatcher's read callback

([#20522](https://github.com/apache/pulsar/pull/20522)) \[improve]\[broker] Choose random thread for consumerFlow in PersistentDispatcherSingleActiveConsumer

([#24758](https://github.com/apache/pulsar/pull/24758)) \[fix]\[broker]\[branch-3.0] Prevent NPE in ownedBundlesCountPerNamespace on first bundle load

([#24752](https://github.com/apache/pulsar/pull/24752)) \[fix]\[client] rollback TopicListWatcher retry behavior

([#24698](https://github.com/apache/pulsar/pull/24698)) \[fix]\[client]TopicListWatcher not closed when calling PatternMultiTopicsConsumerImpl.closeAsync() method

([#24634](https://github.com/apache/pulsar/pull/24634)) Dispatcher did unnecessary sort for recentlyJoinedConsumers and printed noisy error logs

([#24730](https://github.com/apache/pulsar/pull/24730)) \[fix]\[broker] Ensure KeyShared sticky mode consumer respects assigned ranges

([#24743](https://github.com/apache/pulsar/pull/24743)) \[fix]\[client] Fix receiver queue auto-scale without memory limit

([#24731](https://github.com/apache/pulsar/pull/24731)) \[fix]\[broker] Fix cannot shutdown broker gracefully by admin api

([#24654](https://github.com/apache/pulsar/pull/24654)) \[fix]\[io] Improve Kafka Connect source offset flushing logic

([#24725](https://github.com/apache/pulsar/pull/24725)) \[fix]\[client] Avoid recycling the same ConcurrentBitSetRecyclable among different threads

([#24719](https://github.com/apache/pulsar/pull/24719)) \[fix]\[broker] Fix memory leak when metrics are updated in a thread other than FastThreadLocalThread

([#24594](https://github.com/apache/pulsar/pull/24594)) \[improve]\[build] Disable javadoc build failure

([#24580](https://github.com/apache/pulsar/pull/24580)) \[fix]\[broker]Fix never recovered metadata store bad version issue if received a large response from ZK

([#23336](https://github.com/apache/pulsar/pull/23336)) \[fix]\[client] Fix ArrayIndexOutOfBoundsException when using SameAuthParamsLookupAutoClusterFailover

([#23977](https://github.com/apache/pulsar/pull/23977)) \[fix]\[broker] Invalid regex in PulsarLedgerManager causes zk data notification to be ignored

([#24518](https://github.com/apache/pulsar/pull/24518)) ([#24671](https://github.com/apache/pulsar/pull/24671)) \[fix]\[broker]\[branch-3.0] Fix wrong backlog age metrics when the mark delete position point to a deleted ledger

([#24663](https://github.com/apache/pulsar/pull/24663)) \[fix]\[client] Skip schema validation when sending messages to DLQ to avoid infinite loop when schema validation fails on an incoming message

([#24669](https://github.com/apache/pulsar/pull/24669)) \[improve]\[io] Support specifying Kinesis KPL native binary path with 1.0 version specific path

([#24668](https://github.com/apache/pulsar/pull/24668)) \[improve]\[build] Use org.apache.nifi:nifi-nar-maven-plugin:2.1.0 with skipDocGeneration=true

([#24661](https://github.com/apache/pulsar/pull/24661)) \[improve]\[io] Upgrade AWS SDK v1 & v2, Kinesis KPL and KPC versions

([#24639](https://github.com/apache/pulsar/pull/24639)) \[fix]\[broker] Fix race condition in MetadataStoreCacheLoader causing inconsistent availableBroker list caching

([#24666](https://github.com/apache/pulsar/pull/24666)) \[improve]\[build] Increase maven resolver's sync context timeout

([#24662](https://github.com/apache/pulsar/pull/24662)) \[fix]\[client] fix ArrayIndexOutOfBoundsException in SameAuthParamsLookupAutoClusterFailover

([#24649](https://github.com/apache/pulsar/pull/24649)) \[fix]\[offload] Exclude unnecessary dependencies from tiered storage provider / offloader nar files

([#24643](https://github.com/apache/pulsar/pull/24643)) \[fix]\[broker] Add double-check for non-durable cursor creation

([#24633](https://github.com/apache/pulsar/pull/24633)) \[fix]\[test]fix flaky ZeroQueueSizeTest.testZeroQueueGetExceptionWhenReceiveBatchMessage

([#24632](https://github.com/apache/pulsar/pull/24632)) \[fix]\[test] Fix ConcurrentModificationException in Ipv4Proxy

([#24630](https://github.com/apache/pulsar/pull/24630)) \[fix]\[test]fix flaky ZeroQueueSizeTest.testZeroQueueGetExceptionWhenReceiveBatchMessage

([#24626](https://github.com/apache/pulsar/pull/24626)) \[fix]\[proxy] Fix TooLongFrameException with Pulsar Proxy

([#24621](https://github.com/apache/pulsar/pull/24621)) \[fix]\[broker] Fix duplicate watcher registration after SessionReestablished

([#24610](https://github.com/apache/pulsar/pull/24610)) \[fix]\[client]Prevent ZeroQueueConsumer from receiving batch messages when using MessagePayloadProcessor

([#24604](https://github.com/apache/pulsar/pull/24604)) \[improve]\[io] Add dependency file name information to error message when .nar file validation fails with ZipException

([#21361](https://github.com/apache/pulsar/pull/21361)) \[improve]\[broker] Optimize and clean up aggregation of topic stats

([#24601](https://github.com/apache/pulsar/pull/24601)) \[improve]\[doc] Improve the JavaDocs of sendAsync to avoid improper use

([#24599](https://github.com/apache/pulsar/pull/24599)) \[fix]\[client] Retry for unknown exceptions when creating a producer or consumer

([#24450](https://github.com/apache/pulsar/pull/24450)) \[fix]\[broker] Fix REST API to produce messages to single-partitioned topics

([#24595](https://github.com/apache/pulsar/pull/24595)) \[fix]\[ci] Fix code coverage metrics in Pulsar CI

([#24582](https://github.com/apache/pulsar/pull/24582)) \[improve]\[client] Support load RSA PKCS#8 private key

([#24535](https://github.com/apache/pulsar/pull/24535)) \[improve]\[test] Add test for dead letter topic with max unacked messages blocking

([#24532](https://github.com/apache/pulsar/pull/24532)) \[fix]\[misc] Upgrade dependencies to fix critical security vulnerabilities

([#24514](https://github.com/apache/pulsar/pull/24514)) \[improve]\[build] Bump org.apache.commons:commons-lang3 from 3.17.0 to 3.18.0

([#24586](https://github.com/apache/pulsar/pull/24586)) \[improve]\[test] Refactor the way way pulsar-io-debezium-oracle nar file is patched when building the test image

([#24590](https://github.com/apache/pulsar/pull/24590)) \[fix]\[broker] Fix flaky testReplicatorsInflightTaskListIsEmptyAfterReplicationFinished

([#24542)](https://github.com/apache/pulsar/pull/24542))) Revert "\[fix]\[broker]Fix thread safety issues in BucketDelayedDeliveryTracker with StampedLock optimistic reads

([#24554](https://github.com/apache/pulsar/pull/24554)) ([#24571](https://github.com/apache/pulsar/pull/24571)) \[fix]\[client]\[branch-4.0] Partitioned topics are unexpectedly created by client after deletion

([#24576](https://github.com/apache/pulsar/pull/24576)) \[fix]\[test] fix flaky GrowableArrayBlockingQueueTest.testPollBlockingThreadsTermination

([#24569](https://github.com/apache/pulsar/pull/24569)) \[fix]\[broker] Fix ManagedCursor state management race conditions and lifecycle issues

([#24550](https://github.com/apache/pulsar/pull/24550)) \[improve]\[client] Terminate consumer.receive() when consumer is closed

([#24473](https://github.com/apache/pulsar/pull/24473)) \[improve]\[build] replace org.apache.commons.lang to org.apache.commons.lang3

([#24560](https://github.com/apache/pulsar/pull/24560)) \[fix]\[broker] Fix maxTopicsPerNamespace might report a false failure

([#24505](https://github.com/apache/pulsar/pull/24505)) \[fix]\[test]fix flaky test BrokerServiceAutoTopicCreationTest.testDynamicConfigurationTopicAutoCreationPartitioned

([#24472](https://github.com/apache/pulsar/pull/24472)) \[fix] Prevent IllegalStateException: Field 'message' is not set

([#24542](https://github.com/apache/pulsar/pull/24542)) \[fix]\[broker]Fix thread safety issues in BucketDelayedDeliveryTracker with StampedLock optimistic reads

([#24551](https://github.com/apache/pulsar/pull/24551)) \[fix]\[broker] Fix Broker OOM due to too many waiting cursors and reuse a recycled OpReadEntry incorrectly

([#24511](https://github.com/apache/pulsar/pull/24511)) \[fix]\[broker] Fix deduplication replay might never complete for exceptions

([#24522](https://github.com/apache/pulsar/pull/24522)) \[fix]\[ml] Fix the possibility of message loss or disorder when ML PayloadProcessor processing fails

([#24451](https://github.com/apache/pulsar/pull/24451)) \[fix]\[test]\[branch-3.0] Correct topic policy loading logic and improve related tests

([#24552](https://github.com/apache/pulsar/pull/24552)) \[improve]\[test] Remove EntryCacheCreator from ManagedLedgerFactoryImpl

([#24544](https://github.com/apache/pulsar/pull/24544)) \[improve] Upgrade pulsar-client-python to 3.8.0 in Docker image

([#24516](https://github.com/apache/pulsar/pull/24516)) \[fix]\[broker] Fix exclusive producer creation when last shared producer closes

([#24506](https://github.com/apache/pulsar/pull/24506)) \[fix]\[broker] Fix duplicate increment of ADD\_OP\_COUNT\_UPDATER in OpAddEntry

([#24543](https://github.com/apache/pulsar/pull/24543)) \[fix]\[broker] Fix matching of topicsPattern for topic names which contain non-ascii characters

([#24539](https://github.com/apache/pulsar/pull/24539)) \[fix]\[client] Close orphan producer or consumer when the creation is interrupted

([#24517](https://github.com/apache/pulsar/pull/24517)) \[fix]\[client] Fix ClientCnx handleSendError NPE

([#24515](https://github.com/apache/pulsar/pull/24515)) \[fix]\[ml] Fix asyncReadEntries might never complete if empty entries are read from BK

([#24525](https://github.com/apache/pulsar/pull/24525)) \[improve]\[misc] Optimize topic list hashing so that potentially large String allocation is avoided

([#24528](https://github.com/apache/pulsar/pull/24528)) \[fix]\[client] Fix issue in auto releasing of idle connection with topics pattern consumer

([#24529](https://github.com/apache/pulsar/pull/24529)) \[fix]\[proxy] Fix default value of connectionMaxIdleSeconds in Pulsar Proxy

([#24476](https://github.com/apache/pulsar/pull/24476)) \[fix]\[client] NPE in MultiTopicsConsumerImpl.negativeAcknowledge

([#24465](https://github.com/apache/pulsar/pull/24465)) \[fix]\[proxy] Fix proxy OOM by replacing TopicName with a simple conversion method

([#22495](https://github.com/apache/pulsar/pull/22495)) \[fix]\[test] Move ExtensibleLoadManagerImplTest to flaky tests

([#21642](https://github.com/apache/pulsar/pull/21642)) \[fix]\[test] Fix flaky test SimpleProducerConsumerStatTest#testPartitionTopicStats

([#24453](https://github.com/apache/pulsar/pull/24453)) \[fix]\[broker] replication does not work due to the mixed and repetitive sending of user messages and replication markers

([#24424](https://github.com/apache/pulsar/pull/24424)) \[fix]\[broker] Fix the non-persistenttopic's replicator always get error "Producer send queue is full" if set a small value of the config replicationProducerQueueSize

([#24189](https://github.com/apache/pulsar/pull/24189)) \[fix]\[broker]excessive replication speed leads to error: Producer send queue is full

([#22674](https://github.com/apache/pulsar/pull/22674)) \[Fix]\[broker] Limit replication rate based on bytes

([#20931](https://github.com/apache/pulsar/pull/20931)) \[fix]\[broker] Fix ack hole in cursor for geo-replication

([#24443](https://github.com/apache/pulsar/pull/24443)) \[fix]\[txn] Fix negative unacknowledged messages in transactions by ensuring that the batch size is added into CommandAck

([#24421](https://github.com/apache/pulsar/pull/24421)) \[fix]\[build] Add missing `<name>` to submodules

([#24441](https://github.com/apache/pulsar/pull/24441)) \[fix]\[ml] Enhance OpFindNewest to support skip non-recoverable data

([#24459](https://github.com/apache/pulsar/pull/24459)) \[improve]\[broker] change to warn log level for ack validation error

([#24434](https://github.com/apache/pulsar/pull/24434)) \[improve]\[broker] Improve the log when namespace bundle is not available

([#24432](https://github.com/apache/pulsar/pull/24432)) \[fix]\[ml]Still got BK ledger, even though it has been deleted after offloaded

([#21467](https://github.com/apache/pulsar/pull/21467)) \[fix]\[test] Cleanup resources if starting PulsarService fails in PulsarTestContext

([#24351](https://github.com/apache/pulsar/pull/24351)) \[improve]\[broker] Deny removing local cluster from topic level replicated cluster policy

([#24419](https://github.com/apache/pulsar/pull/24419)) \[fix]\[broker] Once the cluster is configured incorrectly, the broker maintains the incorrect cluster configuration even if you removed it

([#24404](https://github.com/apache/pulsar/pull/24404)) \[fix]\[client] Prevent NPE when seeking with null topic in TopicMessageId

([#24405](https://github.com/apache/pulsar/pull/24405)) \[fix]\[ml]Received more than once callback when calling cursor.delete

([#24406](https://github.com/apache/pulsar/pull/24406)) \[fix]\[ml] Cursor ignores the position that has an empty ack-set if disabled deletionAtBatchIndexLevelEnabled

([#24401](https://github.com/apache/pulsar/pull/24401)) \[fix]\[txn] Fix deadlock when loading transaction buffer snapshot

([#24402](https://github.com/apache/pulsar/pull/24402)) \[fix]\[client] Fix some potential resource leak

([#20629](https://github.com/apache/pulsar/pull/20629)) \[improve]\[test] Fix flaky test SimpleProducerConsumerStatTest#testMsgRateExpired

([#21629](https://github.com/apache/pulsar/pull/21629)) \[fix]\[build] Fix potential insufficient protostuff-related configs

([#23594](https://github.com/apache/pulsar/pull/23594)) \[fix] \[broker] No longer allow creating subscription that contains slash

([#24366](https://github.com/apache/pulsar/pull/24366)) \[fix]\[broker]Fix deadlock when compaction and topic deletion execute concurrently

([#24350](https://github.com/apache/pulsar/pull/24350)) \[fix]\[broker] Fix issue that topic policies was deleted after a sub topic deleted, even if the partitioned topic still exists

([#24384](https://github.com/apache/pulsar/pull/24384)) \[fix]\[ml]Revert a behavior change of releasing idle offloaded ledger handle: only release idle BlobStoreBackedReadHandle

([#24397](https://github.com/apache/pulsar/pull/24397)) \[improve]\[misc] Upgrade Netty to 4.1.122.Final and tcnative to 2.0.72.Final

([#24391](https://github.com/apache/pulsar/pull/24391)) \[improve]\[broker] Add managedCursor/LedgerInfoCompressionType settings to broker.conf

([#24392](https://github.com/apache/pulsar/pull/24392)) \[improve]\[broker] Make maxBatchDeletedIndexToPersist configurable and document other related configs

([#24386](https://github.com/apache/pulsar/pull/24386)) \[improve]\[broker] Added synchronized for sendMessages in Non-Persistent message dispatchers

([#24381](https://github.com/apache/pulsar/pull/24381)) \[improve]\[ml]Release idle offloaded read handle only the ref count is 0

([#19783](https://github.com/apache/pulsar/pull/19783)) \[improve]\[offloaders] Automatically evict Offloaded Ledgers from memory

([#24360](https://github.com/apache/pulsar/pull/24360)) \[fix]\[broker] expose consumer name for partitioned topic stats

([#24359](https://github.com/apache/pulsar/pull/24359)) \[improve]\[broker]Improve the log when encountered in-flight read limitation

([#24354](https://github.com/apache/pulsar/pull/24354)) \[fix]\[io] Acknowledge RabbitMQ message after processing the message successfully

([#24352](https://github.com/apache/pulsar/pull/24352)) \[fix]\[broker] Ignore metadata changes when broker is not in the Started state

([#24190](https://github.com/apache/pulsar/pull/24190)) \[fix]\[broker] Resolve the issue of frequent updates in message expiration deletion rate

([#24338](https://github.com/apache/pulsar/pull/24338)) \[fix]\[ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent issue

([#24331](https://github.com/apache/pulsar/pull/24331)) \[fix]\[offload] Complete the future outside of the reading loop in BlobStoreBackedReadHandleImplV2.readAsync

([#24324](https://github.com/apache/pulsar/pull/24324)) \[fix]\[test] Fix flaky AutoScaledReceiverQueueSizeTest.testNegativeClientMemory

([#24316](https://github.com/apache/pulsar/pull/24316)) \[fix]\[io] Fix kinesis avro bytes handling

([#24344](https://github.com/apache/pulsar/pull/24344)) \[improve]\[ml] Offload ledgers without check ledger length

([#24286](https://github.com/apache/pulsar/pull/24286)) \[fix]\[broker]Non-global topic policies and global topic policies overwrite each other

([#24279](https://github.com/apache/pulsar/pull/24279)) \[fix]\[broker]Global topic policies do not affect after unloading topic and persistence global topic policies never affect

([#24349](https://github.com/apache/pulsar/pull/24349)) \[fix]\[io]\[branch-3.0] Backport Kinesis Sink custom native executable support #23762

([#24317](https://github.com/apache/pulsar/pull/24317)) \[fix]\[io]\[branch-3.0]Pulsar-SQL: Fix classcast ex when decode decimal value

([#24313](https://github.com/apache/pulsar/pull/24313)) \[fix]\[broker] Fix potential deadlock when creating partitioned topic

([#24293](https://github.com/apache/pulsar/pull/24293)) \[fix]\[broker] fix wrong method name checkTopicExists.

([#24307](https://github.com/apache/pulsar/pull/24307)) \[fix]\[build] Ensure that buildtools is Java 8 compatible and fix remaining compatibility issue

([#24304](https://github.com/apache/pulsar/pull/24304)) \[fix]\[test] Simplify BetweenTestClassesListenerAdapter and fix issue with BeforeTest/AfterTest annotations

([#24289](https://github.com/apache/pulsar/pull/24289)) \[improve]\[io] Add configuration parameter for disabling aggregation for Kinesis Producers

([#24302](https://github.com/apache/pulsar/pull/24302)) \[improve] Upgrade pulsar-client-python to 3.7.0 in Docker image

([#24299](https://github.com/apache/pulsar/pull/24299)) \[fix]\[test] Fix more Netty ByteBuf leaks in tests

([#24297](https://github.com/apache/pulsar/pull/24297)) \[fix]\[io] Fix SyntaxWarning in Pulsar Python functions

([#24282](https://github.com/apache/pulsar/pull/24282)) \[fix]\[client] Fix producer publishing getting stuck after message with incompatible schema is discarded

([#24283](https://github.com/apache/pulsar/pull/24283)) \[cleanup]\[test] Remove unused parameter from deleteNamespaceWithRetry method in MockedPulsarServiceBaseTest

([#24263](https://github.com/apache/pulsar/pull/24263)) \[improve]\[build] Upgrade zstd version from 1.5.2-3 to 1.5.7-3

([#24281](https://github.com/apache/pulsar/pull/24281)) \[fix]\[test] Fix multiple ByteBuf leaks in tests

([#24275](https://github.com/apache/pulsar/pull/24275)) \[fix]\[broker] Fix HashedWheelTimer leak in PulsarService by stopping it in shutdown

([#24274](https://github.com/apache/pulsar/pull/24274)) \[fix]\[misc] Fix ByteBuf leak in SchemaUtils

([#24254](https://github.com/apache/pulsar/pull/24254)) \[fix]\[broker]Fix incorrect priority between topic policies and global topic policies

([#24266](https://github.com/apache/pulsar/pull/24266)) \[improve]\[ci] Disable detailed console logging for integration tests in CI

([#24261](https://github.com/apache/pulsar/pull/24261)) \[fix]\[test] Fix flaky ManagedCursorTest.testLastActiveAfterResetCursor and disable failing SchemaTest

([#24244](https://github.com/apache/pulsar/pull/24244)) \[fix]\[test] Fix flaky ManagedCursorTest.testSkipEntriesWithIndividualDeletedMessages

([#24248](https://github.com/apache/pulsar/pull/24248)) \[improve]\[io]\[kca] support fully-qualified topic names in source records

([#24260](https://github.com/apache/pulsar/pull/24260)) \[improve]\[build] Upgrade Gradle Develocity Maven Extension dependencies

([#24258](https://github.com/apache/pulsar/pull/24258)) \[fix]\[test] Fix TestNG BetweenTestClassesListenerAdapter listener

([#24257](https://github.com/apache/pulsar/pull/24257)) \[fix]\[broker] Unregister non-static metrics collectors registered in Prometheus default registry

([#24178](https://github.com/apache/pulsar/pull/24178)) \[fix]\[broker]fix memory leak, messages lost, incorrect replication state if using multiple schema versions(auto\_produce)

([#24219](https://github.com/apache/pulsar/pull/24219)) \[improve]\[broker]Improve the feature "Optimize subscription seek (cursor reset) by timestamp": search less entries

([#23919](https://github.com/apache/pulsar/pull/23919)) \[fix]\[broker] Fix seeking by timestamp can be reset the cursor position to earliest

([#22792](https://github.com/apache/pulsar/pull/22792)) \[improve]\[broker] Optimize subscription seek (cursor reset) by timestamp

([#24243](https://github.com/apache/pulsar/pull/24243)) \[improve]\[build] Upgrade SpotBugs to 4.9.x

([#24240](https://github.com/apache/pulsar/pull/24240)) \[improve]\[build] Upgrade to jacoco 0.8.13

([#24237](https://github.com/apache/pulsar/pull/24237)) \[improve]\[build] Upgrade Lombok to 1.18.38 to support JDK 24

([#24221](https://github.com/apache/pulsar/pull/24221)) \[improve]\[io] support kafka connect transforms and predicates

([#24230](https://github.com/apache/pulsar/pull/24230)) \[improve]\[client]Improve transaction log when a TXN command timeout

([#24223](https://github.com/apache/pulsar/pull/24223)) \[fix]\[broker] Orphan schema after disabled a cluster for a namespace

([#24228](https://github.com/apache/pulsar/pull/24228)) \[fix]\[broker] Fix ByteBuf memory leak in REST API for publishing messages

([#24184](https://github.com/apache/pulsar/pull/24184)) \[fix]\[client] Fix incorrect producer.getPendingQueueSize due to incomplete queue implementation

([#24214](https://github.com/apache/pulsar/pull/24214)) \[improve] Upgrade Netty to 4.1.121.Final

([#24212](https://github.com/apache/pulsar/pull/24212)) \[fix]\[test] Fix flaky BatchMessageWithBatchIndexLevelTest.testBatchMessageAck

([#24218](https://github.com/apache/pulsar/pull/24218)) \[fix]\[test] Fix multiple resource leaks in tests

([#24187](https://github.com/apache/pulsar/pull/24187)) \[improve]\[client] validate ClientConfigurationData earlier to avoid resource leaks

([#24216](https://github.com/apache/pulsar/pull/24216)) \[fix]\[broker] Fix HealthChecker deadlock in shutdown

([#24209](https://github.com/apache/pulsar/pull/24209)) \[fix]\[broker] Fix tenant creation and update with null value

([#24192](https://github.com/apache/pulsar/pull/24192)) \[fix]\[admin] Backlog quota's policy is null which causes a NPE

([#24210](https://github.com/apache/pulsar/pull/24210)) \[fix]\[broker] Fix broker shutdown delay by resolving hanging health checks

([#24207](https://github.com/apache/pulsar/pull/24207)) \[fix]\[broker] Fix compaction service log's wrong condition

([#24204](https://github.com/apache/pulsar/pull/24204)) \[fix]\[test] Fix resource leaks in ProxyTest and fix invalid tests

([#24201](https://github.com/apache/pulsar/pull/24201)) \[improve]\[io] Upgrade Kafka client and compatible Confluent platform version

([#24118)](https://github.com/apache/pulsar/pull/24118))) Revert "\[fix]\[broker] Add topic consistency check

([#24154)](https://github.com/apache/pulsar/pull/24154))) Revert "\[fix]\[broker] Directly query single topic existence when the topic is partitioned

([#24032](https://github.com/apache/pulsar/pull/24032)) \[fix]\[broker] Fix missing validation when setting retention policy on topic level

([#24098](https://github.com/apache/pulsar/pull/24098)) \[fix]\[ml] Skip deleting cursor if it was already deleted before calling unsubscribe

([#24181](https://github.com/apache/pulsar/pull/24181)) \[fix]\[proxy] Fix incorrect client error when calling get topic metadata

([#24158](https://github.com/apache/pulsar/pull/24158)) \[fix]\[proxy] Propagate client connection feature flags through Pulsar Proxy to Broker

([#24103](https://github.com/apache/pulsar/pull/24103)) \[fix]\[schema] Reject unsupported Avro schema types during schema registration

([#24091](https://github.com/apache/pulsar/pull/24091)) \[fix]\[broker] Fix some problems in calculate totalAvailableBookies in method getExcludedBookiesWithIsolationGroups when some bookies belongs to multiple isolation groups.

([#21320](https://github.com/apache/pulsar/pull/21320)) \[fix]\[bk] Fix the var name for IsolationGroups

([#24171](https://github.com/apache/pulsar/pull/24171)) \[improve]\[test] Use configured session timeout for MockZooKeeper and TestZKServer in PulsarTestContext

([#24172](https://github.com/apache/pulsar/pull/24172)) \[fix]\[test] Improve reliability of IncrementPartitionsTest

([#24170](https://github.com/apache/pulsar/pull/24170)) \[fix]\[test]flaky-test:ManagedLedgerInterceptorImplTest.testManagedLedgerPayloadInputProcessorFailure

([#23980](https://github.com/apache/pulsar/pull/23980)) \[fix]\[broker] Consumer stuck when delete subscription \_\_compaction failed

([#24167](https://github.com/apache/pulsar/pull/24167)) \[fix]\[ml] Fix ML thread blocking issue in internalGetPartitionedStats API

([#24166](https://github.com/apache/pulsar/pull/24166)) \[fix]\[test] Fix invalid test CompactionTest.testDeleteCompactedLedgerWithSlowAck

([#24150](https://github.com/apache/pulsar/pull/24150)) \[fix]\[broker] The feature brokerDeleteInactivePartitionedTopicMetadataEnabled leaves orphan topic policies and topic schemas

([#24154](https://github.com/apache/pulsar/pull/24154)) \[fix]\[broker] Directly query single topic existence when the topic is partitioned

([#24118](https://github.com/apache/pulsar/pull/24118)) \[fix]\[broker] Add topic consistency check

([#24056](https://github.com/apache/pulsar/pull/24056)) \[fix]\[test] Update partitioned topic subscription assertions in IncrementPartitionsTest

([#24033](https://github.com/apache/pulsar/pull/24033)) \[cleanup]\[misc] Add override annotation

([#24161](https://github.com/apache/pulsar/pull/24161)) \[fix]\[test] Fix flaky BrokerServiceChaosTest.testFetchPartitionedTopicMetadataWithCacheRefresh

([#24162](https://github.com/apache/pulsar/pull/24162)) \[fix]\[test] Fix flaky BrokerServiceChaosTest

([#24097](https://github.com/apache/pulsar/pull/24097)) \[fix] \[broker] topics infinitely failed to delete after remove cluster from replicated clusters modifying when using partitioned system topic

([#22261](https://github.com/apache/pulsar/pull/22261)) \[fix] Bump google.golang.org/protobuf from 1.32.0 to 1.33.0 in /pulsar-function-go

([#24132](https://github.com/apache/pulsar/pull/24132)) \[fix]\[io] Fix KinesisSink json flattening for AVRO's SchemaType.BYTES

([#20984](https://github.com/apache/pulsar/pull/20984)) \[fix]\[broker] Fix get outdated compactedTopicContext after compactionHorizon has been updated

([#20697](https://github.com/apache/pulsar/pull/20697)) \[improve]\[broker] Improve CompactedTopicImpl lock

([#24131](https://github.com/apache/pulsar/pull/24131)) \[fix]\[ml] Return 1 when bytes size is 0 or negative for entry count estimation

([#24128](https://github.com/apache/pulsar/pull/24128)) \[improve]\[io] Enhance Kafka connector logging with focused bootstrap server information

([#24125](https://github.com/apache/pulsar/pull/24125)) \[fix]\[ml] Don't estimate number of entries when ledgers are empty, return 1 instead

([#24123](https://github.com/apache/pulsar/pull/24123)) \[improve]\[client] Prevent NullPointException when closing ClientCredentialsFlow

([#24124](https://github.com/apache/pulsar/pull/24124)) \[improve]\[io] Remove sleep when sourceTask.poll of kafka return null

([#24116](https://github.com/apache/pulsar/pull/24116)) \[improve]\[broker] Change topic exists log to warn

([#24104](https://github.com/apache/pulsar/pull/24104)) \[fix]\[client] Pattern subscription regression when broker-side evaluation is disabled

([#24100](https://github.com/apache/pulsar/pull/24100)) \[fix]\[client] Fix consumer leak when thread is interrupted before subscribe completes

([#24089](https://github.com/apache/pulsar/pull/24089)) \[fix]\[ml] Fix issues in estimateEntryCountBySize

([#24073](https://github.com/apache/pulsar/pull/24073)) \[improve]\[broker] Optimize message expiration rate repeated update issues

([#24087](https://github.com/apache/pulsar/pull/24087)) \[fix]\[broker] Avoid IllegalStateException when marker\_type field is not set in publishing

([#24083](https://github.com/apache/pulsar/pull/24083)) \[fix]\[ci] Bump dependency-check to 12.1.0 to fix OWASP Dependency Check job

([#24082](https://github.com/apache/pulsar/pull/24082)) \[clean]\[client] Clean code for the construction of retry/dead letter topic name

([#24079](https://github.com/apache/pulsar/pull/24079)) \[fix]\[broker] Fix NPE while publishing Metadata-Event with not init producer

([#24080](https://github.com/apache/pulsar/pull/24080)) \[fix]\[broker] Fix Metadata event synchronizer should not fail with bad version

([#24081](https://github.com/apache/pulsar/pull/24081)) \[fix]\[broker] Fix Metadata Event Synchronizer producer creation retry so that the producer gets created eventually

([#24048](https://github.com/apache/pulsar/pull/24048)) \[fix]\[broker] Fix UnsupportedOperationException while setting subscription level dispatch rate policy

([#24054](https://github.com/apache/pulsar/pull/24054)) \[fix]\[ml] Corrected pulsar\_storage\_size metric to not multiply offloaded storage by the write quorum

([#24067](https://github.com/apache/pulsar/pull/24067)) \[fix]\[broker] http metric endpoint get compaction latency stats always be 0

([#24064](https://github.com/apache/pulsar/pull/24064)) \[improve]\[broker] Optimize ThresholdShedder with improved boundary checks and parameter reuse

([#24055](https://github.com/apache/pulsar/pull/24055)) \[fix] Avoid negative estimated entry count

([#24060](https://github.com/apache/pulsar/pull/24060)) \[improve]\[monitor] Add version=0.0.4 to /metrics content type for Prometheus 3.x compatibility

([#24059](https://github.com/apache/pulsar/pull/24059)) \[fix]\[client] Copy eventTime to retry letter topic and DLQ messages

([#24061](https://github.com/apache/pulsar/pull/24061)) \[fix]\[client] Fix building broken batched message when publishing

([#24063](https://github.com/apache/pulsar/pull/24063)) \[fix]\[broker]Fix failed consumption after loaded up a terminated topic

([#24072](https://github.com/apache/pulsar/pull/24072)) \[fix]\[broker] Pattern subscription doesn't work when the pattern excludes the topic domain.

([#24049](https://github.com/apache/pulsar/pull/24049)) \[improve] Upgrade Netty to 4.1.119.Final

([#23975](https://github.com/apache/pulsar/pull/23975)) \[fix]\[broker] Add expire check for replicator

([#24023](https://github.com/apache/pulsar/pull/24023)) \[fix]\[doc] fix doc related to chunk message feature.

([#23962](https://github.com/apache/pulsar/pull/23962)) \[improve]\[ml] Use lock-free queue in InflightReadsLimiter since there's no concurrent access

([#23978](https://github.com/apache/pulsar/pull/23978)) \[improve]\[cli] Support additional msg metadata for V1 topic on peek message cmd

([#24014](https://github.com/apache/pulsar/pull/24014)) \[fix]\[broker] Fix BucketDelayedDeliveryTracker thread safety

([#24019](https://github.com/apache/pulsar/pull/24019)) \[fix]\[test]Fix flaky test V1\_ProducerConsumerTest.testConcurrentConsumerReceiveWhileReconnect

([#24011](https://github.com/apache/pulsar/pull/24011)) \[fix]\[test] Fix flaky test OneWayReplicatorUsingGlobalZKTest.testConfigReplicationStartAt

([#23931](https://github.com/apache/pulsar/pull/23931)) \[improve] \[broker] Make the estimated entry size more accurate

([#24004](https://github.com/apache/pulsar/pull/24004)) \[improve]\[ci] Upgrade Gradle Develocity Maven Extension to 1.23.1

([#23697](https://github.com/apache/pulsar/pull/23697)) \[fix]\[broker] Geo Replication lost messages or frequently fails due to Deduplication is not appropriate for Geo-Replication

([#24006](https://github.com/apache/pulsar/pull/24006)) \[fix]\[broker] fix broker identifying incorrect stuck topic

([#23286](https://github.com/apache/pulsar/pull/23286)) \[improve]\[broker] Fix non-persistent system topic schema compatibility

([#23881](https://github.com/apache/pulsar/pull/23881)) \[improve]\[fn] Set default tenant and namespace for ListFunctions cmd

([#23730](https://github.com/apache/pulsar/pull/23730)) \[fix]\[admin] Verify is policies read only before revoke permissions on topic

([#24003](https://github.com/apache/pulsar/pull/24003)) \[improve]\[test] Upgrade Testcontainers to 1.20.4 and docker-java to 3.4.0

### StreamNative Pulsar Plugins

Fix detector deadlock when subDetector panics or fails

### pulsarctl

e7d5e82 Use snstage docker image

Fix jwt cve

fix code check

feat: Subscription get message by id json output

Update subscription get message by id typo lederId to ledgerId

fix: upgrade golang version to fix CVE

Setup go version to 1.22 fix cve

update pulsar-client-go to master latest commit

2af1258 fix ci

Bump the pulsar-client-go to the master version

6f25051 Fix TestDeleteNonExistPartitionedTopic

Fix json marshal error for Secrets and UserConfigs when creating/updating functions

Support create token with headers

Upgrade the dependency version to fix vulnerabilities

Add trivy scan workflow to avoid vulnerabilities

\[fix] Upgrade go version to 1.21 to fix CVE-2023-24538

fix source test typo

fix source test

Auth SN docker hub

Support no auth context

fix token

Add docker hub login

Auth SN docker hub

5cb0593 Disable bk unit test and fix it later ---

Add method to mark bookie readonly

Build arm64 linux executable binary artifact

Update jose2go to fix GHSA-mhpq-9638-x6pw

Update golang.org/x/net

Replace apache pulsar client go repo on 3.0 branch

Fixed remove auth plugin suffix

Removed error char

Bump pulsar version to 3.0.0.1

fix: Fix TestUpdateTopicNotExist and TestUpdateNonPartitionedTopic

### Function Mesh Worker Service

Use new way to build sn-operator image

Use sn-operator to deploy pulsar cluster in CI

fix agent cannot update some fields error

Fix ci failure

fix build and license header

Reject request when agent name is too long or sessionId is not valid


# StreamNative Weekly Release Notes v3.1.0.1
Source: https://docs.streamnative.io/release-notes/pulsar/v3.1/component-changelogs-v3.1.0.1



# StreamNative Weekly Release Notes v3.1.0.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.1.0.1](https://github.com/streamnative/pulsar/releases/tag/v3.1.0.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.1.0.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.1.0.1/images/sha256-0432537447b972f3e6276a101ce501a204896a2c14a7a9d64689d0fc22455167)

## General Changes


# StreamNative Weekly Release Notes v3.1.0.2
Source: https://docs.streamnative.io/release-notes/pulsar/v3.1/component-changelogs-v3.1.0.2



# StreamNative Weekly Release Notes v3.1.0.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.1.0.2](https://github.com/streamnative/pulsar/releases/tag/v3.1.0.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.1.0.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.1.0.2/images/sha256-eac91671afe80a574cfd36991026018a2dd3b3a2f342d28ff25b4f0ed2401695)

## General Changes

### Apache Pulsar

\[fix]\[broker] Make sure all inflight writes have finished before completion of compaction

\[fix]\[broker] Fix can't stop phase-two of compaction even though messageId read reaches lastReadId

\[fix]\[broker] Use MessageDigest.isEqual when comparing digests

\[improve]\[proxy] Support disabling metrics endpoint

\[fix]\[broker] fix ModularLoadManagerImpl always delete active bundle-data. sec ver.

\[fix]\[client] Fix RawReader hasMessageAvailable returns true when no messages

\[fix]\[meta] Fix deadlock in AutoRecovery.

\[fix]\[broker] Fix incorrect unack msk count when dup ack a message

\[fix]\[broker] Fix compaction subscription delete by inactive subscription check.

\[fix] \[admin] Fix get topic stats fail if a subscription catch up concurrently

### SN KoP

\[proxy] Support authentication and authorization

\[proxy] Support TLS encryption between client and proxy

\[improve] Use CompactedTopicUtils.asyncReadCompactedEntries instead of readCompactedEntries

\[fix]\[compaction] Fix can't fetch compacted topic offset by timestamp if ledger is trimmed

ea09d8b2 Bump Pulsar to 3.1.0.1

\[fix] Memory leak of the map in ThreadLocalAccessor

\[improve] Fix the GC issue by removing unused delayed produce

\[proxy] Do not share the same connection among different clients

\[proxy] Support receiving messages from a single broker

\[fix]\[admin] Describe topic cleanup policy config should return correct config

Try fixing flaky testListOffsetForEmptyRolloverLedger and add more logs

\[build] Remove the build process for the binary tarball

\[transaction] Check producer epoch before publish message

\[proxy] Fix message disordering when sending messages

\[build] Fix oauth client release

\[fix]\[transaction] Use pulsar format when write marker to \_\_consumer\_offsets topic

Fix topic lookup failed exceptionally with ServiceUnitNotReady

\[fix]\[transaction] Contains UUID when auto topic creation

\[fix]\[transaction] Handle some failures in transaction

Update Kafka wire protocol to 3.4.0 and implement KIP-699 and KIP-709

Add Time based metric exipration.

### pulsarctl

Migrate pulsar-admin-go to apache pulsar client go

### StreamNative Pulsar Plugins

Create pool netty buffer instead of nio buffer in rebatch kafka format

Improve KafkaCompactionParser according RecordBatch javadoc

### Cloud Pulsar Plugins

Fixed pulsar api key audience list issuer

### Function Mesh Worker Service

Bump fm to 0.16.0

Fix error that `--retain-[key-]ordering` not working

Revert "Remove enableStateStore from CustomRuntimeOptions

Fix producerConfig cannot be updated error

Expose some internal errors to users

Fix typos and broken links

Fix error that --batch-builder doesn't work for functions

Add FM Worker service content

### Aws EventBridge Connector

Fix docs format for Note.

Fix wrong words in docs


# StreamNative Weekly Release Notes v3.1.0.3
Source: https://docs.streamnative.io/release-notes/pulsar/v3.1/component-changelogs-v3.1.0.3



# StreamNative Weekly Release Notes v3.1.0.3

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.1.0.3](https://github.com/streamnative/pulsar/releases/tag/v3.1.0.3)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.1.0.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.1.0.3/images/sha256-b6a1e47d6b560242899447885b09dc13412be84cde04fdfd5b6b863217491319)

## General Changes

### Apache Pulsar

\[fix]\[proxy] Fix Proxy 502 gateway error when it is configured with Keystore TLS and admin API is called

\[fix]\[broker] Fix unack count when mixing non batch index and batch index acks

\[fix]\[broker] Fix web tls url null cause NPE

\[improve] \[broker] Improve logs for troubleshooting

\[improve]\[broker] Upgrade bookkeeper to 4.16.3

\[fix]\[broker] revert remove duplicate topics name when deleteNamespace

\[fix]\[broker] Fix deleting topic not delete the related topic policy and schema.

\[fix] \[broker] consider iowait as idle.

\[fix]\[client] Fix repeat consume when using n-ack and batched messages

\[fix]\[fn] Fix the --batch-builder not working error for functions

\[fix] \[broker] Fix isolated group not work problem.

\[improve]\[bk] Improve getIsolationGroup by avoid creating arrayList

\[fix]\[auto-recovery] Improve to the ReplicaitonWorker performance by deleting invalid underreplication nodes

\[improve]\[meta] Improve fault tolerance of blocking calls by supporting timeout

\[fix]\[client] Avoid ack hole for chunk message

\[fix]\[broker] Fix unsubscribe non-durable subscription error

\[fix]\[client] Fix logging problem in pulsar client

\[fix]\[broker] Avoid splitting one batch message into two entries in StrategicTwoPhaseCompactor

\[fix]\[broker] Cleanup correctly heartbeat bundle ownership when handling broker deletion event

\[fix]\[broker] Fix write duplicate entries into the compacted ledger after RawReader reconnects

\[fix] \[broker] remove bundle-data in local metadata store.

\[improve]\[broker] Make read compacted entries support maxReadSizeBytes limitation

\[fix]\[misc] Bump broker okio version to 3.4.0

\[fix]\[io] Fix --retain\[-key]-ordering not working error for sink

\[fix]\[misc] Bump GRPC version to 1.55.3 to fix CVE

\[fix]\[broker] Fix potential case cause retention policy not working on topic level

\[fix]\[fn] Fix ProducerConfig cannot update error

\[fix]\[broker]Fix chunked messages will be filtered by duplicating

\[improve] \[ml] Persist mark deleted ops to ZK if create cursor ledger was failed

\[fix]\[client] Fix cannot retry chunk messages and send to DLQ

\[fix]\[client] Fix consumer can't consume resent chunked messages

### KoP

\[fix]\[proxy] Disconnect the coordinator broker for NOT\_COORDINATOR error

\[proxy] Support transaction and fix wrong coordinator lookup

\[proxy] Support receiving messages from multiple brokers

\[proxy] Support existing admin APIs

\[proxy] Support sending messages to multiple brokers

\[proxy] Reset the connection field to broker when broker is down

### AMQP1\_0 Connector

\[CI] Adjust CI to test the corresponding Pulsar image version

Fix integration test.

### AWS SQS Connector

Improve sqs source doc.

Improve sqs sink docs.

### StreamNative Pulsar Plugins

Enable broker interceptor for audit log test

Remove the disableBrokerInterceptors() method from test

Implement Oxia State Store

### Function Mesh Worker Service

Fix possible NPE errors

Set retain\[Key]Ordering to false if it is null

Load connector definition from ConnectorCataLog CRD.

Support json format logs and yaml format log config file

Change integration test ci trigger mode to pull\_request.

Support using sidecar to send logs to pulsar

Use AuthConfig.GenericAuth field to replace auth secret


# StreamNative Weekly Release Notes v3.1.0.4
Source: https://docs.streamnative.io/release-notes/pulsar/v3.1/component-changelogs-v3.1.0.4



# StreamNative Weekly Release Notes v3.1.0.4

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.1.0.4](https://github.com/streamnative/pulsar/releases/tag/v3.1.0.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.1.0.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.1.0.4/images/sha256-4da43062682870cfdcd3ac472f806d32f0a5cf7cca2eefd1f6af9cf4e9398548)

## General Changes

### Apache Pulsar

([#21015)](https://github.com/apache/pulsar/pull/21015))) Revert "\[fix]\[broker] Fix PulsarService.getLookupServiceAddress returns wrong port if TLS is enabled

\[fix] \[auto-recovery] Fix pulsar ledger auditor dead lock problem.

\[fix] \[auto-recovery] Fix PulsarLedgerUnderreplicationManager notify problem.

\[fix]\[broker] Fix PulsarService.getLookupServiceAddress returns wrong port if TLS is enabled

\[improve] \[broker] disable balancing based on DirectMemory.

\[fix]\[auto-recovery] Fix metadata store deadlock due to BookkeeperInternalCallbacks.Processor

\[fix] \[bookie] Fix RocksDB configuration

\[fix]\[broker] fix bug caused by optimistic locking

\[fix]\[broker] Backport fix UniformLoadShedder selecet wrong overloadbroker and underloadbroker

\[fix]\[ci] Enable CI for branch-3.1

\[fix] \[client] fix same producer/consumer use more than one connection per broker

### KoP

Add authorization check for schema registry when disabling multi-tenant metadata

Reduce the proxy NAR size by excluding the schema registry dependency

Upload proxy NAR package to release

Init schema registry topic for default tenant when starting protocol handler

\[schema registry] Support handling Bearer token for schema registry service

Optimize authorization by caching authorization results

### StreamNative Pulsar Plugins

Fix OxiaStateStoreProviderImpl int to long error

### Cloud Pulsar Plugins

Fix REST API interceptor check for creating partitioned topic with properties

### Function Mesh Worker Service

Change connectorSearchIntervalSeconds default value to 600s.

### Google BigQuery Sink Connector

Improve sink docs.


# V3.1.0.5
Source: https://docs.streamnative.io/release-notes/pulsar/v3.1/v3.1.0.5



# StreamNative Weekly Release Notes v3.1.0.5

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.1.0.5](https://github.com/streamnative/pulsar/releases/tag/v3.1.0.5)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.1.0.5/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.1.0.5/images/sha256-634ed81f74478aec158c5958866ab851ef707dd3aafb0d915f326e86156eec35)

## General Changes

### Apache Pulsar

\[fix]\[broker] Fix unload operation stuck when use ExtensibleLoadManager

\[improve]\[broker] use ConcurrentHashMap in ServiceUnitStateChannel and avoid recursive update error

\[fix] \[bk-client] Fix bk client MinNumRacksPerWriteQuorum and EnforceMinNumRacksPerWriteQuorum not work problem.

\[fix]\[broker] Fix heartbeat namespace create event topic and cannot delete heartbeat topic

\[fix]\[broker] Fix heartbeat namespace create transaction internal topic

\[fix]\[broker] Fix inconsistent topic policy

\[fix]\[broker] rackaware policy is ineffective when delete zk rack info after bkclient initialize

\[improve]\[ci] Add new CI unit test group "Broker Group 4" with cluster migration tests

\[fix]\[broker]\[branch-3.1] Fix lookup heartbeat and sla namespace bundle when using extensible load manager

\[fix]\[test] Fix resource leaks with Pulsar Functions tests

\[fix]\[test] Fix some resource leaks in compaction tests

\[feat]\[sql] Support UUID for json and avro

\[fix]\[test] Fix a resource leak in ClusterMigrationTest

\[fix] \[broker] Make specified producer could override the previous one

\[fix]\[ci] Fix docker image building by releasing more disk space before building

\[fix]\[broker]\[branch-3.1] Fix inconsistent topic policy

\[fix] \[metadata] Fix zookeeper related flacky test

\[improve] \[auto-recovery] \[branch-3.1] Migrate the replication testing from BookKeeper to Pulsar.

\[fix] \[broker] fix flaky test PatternTopicsConsumerImplTest

\[fix]\[sec] Fix MultiRoles token provider when using anonymous clients

\[fix]\[broker]Check that the super user role is in the MultiRolesTokenAuthorizationProvider plugin

\[fix]\[test] Fix flaky CompactionTest.testDispatcherMaxReadSizeBytes

\[fix]\[test] Fix flaky test NarUnpackerTest

\[fix]\[ml] Fix thread safe issue with RangeCache.put and RangeCache.clear

\[fix]\[build] Upgrade Lombok to 1.18.30 to support compiling with JDK21

\[fix]\[sec] Add OWASP Dependency Check suppressions

\[fix]\[txn] fix the consumer stuck due to deduplicated messages in pending ack state

\[improve] \[client] Merge lookup requests for the same topic

\[fix]\[fn] fix functions\_log4j2.xml delete strategy config

\[fix]\[broker]Fixed produce and consume when anonymousUserRole enabled

\[fix]\[broker] Fixed reset for AggregatedNamespaceStats

\[fix]\[broker] replicator leak when removeReplicator in NonPersistentTopic

### KoP

Attach Kafka txn\_metadata to messageMetadata in KafkaEntryFormatter

\[fix] Run messageReadStats metrics registerFailedEvent execute on netty thread

Ignore metadata init exception to avoid rolling upgrade from failing

Ignore the flaky MultiLedgerTest.testListOffsetForEmptyRolloverLedger

Fix txn marker to the offset topic cannot be read

Support alter cleanup policy

### AMQP1\_0 Connector

Improve sink and source connector docs.

### pulsarctl

Support status check for pulsarctl command

### StreamNative Pulsar Plugins

\[audit-log]\[fix] Fix audit log producer cache

Fix passing transactional flag in KafkaCompactionParser

### Cloud Pulsar Plugins

\[branch-3.1] Revert Pulsar version and remove useless repo

\[rest-api-interceptor] Ignore system topics for max topic count check

### Function Mesh Worker Service

23e78851 Bump function mesh to v0.17.0

disable golang runtime by default

allow submit very long name resources

Set usingInsecureAuth to false by default

Add configs: javaOpts/labels/logConfig to CustomRuntimeOptions

Read connector catalogs from namespaces.

### Google BigQuery Sink Connector

Add dockerfile metadata on source docs.

Improve source connector docs.

### Snowflake Connector

\[fix] Remove the dependency on `com.beust.jcommander`

\[fix] Fix the tableName configuration is not respected

### Aws EventBridge Connector

\[fix] Fix getting wrong metadata value of `sequence_id` and `producer_name`


# V3.1.0.6
Source: https://docs.streamnative.io/release-notes/pulsar/v3.1/v3.1.0.6



# StreamNative Weekly Release Notes v3.1.0.6

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.1.0.6](https://github.com/streamnative/pulsar/releases/tag/v3.1.0.6)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.1.0.6/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.1.0.6/images/sha256-735e53216cc6457c45ab98d5c1935568ba4b8910fc95cebea83beeba62603d89)

## General Changes

### Apache Pulsar

\[fix]\[broker]\[branch-3.1] Fix issue with consumer read uncommitted messages from compacted topic

([#21465)](https://github.com/apache/pulsar/pull/21465))) Revert "\[fix]\[broker] Fix issue with consumer read uncommitted messages from compacted topic

([#21270)](https://github.com/apache/pulsar/pull/21270))) Revert "\[fix]\[client] Avert extensive time consumption during table view construction

\[fix] \[ml] Fix orphan scheduled task for ledger create timeout check

\[fix] \[broker] Fix thousands orphan PersistentTopic caused OOM

\[fix]\[ml] Fix unfinished callback when deleting managed ledger

\[fix]\[client] Fix print error log 'Auto getting partitions failed' when expend partition.

\[fix]\[broker] Fix failure while creating non-durable cursor with inactive managed-ledger

\[fix]\[broker] Fix the deadlock when using BookieRackAffinityMapping with rackaware policy

\[fix]\[broker] Fix create topic with different auto creation strategies causes race condition

\[fix]\[broker] Fix namespace bundle stuck in unloading status

\[fix]\[broker] Fix PulsarService/BrokerService shutdown when brokerShutdownTimeoutMs=0

\[fix]\[broker] Fix issue with consumer read uncommitted messages from compacted topic

\[fix]\[broker] Avoid pass null role in MultiRolesTokenAuthorizationProvider

\[fix]\[txn] OpRequestSend reuse problem cause tbClient commitTxnOnTopic timeout unexpectedly

\[fix]\[test] Fix LocalBookkeeperEnsemble resource leak in tests

\[fix]\[client] Avert extensive time consumption during table view construction

\[fix]\[txn] Ack all message ids when ack chunk messages with transaction.

\[fix]\[build] Fix apt download issue in building the docker image

\[fix]\[broker] Ignore individual acknowledgment for CompactorSubscription when an entry has been filtered.

\[fix]\[broker] Fix MultiRoles token provider NPE when using anonymous clients

\[fix]\[sec] Upgrade Zookeeper to 3.8.3 to address CVE-2023-44981

119b83201f Bump version to 3.1.2-SNAPSHOT

\[fix]\[sec] Upgrade Netty to 4.1.100 to address CVE-2023-44487

\[fix]\[sec] Upgrade Jetty to 9.4.53 to address CVE-2023-44487

\[fix]\[proxy] Move status endpoint out of auth coverage

\[fix]\[sec] Upgrade snappy-java to 1.1.10.5

### KoP

\[schema-registry]\[fix] Fix schema ID generation logic

\[Transaction] Make the list offset request aware of the read-committed isolation level

\[fix] Use log end offset as the earliest offset when topic is empty

\[fix]\[schema-registry] Fix conflict schema version

### Cloud Storage Connector

Optimize Azure blob storage connector config validation logic

Add azure blob storage sink connector docs.

Add Google Cloud Storage sink connector docs.

Add AWS s3 sink connector docs.

### AWS Lambda Connector

SNIP-105: Support new data format for AWS lambda connector

Make AWS Lambda sink connector private

### pulsarctl

Fix error that functions cannot set auto-ack to false

Fix source built\_int list called error interface.

### Function Mesh Worker Service

02f6d2c3 Use local registry for sn-java and generic runner images

align version for branch 3.1

Support read customize connector catalogs.

Support generic runtime

Append version to the description field of ConnectorDefinition for load from connector catalog

Fix state store and add tests for oxia state store

Bump function-mesh to v0.18.0

### Lakehouse Connector

Update snappy dependency

### Snowflake Connector

Fix wrong value for configuration `processingGuarantees`


# V3.1.0.7
Source: https://docs.streamnative.io/release-notes/pulsar/v3.1/v3.1.0.7



# StreamNative Weekly Release Notes v3.1.0.7

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.1.0.7](https://github.com/streamnative/pulsar/releases/tag/v3.1.0.7)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.1.0.7/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.1.0.7/images/sha256-433a58ec9addaaae7b219fb0f822c7fa5fb029a0a07a6aa8e0f270bd0b4c7d08)

## General Changes

### Apache Pulsar

\[fix] \[log] fix the vague response if topic not found

\[improve] \[broker] Let the producer request success at the first time if the previous one is inactive

\[fix] \[build] rename schema\_example.conf to schema\_example.json

\[fix]\[sec] Upgrade rabbitmq client to address CVE-2023-46120

\[cleanup]\[client] Fix inconsistent API annotations of `getTopicName`

\[fix]\[broker] Fix resource\_quota\_zpath

\[fix]\[broker] Do not write replicated snapshot marker when the topic which is not enable replication

\[fix]\[broker] Fix setReplicatedSubscriptionStatus incorrect behavior

\[fix] \[broker] Delete topic timeout due to NPE

\[improve]\[broker] Support not retaining null-key message during topic compaction

\[fix]\[broker] Duplicate LedgerOffloader creation when namespace/topic…

### AoP

Add rabbitmq amqp-client dependency for test

### KoP

\[schema-registry] Support schema references (part-1)

\[schema-registry] Support subject version permanent delete

\[schema-registry] Normalize the short subject name with default namespace

\[schema-registry] Handle URL encoded subject name

\[improve] Skip compacted message when recovering txn entries

\[schema-registry]\[tests] Migrate some passed tests of RestApiTest

\[refactor]\[schema-registry] Refactor schema registry and remove MemorySchemaStorage

\[schema-registry] Fix the breaking change where long topic names cannot be applied to the URL

\[schema-registry] Support list soft deleted versions

\[fix] Remove find lastCompactedOffset logic when fetching end offset

Fix NPE for OffsetsFetch v7 or earlier requests

Apply restrict checkstyle rules for indents and spaces

### Cloud Storage Connector

Allow set none for avroCodec and parquetCodec

fix(doc): fix incorrect values for `formatType`

### AWS SQS Connector

Try fix auto labeling.

### AWS Lambda Connector

8d635e9 Fix auto label bot not work.

feat(doc): improve connector documentation

feat(config): update connector yaml configuration file

Support configuring AWS Access Key and Secret Key

\[feat] Support ISO-8601 data format for event time

### StreamNative Pulsar Plugins

\[SNCompactor] Support not retaining null-key messages in SNCompactor

([#1331)](https://github.com/streamnative/sn-pulsar-plugins/pull/1331))) Revert "\[SNCompactor] Support not retaining null-key messages in SNCompactor

\[SNCompactor] Support not retaining null-key messages in SNCompactor

Bump pulsar version to 3.1.0.7

Fix cve

### Function Mesh Worker Service

Fallback to reason field if the lastState's message is empty

make memory padding configurable

Fix getSinkList and getSourceList impl to avoid showing fields details.

### Google Pub / Sub Connector

Fix auto label bot not work.

### Google BigQuery Sink Connector

Fix auto label bot not work.

### Snowflake Connector

Improve snowflake sink connector doc

### Aws EventBridge Connector

Fix auto label bot not work.

Fix some docs and deprecated eventBusResourceName config.


# V3.1.1.1
Source: https://docs.streamnative.io/release-notes/pulsar/v3.1/v3.1.1.1



# StreamNative Weekly Release Notes v3.1.1.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.1.1.1](https://github.com/streamnative/pulsar/releases/tag/v3.1.1.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.1.1.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.1.1.1/images/sha256-d2d1aeb5fe2201187e3f9d4e3e18f4b31a282fbb1abc1bd8de75d955ee691bb4)

## General Changes

### Apache Pulsar

\[fix]\[broker] Record GeoPersistentReplicator.msgOut before producer#sendAsync

\[improve]\[broker] Print recoverBucketSnapshot log if cursorProperties are empty

\[fix]\[broker] Fix typo in the config key

\[fix]\[offload] Don't cleanup data when offload met MetaStore exception

\[improve]\[build] Upgrade Apache ZooKeeper to 3.9.1

\[fix]\[broker] Fix incorrect unack count when using shared subscription on non-persistent topic

\[fix]\[broker] Fixed getting incorrect KeyValue schema version

\[fix]\[admin] Fix KeyValue schema compatibility check caused OOM

\[fix]\[broker] Fix lookupRequestSemaphore leak when topic not found

\[fix]\[broker] Fix memory leak during topic compaction

\[improve]\[admin] Add clusters check when set replication clusters

\[fix]\[build] Fix Stage Docker images fail on M1 Mac

### KoP

\[schema-registry] Add JSON schema provider

\[improve] Gets the eventExecutor from the request context to register event instead of the fixed eventExecutor

\[schema-registry] Add avro schema provider

\[SNIP-110] KoP topic compaction work with transactions - part1

### Cloud Storage Connector

Update permission describe for AWS S3.

### AMQP1\_0 Connector

Load sensitive fields from secrets

### AWS Lambda Connector

\[fix]\[doc] Fix incorrect link in doc

\[Fix] Fix flush timer terminates upon first check if no flush is needed

### pulsarctl

Update golang.org/x/net

### StreamNative Pulsar Plugins

Replace GCS hadoop connector shaded artifact

Replace hadoop-common guava-shaded dependency

Fix the packages cloud storage failed to find gs schema

\[pulsarctl-plugin] Bump client-go to `0.20.15`

Remove shaded protobuf for hadoop-common also from pulsar-tools

Fix the metadata tool CI

Removed shaded Protobuf dependency from hadoop

a695c0fe Fixed more Go x/net version update

Fix the dependency conflict with pulsar broker

Fix detector go dep

Update aws-java-sdk

Update go dependencies to fix CVEs

### Google Pub / Sub Connector

Load sensitive fields from secrets

### Activemq Connector

Load sensitive fields from secrets


# V3.1.1.2
Source: https://docs.streamnative.io/release-notes/pulsar/v3.1/v3.1.1.2



# StreamNative Weekly Release Notes v3.1.1.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.1.1.2](https://github.com/streamnative/pulsar/releases/tag/v3.1.1.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.1.1.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.1.1.2/images/sha256-98e9c44cc70114216300c5f2674943920a84b88fb09832f937e5ea4f1c9569cc)

## General Changes

### Apache Pulsar

\[fix]\[broker] Fixed the ExtensibleLoadManagerImpl internal system getTopic failure when the leadership changes #21764

d276550533 Fix testNoCleanupOffloadLedgerWhenMetadataExceptionHappens

\[fix]\[client] Fix producer thread block forever on memory limit controller

\[fix]\[broker] Fix the issue of topics possibly being deleted.

\[fix]\[broker] Skip topic auto-creation for ExtensibleLoadManager internal topics

\[fix]\[broker] Fixed ServiceUnitStateChannel monitor to tombstone only inactive bundle states

\[improve]\[broker] Avoid record inactiveproducers when deduplication is disable.

\[fix]\[sec] Upgrade org.bouncycastle:bc-fips to 1.0.2.4

\[fix] \[broker] network package lost if enable haProxyProtocolEnabled

\[fix]\[sec] Bump avro version to 1.11.3 for CVE-2023-39410

### KoP

\[schema-registry] Support normalize for registry schema

Add KSN proxy dashboard

Remove unused CI workflows

\[schema-registry] Support subject permanent delete

Fix wrong ENTRY\_ORIGINAL\_BASEOFFSET\_KEY

Support configuring kopAllowedNamespaces dynamically

\[schema-registry] Support parse schema including references

SNIP-112: Return short topic names for OffsetFetch requests without topics

\[schema-registry] Add Protobuf schema provider

\[Proxy] Fix thread safety issues in BrokerConnectionGroup

Add Proxy metrics

([#227)](https://github.com/streamnative/sn-kop/pull/227))) Revert "\[SNIP-110] KoP topic compaction work with transactions - part1

### Cloud Storage Connector

Update nick-invision to nick-fields

Add sensitive column for configuration properties

d634875 Fix format errors for note.

Fix typos in doc

Refactor create a connector section docs.

### AMQP1\_0 Connector

Add sensitive column for configuration properties

Fix format errors for note.

Refactor create a connector section docs.

Fix not success upload image

### AWS SQS Connector

Update nick-invision to nick-fields

Add sensitive column for configuration properties

\[Doc] PIP-01: Improve data format for AWS SQS sink connector.

\[Implement] PIP-01: Improve data format for AWS SQS sink connector.

Support load config from secrets

Upgrade pulsar version to 3.1.x of master branch.

cbdd5e0 Fix format errors for note.

Refactor create a connector section docs.

### AWS Lambda Connector

Refactor connector creation doc

Update nick-invision to nick-fields

Add sensitive column for configuration properties

### StreamNative Pulsar Plugins

Update go x/crypto to 0.17 to fix CVE-2023-48795

### Function Mesh Worker Service

Support load docsLink and iconLink for connector catalog.

update retry github action owner

### Google Pub / Sub Connector

Update nick-invision to nick-fields

Add sensitive column for configuration properties

### Google BigQuery Sink Connector

Update nick-invision to nick-fields

Add sensitive column for configuration properties

86abce6 Fix format errors for note.

Refactor create a connector section docs.

### Snowflake Connector

Update nick-invision to nick-fields

Add sensitive column for configuration properties

Refactor connector creation doc

### Aws EventBridge Connector

Update nick-invision to nick-fields

Add sensitive column for configuration properties

Fix typos in doc

Refactor create a connector section docs.

### Activemq Connector

Update nick-invision to nick-fields

Add sensitive column for configuration properties


# V3.1.2.1
Source: https://docs.streamnative.io/release-notes/pulsar/v3.1/v3.1.2.1



# StreamNative Weekly Release Notes v3.1.2.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.1.2.1](https://github.com/streamnative/pulsar/releases/tag/v3.1.2.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.1.2.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.1.2.1/images/sha256-3ef9ef55cf921c5496de6f34bc9517d5f674e8596be8d8dd694b6504fadf9acd)

## General Changes

### Apache Pulsar

\[improve]\[broker] Improve NamespaceUnloadStrategy error message

\[fix]\[txn] Fix getting last message ID when there are ongoing transactions

\[improve]\[broker] Skip loading the NAR packages if not configured

\[fix]\[sec] Exclude avro from hadoop-client

\[improve] \[client] Prevent reserve memory with a negative memory size to avoid send task stuck

\[fix]\[client] Fix messages in the batch container timed out unexpectedly

\[fix]\[broker] fix the wrong value of BrokerSrevice.maxUnackedMsgsPerDispatcher

\[fix]\[broker]Fix NonPersistentDispatcherMultipleConsumers ArrayIndexOutOfBoundsException

\[improve]\[build] Add a default username in the image

\[fix] \[broker] Fix break change: could not subscribe partitioned topic with a suffix-matched regexp due to a mistake of PIP-145

\[fix] \[client] Messages lost due to TopicListWatcher reconnect

\[improve]\[broker] Don't rollover empty ledgers based on inactivity

\[fix]\[broker] Fix compaction/replication data loss when expire messages

\[fix] \[ml] Fix retry mechanism of deleting ledgers to invalidate

\[improve]\[broker] defer the ownership checks if the owner is inactive (ExtensibleLoadManager)

\[fix] \[broker] Update topic policies as much as possible when some ex was thrown

\[improve]\[io] Make connectors load sensitive fields from secrets

\[fix]\[broker] Delete compacted ledger when topic is deleted

\[fix]\[broker] Avoid compaction task stuck when the last message to compact is a marker

\[fix]\[fn] Fix Deadlock in Functions Worker LeaderService

\[fix]\[test] Fix PerformanceProducer send count error

\[improve]\[broker] cleanup the empty subscriptionAuthenticationMap in zk when revoke subscription permission

\[fix]\[broker] Fix TableViewLoadDataStoreImpl NPE

\[improve]\[proxy] Fix comment about enableProxyStatsEndpoints

b0e8b5abb5 Upgrade OWASP dependency check maven plugin version

### KoP

Fix offset commit timeout error due to incorrect send timer implementation

9f421ac6 Set project version with 3.1.x

\[schema-registry] Support TLS authentication

\[schema-registry] Wait reader read latest data after writing

\[schema-registry] Support list subjects for schema

\[feature] Support TLS authentication

Only authentication SASL\_PLAINTEXT OR SASL\_SSL endpoint

Increase the default maxReadEntriesNum to 50

\[schema-registry] Support `https` protocol and multi-listeners

Make loading offsets synchronous to avoid race condition

### Cloud Storage Connector

\[Doc] Add doc for `includeTopicToMetadata` configuration

Add default value for the partitionType to `partition`

\[feat]\[proposal-2] Support including the topic name to the metadata(#836)

\[Proposal-2] Support including the topic name to the metadata

Add AWS-S3-Sink doc for proposal-1

Add doc for new partitioner

\[Proposal-1] Support Partitioner Refactoring

Proposal-1: Partitioner Refactoring

### AMQP1\_0 Connector

Fix incorrect docker images link.

### pulsarctl

Update jose2go to fix GHSA-mhpq-9638-x6pw

### StreamNative Pulsar Plugins

\[fix]\[cve] Exclude logback from zookeeper

\[fix]\[detector] Cleanup inactive broker's e2e detector

Include one older txn log file in the backup when needed

Update jose2go to 1.6.0 to address GHSA-mhpq-9638-x6pw

Upgrade pulsar to 3.1.1.2

Release SN-RBAC to branch-3.1

### Cloud Pulsar Plugins

Fix build.sh

Release apikeys\&oauth2 module first in build.sh

Fix test pom

Release SN-RBAC to branch-3.1

### Function Mesh Worker Service

add transformFunctionEnabled and disabled by default


# V3.1.2.2
Source: https://docs.streamnative.io/release-notes/pulsar/v3.1/v3.1.2.2



# StreamNative Weekly Release Notes v3.1.2.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.1.2.2](https://github.com/streamnative/pulsar/releases/tag/v3.1.2.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.1.2.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.1.2.2/images/sha256-02909cfa707efe3493d236ea906c3cc3a7a7327a5dd5bb892f3ac6903761299b)

## General Changes

### Apache Pulsar

\[fix]\[broker] Fix issue with GetMessageIdByTimestamp can't find match messageId from compacted ledger

([#21600)](https://github.com/apache/pulsar/pull/21600))) Revert "\[fix]\[broker] Fix issue with GetMessageIdByTimestamp can't find match messageId from compacted ledger

\[improve]\[ci] Upgrade pulsar-client-python to 3.4.0 to avoid CVE-2023-1428

\[fix]\[broker] Fix issue with GetMessageIdByTimestamp can't find match messageId from compacted ledger

\[fix]\[broker] Avoid consumers receiving acknowledged messages from compacted topic after reconnection

\[fix] \[broker] Fix write all compacted out entry into compacted topic

\[fix] \[broker] Replication stopped due to unload topic failed

\[fix]\[broker] Fix deadlock while skip non-recoverable ledgers.

\[fix]\[broker] Fix getMessageById throws 500

\[fix]\[client] Fix multi-topics consumer could receive old messages after seek

\[fix] \[broker] Fix reader stuck when read from compacted topic with read compact mode disable

\[improve] \[proxy] Add a check for brokerServiceURL that does not support multi uri yet

\[fix]\[broker] Fix schema deletion error when deleting a partitioned topic with many partitions and schema

\[fix]\[broker] Correct schema deletion for parititioned topic

\[fix]\[client] Fix ConsumerBuilderImpl#subscribe silent stuck when using pulsar-client:3.0.x with jackson-annotations prior to 2.12.0

\[fix] \[broker] add timeout for health check read.

\[improve] \[bk] Upgrade BookKeeper dependency to 4.16.4

\[fix]\[test] Make base test class method protected so that it passes ReportUnannotatedMethods validation

\[fix]\[broker] Restore the broker id to match the format used in existing Pulsar releases

\[fix]\[broker] Fix leader broker cannot be determined when the advertised address and advertised listeners are configured

\[fix]\[broker] Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set

\[fix]\[broker] Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set

\[fix]\[broker] Fix PulsarService.getLookupServiceAddress returns wrong port if TLS is enabled

([#21633)](https://github.com/apache/pulsar/pull/21633))) Revert "\[fix]\[broker] Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set

\[fix]\[broker] Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set

### MoP

Fix broker enable dedup cause client publish msg NPE

Add test for resubscribe

Add filter system topic when using EventCenter

Fix unsubscribe topic cause the test failed.

call equals on formatted strings since they will never be null

remove subs from subscription manager on unsubscribe

### KoP

\[transactions] Implement KIP-664 DescribeProducers

### Cloud Storage Connector

fix: cleanup the current batch after flush crashed

Update base image

Update base image

### AMQP1\_0 Connector

Update base image

### AWS SQS Connector

Update base image

### AWS Lambda Connector

update-base-image

### pulsarctl

Add method to mark bookie readonly

Build arm64 linux executable binary artifact

### StreamNative Pulsar Plugins

Upgrade Oxia Java Client to 0.1.1

Update Oxia to 0.0.10

e944ad01 Fix flaky test

rest: Increase test await time

fix test SNCompactionServiceTest

\[pulsar-detector] Fix wrong file name and line number in logs

\[pulsar-detector] Apply golangci-lint checks

Add arm64 artifacts release

Enhance find orphan ledger command.

### Cloud Pulsar Plugins

\[fix]\[sn-rbac] skip sn-rbac for adminTenants API

Fix: return error future when JWT expired

Fix: do not throws EX when calling AuthorizationProviderOAuth.isSuperUser

### Function Mesh Worker Service

reduce integration test image size with slim base image

clean up the disk

bump function-mesh to 0.19.0

Ignore exception when connector customize catalogs is empty.

Use oxia:0.2 image for testing

### Google Pub / Sub Connector

Improve source connector docs.

Improve sink connector docs.

Add release image workflow

### Google BigQuery Sink Connector

Update docker base

### Snowflake Connector

Update base image


# V3.1.2.3
Source: https://docs.streamnative.io/release-notes/pulsar/v3.1/v3.1.2.3



## StreamNative Weekly Release Notes v3.1.2.3

#### General Changes

### Apache Pulsar

\[improve]\[broker] Consistently add fine-grain authorization to REST API

\[fix]\[broker]\[branch-3.1] Fix broker not starting when both transactions and the Extensible Load Manager are enabled

7e28e8404f Fix presto-distribution/LICENSE

\[improve]\[fn] Add configuration for connector & functions package url sources

\[fix]\[offload] Fix Offload readHandle cannot close multi times.

\[fix] \[broker] print non log when delete partitioned topic failed

\[fix]\[txn]Fix TopicTransactionBuffer potential thread safety issue

\[fix]\[sec] Upgrade Jetty to 9.4.54.v20240208 to address CVE-2024-22201

\[fix]\[test] Fix test testAsyncFunctionMaxPending

\[fix] \[client] Do no retrying for error subscription not found when disabled allowAutoSubscriptionCreation

\[fix]\[broker]Support setting `autoSkipNonRecoverableData` dynamically in expiryMon…

\[fix]\[fn] Use unified PackageManagement service to download packages

\[fix] \[broker] Expire messages according to ledger close time to avoid client clock skew

\[fix]\[build]\[branch-3.1] Fix compile issue in test

([#22101)](https://github.com/apache/pulsar/pull/22101))) Revert "\[improve]\[admin] Expose the offload threshold in seconds to the amdin

\[improve]\[broker] Add fine-grain authorization to retention admin API

\[fix]\[broker]\[branch-3.0] Return getOwnerAsync without waiting on source broker upon Assigning and Releasing and handle role change during role init

\[fix]\[broker]\[branch-3.0] Set ServiceUnitStateChannel topic compaction threshold explicitly, improve getOwnerAsync, and fix other bugs

\[improve]\[broker] Add an error log to troubleshoot the failure of starting broker registry.

\[fix]\[ml] Make mlOwnershipChecker asynchronous so that it doesn't block/deadlock threads

\[improve] \[broker] Do not try to open ML when the topic meta does not exist and do not expect to create a new one. #21995

\[fix]\[sec] Add a check for the input time value

\[fix] \[txn] Get previous position by managed ledger.

\[improve]\[broker] Do not close the socket if lookup failed due to LockBusyException

\[improve] \[broker] Not close the socket if lookup failed caused by bundle unloading or metadata ex

\[fix] \[broker] Fix can not subscribe partitioned topic with a suffix-matched regexp

\[fix] \[broker] Subscription stuck due to called Admin API analyzeSubscriptionBacklog

\[improve] \[broker] Do not print an Error log when responding to `HTTP-404` when calling `Admin API` and the topic does not exist.

\[improve]\[broker] Do not retain the data in the system topic

\[fix]\[test] fix test testSyncNormalPositionWhenTBRecover

\[fix]\[broker] Fix hash collision when using a consumer name that ends with a number

\[fix] \[client] fix huge permits if acked a half batched message

\[fix] \[broker] Enabling batch causes negative unackedMessages due to ack and delivery concurrency

\[improve]\[admin] Expose the offload threshold in seconds to the amdin

\[improve]\[broker] Cache the internal writer when sent to system topic.

4da9a2070c Bump version to 3.1.3-SNAPSHOT

\[fix]\[test] Fix test testTransactionBufferMetrics

\[improve]\[ci] Exclude jose4j to avoid CVE-2023-31582

\[fix] Bump org.apache.solr:solr-core from 8.11.1 to 8.11.3 in /pulsar-io/solr

\[fix] \[bk] Fix the BookKeeper license

a13326b23a Fix byte-buddy version in presto LICENSE

\[improve]\[fn] Optimize Function Worker startup by lazy loading and direct zip/bytecode access

\[fix]\[sec] Upgrade commons-compress to 1.26.0

\[fix]\[broker] Support running docker container with gid != 0

\[fix]\[broker]\[branch-3.1] Avoid PublishRateLimiter use an already closed RateLimiter

\[fix]\[broker] Sanitize values before logging in apply-config-from-env.py script

\[improve]\[ml] Filter out deleted entries before read entries from ledger.

### AoP

\[fix]\[test] Improve the declare exchange test

### KoP

Update LICENSE

### Cloud Storage Connector

\[fix] Fix small batches flushed into S3 together with normal large batches

### AWS Lambda Connector

Enable unit tests for weekly release

### pulsarctl

201be18 Disable bk unit test and fix it later ---

### StreamNative Pulsar Plugins

Use an old version of the sn/charts

Update pulsar-placement-policy module README

Add pulsar placement policy model to release channel

Add Pulsar placement policy module

736ad09e \[fix]\[sec] Upgrade commons-compress to 1.26.0

### Function Mesh Worker Service

a52f4467 Fix ci

d399115d Deprecate classloader


# V3.2.0.2
Source: https://docs.streamnative.io/release-notes/pulsar/v3.2/v3.2.0.2



## StreamNative Weekly Release Notes v3.2.0.2

#### General Changes

### Apache Pulsar

\[improve]\[broker] Consistently add fine-grain authorization to REST API

\[fix]\[broker]\[branch-3.2] Fix broker not starting when both transactions and the Extensible Load Manager are enabled

\[improve]\[fn] Add configuration for connector & functions package url sources

\[fix]\[test] fix test testSyncNormalPositionWhenTBRecover

\[fix]\[test] Fix test testAsyncFunctionMaxPending

\[fix]\[sec] Add a check for the input time value

\[improve]\[broker] Add fine-grain authorization to retention admin API

\[fix]\[sec] Upgrade Jetty to 9.4.54.v20240208 to address CVE-2024-22201

\[fix] \[broker] print non log when delete partitioned topic failed

### AoP

\[fix]\[test] Improve the declare exchange test

### KoP

\[test] Add test for abort transaction with Kafka admin

### AWS Lambda Connector

Enable unit tests for weekly release

### pulsarctl

9b654e7 Disable bk unit test and fix it later ---

### StreamNative Pulsar Plugins

Use an old version of the sn/charts

Update pulsar-placement-policy module README

Add pulsar placement policy model to release channel

Add Pulsar placement policy module

### Function Mesh Worker Service

fd893058 Fix ci


# V3.2.1.1
Source: https://docs.streamnative.io/release-notes/pulsar/v3.2/v3.2.1.1



## StreamNative Weekly Release Notes v3.2.1.1

#### General Changes

### Apache Pulsar

\[fix]\[sec] Revert "\[fix]\[sec] Add a check for the input time value (apache#22023)"

Fix the tests with same namespace name

\[fix]\[client] fix Reader.hasMessageAvailable might return true after seeking to latest

\[fix]\[client] GenericProtobufNativeSchema not implement getNativeSchema method.

\[fix]\[test] Fix flaky test BrokerServiceAutoSubscriptionCreationTest

\[fix]\[offload] Fix Offload readHandle cannot close multi times.

\[fix]\[txn]Fix TopicTransactionBuffer potential thread safety issue

\[fix] \[client] Do no retrying for error subscription not found when disabled allowAutoSubscriptionCreation

### KoP

\[fix]\[transaction] Fix send messages with transaction in async way

### StreamNative Pulsar Plugins

\*: upgrade branch-3.2 vesion to 3.2.0.2

\[SNP-RBAC] Support conditional role binding

\[SNP-RBAC] Support predefined roles configuration

### Cloud Pulsar Plugins

ApiKeys: Avoid check JWT token expired time in authorization

\[SN-RBAC] added functions, sources, sinks interceptor path

Oauth2: Avoid check JWT token expired time in authorization

### Function Mesh Worker Service

Validate functions\&connectors package url

3de3d7d6 Avoid error in tune runner vm

### Google BigQuery Sink Connector

feat: Support protobuf native schema.


# V3.2.1.2
Source: https://docs.streamnative.io/release-notes/pulsar/v3.2/v3.2.1.2



## StreamNative Weekly Release Notes v3.2.1.2

#### General Changes

### MoP

Fix ClassCastException when scheduling to look up

### KoP

\[proxy] Fix duplicated sends when pending produce requests are ignored by network issue

### StreamNative Pulsar Plugins

Update pulsar placement policy module name

rbac: support permissions list

### Function Mesh Worker Service

4c99dd47 Cleanup disk

### Lakehouse Connector

2f2994e fix interface change


# V3.2.1.3
Source: https://docs.streamnative.io/release-notes/pulsar/v3.2/v3.2.1.3



# StreamNative Weekly Release Notes v3.2.1.3

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.2.1.3](https://github.com/streamnative/pulsar/releases/tag/v3.2.1.3)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.2.1.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.2.1.3/images/sha256-00d8a6020249811b3231a2cf24483ffaa10af00a2ad21278238dd66b6875a846)

## General Changes

### Apache Pulsar

\[fix]\[broker] Fix wrong double-checked locking for readOnActiveConsumerTask in dispatcher

\[fix] \[client] Unclear error message when creating a consumer with two same topics

\[improve]\[broker] Add fine-grain authorization to ns/topic management endpoints

\[improve]\[broker] Add missing configuration keys for caching catch-up reads

\[improve]\[misc] Upgrade checkstyle to 10.14.2

\[improve] \[broker] Support create RawReader based on configuration

\[fix]\[ci] Enable CI for branch-3.2

### MoP

Fix proxy keepalive issue

### KoP

\[proxy] Fix duplicated sends when pending produce requests are ignored by network issue

### AMQP1\_0 Connector

Auth SN docker hub

### AWS SQS Connector

Auth SN docker hub

### pulsarctl

Auth SN docker hub

### StreamNative Pulsar Plugins

Auth SN dockerhub

### Function Mesh Worker Service

Check null value before use VpaSpec

Auth SN docker hub


# V3.2.1.4
Source: https://docs.streamnative.io/release-notes/pulsar/v3.2/v3.2.1.4



# StreamNative Weekly Release Notes v3.2.1.4

<Note title="Note">
  Please note this StreamNative Pulsar distribution will require a valid StreamNative subscription license key to run otherwise the image will fail to start.
</Note>

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.2.1.4](https://github.com/streamnative/pulsar/releases/tag/v3.2.1.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.2.1.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.2.1.4/images/sha256-c26415e5c08227ccc5f6e07b015f8b9084c84f34a8b2eb15dabcc93bc09735ea)

## General Changes

### Apache Pulsar

\[fix]\[broker] Fix issue of field 'topic' is not set when handle GetSchema request

6255b1282e \[improve]\[test]\[branch-3.2] Improve ManagedLedgerTest.testGetNumberOfEntriesInStorage

9927b55b17 \[fix]\[test]\[branch-3.2] Fix broken ManagedLedgerTest.testGetNumberOfEntriesInStorage

\[improve]\[misc] Remove the call to sun InetAddressCachePolicy

\[fix]\[broker] Check cursor state before adding it to the `waitingCursors`

\[fix]\[broker] Avoid expired unclosed ledgers when checking expired messages by ledger closure time

\[fix]\[misc] Make ConcurrentBitSet thread safe

\[fix]\[test] Fix flaky RGUsageMTAggrWaitForAllMsgsTest

\[fix]\[client] Fix wrong results of hasMessageAvailable and readNext after seeking by timestamp

\[fix]\[broker] Avoid execute prepareInitPoliciesCacheAsync if namespace is deleted

\[fix]\[fn] fix broken function-go test

\[fix]\[sec] Go Functions security updates

\[fix]\[test] Fix flaky ManagedLedgerErrorsTest.recoverAfterZnodeVersionError

\[fix] \[test] Fix flaky test ManagedLedgerTest.testGetNumberOfEntriesInStorage

\[fix]\[client] Consumer lost message ack due to race condition in acknowledge with batch message

\[fix]\[broker] Fix OpReadEntry.skipCondition NPE issue

\[fix]\[ml]Expose ledger timestamp

\[improve]\[misc] Include native epoll library for Netty for arm64

\[improve]\[misc] Upgrade Netty version to 4.1.105.Final

\[fix]\[client]Fixed getting an incorrect `maxMessageSize` value when accessing multiple clusters in the same process

\[improve]\[admin] Fix the `createMissingPartitions` doesn't response correctly

\[fix]\[broker] Fix ResourceGroups loading

\[fix]\[broker] Fix ResourceGroup report local usage

\[fix] \[broker] fix mismatch between dispatcher.consumerList and dispatcher.consumerSet

\[fix] \[broker] Close dispatchers stuck due to mismatch between dispatcher.consumerList and dispatcher.consumerSet

### KoP

\[SNIP-122] Part 3: Support other admin protocols for dot-separated namespace prefix

### pulsarctl

Support no auth context

fix token

Add docker hub login

### StreamNative Pulsar Plugins

Update license error message for 3.2

Cherry pick license feature to branch 3.2

### Function Mesh Worker Service

1f042764 Add brokerAdditionalServlet

allow passing allowed runtimeFlags for java runtime


# V3.2.2.1
Source: https://docs.streamnative.io/release-notes/pulsar/v3.2/v3.2.2.1



# StreamNative Weekly Release Notes v3.2.2.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.2.2.1](https://github.com/streamnative/pulsar/releases/tag/v3.2.2.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.2.2.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.2.2.1/images/sha256-536d9385bd4c25a4518820f0194fa57ff71d758ba65db2a4ca166f371716e911)

## General Changes

### Apache Pulsar

\[improve]\[io]: Add validation for JDBC sink not supporting primitive schema

\[fix]\[ml] No rollover inactive ledgers when metadata service invalid

\[improve] \[broker] Servlet support response compression

\[fix]\[broker] Skip topic.close during unloading if the topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to use ns.checkOwnershipPresentAsync for ExtensibleLoadBalancer

\[fix]\[build] Fix networkaddress.cache.negative.ttl config

\[improve]\[broker] Don't log brokerClientAuthenticationParameters and bookkeeperClientAuthenticationParameters by default

aece9fc843 Bump version to next snapshot version

\[improve] \[broker] Avoid repeated Read-and-discard when using Key\_Shared mode

### KoP

\[CI] Fix docker-compose command not found

### Cloud Storage Connector

Use the Apache images to run tests, in order to avoid permission issues.

### StreamNative Pulsar Plugins

\[test] Fix metadata integration test

### Google Pub / Sub Connector

Auth SN docker hub


# V3.2.2.3
Source: https://docs.streamnative.io/release-notes/pulsar/v3.2/v3.2.2.3



# StreamNative Weekly Release Notes v3.2.2.3

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.2.2.3](https://github.com/streamnative/pulsar/releases/tag/v3.2.2.3)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.2.2.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.2.2.3/images/sha256-c00124631b9b5905c93e9a2e4a043ed24eb0742458b34e441c7ef535eb3fabfe)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.2.2.3/images/sha256-b7131b71a61cbf230a1381f192b80c7990ad5983ad12f26244b6e541463b0515)

## General Changes

### Apache Pulsar

\[improve]\[offload] Apply autoSkipNonRecoverableData configuration to tiered storage

\[fix]\[broker] Fix NPE causing dispatching to stop when using Key\_Shared mode and allowOutOfOrderDelivery=true

\[improve]\[build] Upgrade OWASP Dependency check version to 9.1.0

\[fix]\[broker] Fix a deadlock in SystemTopicBasedTopicPoliciesService during NamespaceEventsSystemTopicFactory init

\[improve]\[broker] Optimize gzip compression for /metrics endpoint by sharing/caching compressed result

\[fix]\[io] Kafka Source connector maybe stuck

\[fix]\[sec] Upgrade Bouncycastle to 1.78

\[fix]\[test] Flaky-test: testMessageExpiryWithTimestampNonRecoverableException and testIncorrectClientClock

\[fix]\[broker] Create new ledger after the current ledger is closed

\[fix]\[broker] Optimize /metrics, fix unbounded request queue issue and fix race conditions in metricsBufferResponse mode

\[improve]\[broker] Improve Gzip compression, allow excluding specific paths or disabling it

\[improve]\[test] Replace usage of curl in Java test and fix stream leaks

\[fix] \[broker] Prevent long deduplication cursor backlog so that topic loading wouldn't timeout

([#22479)](https://github.com/apache/pulsar/pull/22479))) Revert "\[fix] \[broker] Prevent long deduplication cursor backlog so that topic loading wouldn't timeout

\[fix] \[broker] Prevent long deduplication cursor backlog so that topic loading wouldn't timeout

\[fix]\[txn]Handle exceptions in the transaction pending ack init

### KoP

Add metrics documents for network in/out bytes

### AWS Lambda Connector

Upgrade commons-compress to fix CVE

### StreamNative Pulsar Plugins

22f7059e test

24dceac0 align test image and pulsar version for rbac

Upgrade ZK, aws client and commons-configuration2

Upgrade x/net and protobuf to fix vulnerabilities

### Google BigQuery Sink Connector

Upgrade checkstyle version


# V3.2.2.4
Source: https://docs.streamnative.io/release-notes/pulsar/v3.2/v3.2.2.4



# StreamNative Weekly Release Notes v3.2.2.4

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.2.2.4](https://github.com/streamnative/pulsar/releases/tag/v3.2.2.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.2.2.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.2.2.4/images/sha256-b668b207b1720c7543bdc5a612fcf12258fc0cf7a74b622d72cdbed1c0835915)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.2.2.4/images/sha256-dac89a3c58c06dd6392367379afcf6740749c3ba948f5e33d7751b92efa507e4)

## General Changes

### Apache Pulsar

\[fix] \[broker] Fix metrics pulsar\_topic\_load\_failed\_count is 0 when load non-persistent topic fails and fix the flaky test testBrokerStatsTopicLoadFailed

\[fix]\[test] Fix the flaky tests of ManagedLedgerImplUtilsTest

\[fix]\[broker] Reader stuck after call hasMessageAvailable when enable replicateSubscriptionState

\[fix]\[test] Flaky-test: ManagedLedgerTest.testTimestampOnWorkingLedger

\[improve]\[broker] Propagate cause exception in TopicBusyException when applicable

\[improve]\[meta] Log a warning when ZK batch fails with connectionloss

\[fix]\[test] Clear fields in test cleanup to reduce memory consumption

\[improve]\[admin] Check if the topic existed before the permission operations

\[fix]\[admin] Fix namespace admin api exception response

\[fix]\[broker] Fix BufferOverflowException and EOFException bugs in /metrics gzip compression

\[fix] Include swagger annotations in shaded client lib

\[fix]\[io] CompressionEnabled didn't work on elasticsearch sink

\[fix]\[offload] Increase file upload limit from 2048MiB to 4096MiB for GCP/GCS offloading

\[fix]\[broker] upgrade jclouds 2.5.0 -> 2.6.0

\[fix]\[ml] Fix NPE of getValidPositionAfterSkippedEntries when recovering a terminated managed ledger

\[improve]\[broker] Support X-Forwarded-For and HA Proxy Protocol for resolving original client IP of http/https requests

\[fix]\[broker] Fix broken topic policy implementation compatibility with old pulsar version

\[fix]\[broker] Fix typos in Consumer class

\[improve]\[test] Move ShadowManagedLedgerImplTest to flaky tests

\[improve]\[broker] Repeat the handleMetadataChanges callback when configurationMetadataStore equals localMetadataStore

\[improve]\[broker] Add topic name to emitted error messages.

\[improve] Make the config `metricsBufferResponse` description more effective

\[fix]\[test] SchemaMap in AutoConsumeSchema has been reused

\[improve]\[broker] backlog quota exceed limit log replaced with `debug`

\[fix]\[broker] Fix message drop record in producer stat

\[fix]\[broker] Update topic partition failed when config `maxNumPartitionsPerPartitionedTopic<0`

### AoP

\[fix] Fix AoP can't work when enabling Pulsar transaction

### KoP

Fix requiredAcks == 0 handling

### Cloud Storage Connector

fix: Read JSON directly from the original data when formatType=json

### AMQP1\_0 Connector

Fix integration test due to invalid package storage path

### AWS SQS Connector

Upgrade commons-compress to fix CVE

fix integrate test.

Update license to proprietary one

### StreamNative Pulsar Plugins

\[RBAC] Bump RBAC test image to fix compatible problem

\[RBAC] Fix test json data

auditlog: perf: create AuditLogEvent instances only when there's a matching rule

Deferred generation id for AuditLogEvent

auditlog: replace rw lock & HashMaps with ConcurrentHashMap to reduce blocking code

auditlog: optimize uri matching by organizing condition and eliminating streams

auditlog: Cache regex compilation

\[RBAC] Skip unsupported operations for rbac authorization provider

### Google BigQuery Sink Connector

upgrade depend to fix cve

### Aws EventBridge Connector

Bump org.apache.avro:avro from 1.10.2 to 1.11.3

### StreamNative Tiered storage

Fix complex pojo encode json message error

Use zk lock by default

Fix avro didn't support enum type problem

Transfer schema before acquire lock.

Improve delta compactor


# V3.2.2.5
Source: https://docs.streamnative.io/release-notes/pulsar/v3.2/v3.2.2.5



# StreamNative Weekly Release Notes v3.2.2.5

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.2.2.5](https://github.com/streamnative/pulsar/releases/tag/v3.2.2.5)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.2.2.5/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.2.2.5/images/sha256-45e0e6af4e508acc88b16535167437297ef1ecd4333576958b54d2e79045432d)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.2.2.5/images/sha256-5a1a0ca883d72a15c24a488a1f66c3166dcc7f287563967bcbc68e518481ffd6)

## General Changes

### KoP

Fix requiredAcks == 0 handling


# V3.2.2.6
Source: https://docs.streamnative.io/release-notes/pulsar/v3.2/v3.2.2.6



# StreamNative Weekly Release Notes v3.2.2.6

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.2.2.6](https://github.com/streamnative/pulsar/releases/tag/v3.2.2.6)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.2.2.6/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.2.2.6/images/sha256-563fb449ad09a7dbda9185958788ad809b238f75b42efdf51f6b42d0e9f57d9b)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.2.2.6/images/sha256-447de1d4f3c2dcf8bbc65ade0fa4f72505eafa8bdad34239778a2d48189b54a3)

## General Changes

### Apache Pulsar

\[improve] \[broker] Add additionalSystemCursorNames ignore list for TTL check

2c22af5c07 \[fix]\[build]\[branch-3.2] Remove unused import added in cherry-picking

\[improve]\[offload] Replace usage of shaded class in OffsetsCache

\[fix]\[offload] Fix OOM in tiered storage, caused by unbounded offsets cache

\[fix] \[broker] Fix nothing changed after removing dynamic configs

Revert "\[fix]\[sec] Upgrade Debezium oracle connector version to avoid…

\[fix] Fix Reader can be stuck from transaction aborted messages.

\[fix]\[broker] avoid offload system topic

\[improve]\[ws] Add memory limit configuration for Pulsar client used in Websocket proxy

\[fix]\[broker] Disable system topic message deduplication

\[fix]\[fn]make sure the classloader for ContextImpl is `functionClassLoader` in different runtimes

\[fix]\[test] Clear MockedPulsarServiceBaseTest fields to prevent test runtime memory leak

\[fix]\[sec] Upgrade Debezium oracle connector version to avoid CVE-2023-4586

\[fix]\[sec] Upgrade elasticsearch-java version to avoid CVE-2023-4043

\[fix]\[sec] Upgrade aws-sdk.version to avoid CVE-2024-21634

\[fix] \[client] Fix Consumer should return configured batch receive max messages

\[fix]\[broker] Avoid being stuck when closing the broker with extensible load manager

\[fix]\[io] Fix es index creation

\[improve] Retry re-validating ResourceLock with backoff after errors

\[fix] \[test] Fix flaky test ReplicatorTest

\[fix]\[broker] One topic can be closed multiple times concurrently

\[fix] \[broker] Part-2: Replicator can not created successfully due to an orphan replicator in the previous topic owner

\[improve] \[broker] Create partitioned topics automatically when enable topic level replication

\[fix] \[broker] Part-1: Replicator can not created successfully due to an orphan replicator in the previous topic owner

\[fix] \[ml] Mark delete stuck due to switching cursor ledger fails

### AoP

Bump Pulsar 3.2.2.6

### MoP

6525444 Fix compile issue

### KoP

Use BrokerService#registerCustomDynamicConfiguration to register the dynamic config

### Function Mesh Worker Service

Support list functions/connectos across tenants and namespaces

Update doc to correct functionality on REST api

### StreamNative Tiered storage

Upgrade bk and spark dependency to fix CVEs


# V3.2.2.7
Source: https://docs.streamnative.io/release-notes/pulsar/v3.2/v3.2.2.7



# StreamNative Weekly Release Notes v3.2.2.7

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.2.2.7](https://github.com/streamnative/pulsar/releases/tag/v3.2.2.7)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.2.2.7/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.2.2.7/images/sha256-05adff79da3b8fd60c5c785160389400de32b4cda05aa93ce11581d56c1648e6)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.2.2.7/images/sha256-7e6a269a07fb5e1ae9e4b325c49dab2fd8f3889c2ee3eb187984b5ee0f30057e)

## General Changes

### Apache Pulsar

\[fix]\[broker] Fix cursor should use latest ledger config

\[cleanup]\[ml] ManagedCursor clean up.

c2532b9741 \[fix]\[test]\[branch-3.2] Fix DeduplicationDisabledBrokerLevelTest. Adjust to PR 22034 presence.

41610ee016 Revert "\[fix]\[test]\[branch-3.2] Fix broken ManagedLedgerTest.testGetNumberOfEntriesInStorage"

d8a35ef797 Revert "\[improve]\[test]\[branch-3.2] Improve ManagedLedgerTest.testGetNumberOfEntriesInStorage"

\[fix]\[admin] Fix can't delete tenant for v1

0a68f82f1d \[improve]\[ci]\[branch-3.2] Upgrade actions in pulsar-ci and pulsar-ci-flaky, port owasp cache change

\[fix]\[test] Fix NPE in BookKeeperClusterTestCase tearDown

\[fix]\[broker] fix replicated subscriptions for transactional messages

\[fix]\[sec] Upgrade postgresql version to avoid CVE-2024-1597

\[fix]\[client] Fix ReaderBuilder doest not give illegalArgument on connection failure retry

\[fix]\[broker] Fix ProducerBusy issue due to incorrect userCreatedProducerCount on non-persistent topic

\[fix] \[broker] rename to changeMaxReadPositionCount

\[fix]\[storage] ReadonlyManagedLedger initialization does not fill in the properties

\[fix]\[broker] usedLocallySinceLastReport should always be reset

### AoP

Bump Pulsar 3.2.2.7

### KoP

Apply StreamNative copyright header

### AWS SQS Connector

Update pulsar version

### pulsarctl

Add trivy scan workflow to avoid vulnerabilities

\[fix] Upgrade go version to 1.21 to fix CVE-2023-24538

### StreamNative Pulsar Plugins

fe44bfc0 Revert "detecotr: add goreleaser to support coress compile with CGO"

detector: support build detector for amd64 with cgo

32831887 detecotr: add goreleaser to support coress compile with CGO

rbac: move produce and consume permission to messages resource type

rbac: support subscription level in srn

rbac: support subscription level permission

rbac: add permission validation when create/update role

rbac: default value for organization and instance

rbac: add swagger file to release artifacts

rbac: using jackson deserializer to null check.

rbac: fix wrong response entity

rbac: add permission endpoint to swagger file

rbac: fix deny when role binding without condition

detector: fix build on detector

detector: support SLA and latency detecting for Kafka protocol

### Snowflake Connector

Fix `topic2table` not working and fix the doc

Fix messageId2Long cannot handle TopicMessageId

Add documentation for the schema conversion rule

### StreamNative Tiered storage

Fix timeout exception type and support fetch timeout configurable

Remove the workflow that publish the ts to the gcr

Improve container quarantined when resource is enough


# V3.2.3.1
Source: https://docs.streamnative.io/release-notes/pulsar/v3.2/v3.2.3.1



# StreamNative Weekly Release Notes v3.2.3.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.2.3.1](https://github.com/streamnative/pulsar/releases/tag/v3.2.3.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.2.3.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.2.3.1/images/sha256-a38cbeb5a91e583074ace34ded03d6e26aa5d440b47dd9e460cadb161e14b6d2)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.2.3.1/images/sha256-9b206e5299183f122aa39da9b2ccdf660495016ae8eda3cc080f8c3e48a690e7)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.2.3.1/images/sha256-9b206e5299183f122aa39da9b2ccdf660495016ae8eda3cc080f8c3e48a690e7)

## General Changes

### Apache Pulsar

\[improve] \[broker] Trigger offload on topic load

\[fix]\[ml]: subscription props could be lost in case of missing ledger during recovery

\[fix] \[ml] Add entry fail due to race condition about add entry failed/timeout and switch ledger

\[fix]\[broker] Make ExtensibleLoadManagerImpl.getOwnedServiceUnits async

\[fix]\[offload] Break the fillbuffer loop when met EOF

### KoP

Remove unnecessary debug log during the entry encode

### AWS Lambda Connector

Update license headers to proprietary

### pulsarctl

Upgrade the dependency version to fix vulnerabilities

### StreamNative Pulsar Plugins

enable zookeeper in detector test

### Cloud Pulsar Plugins

\[improve]\[api-keys] Improve error logs when initialize failed

### Google Pub / Sub Connector

Update license header

### Google BigQuery Sink Connector

Update license headers

### Snowflake Connector

Update license header

### Aws EventBridge Connector

Update license header


# V3.2.3.2
Source: https://docs.streamnative.io/release-notes/pulsar/v3.2/v3.2.3.2



# StreamNative Weekly Release Notes v3.2.3.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.2.3.2](https://github.com/streamnative/pulsar/releases/tag/v3.2.3.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.2.3.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.2.3.2/images/sha256-31c39b07f151480c5f8ea1134b4d125cf6aceb0cf45eaf09ff81bf2902aed8eb)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.2.3.2/images/sha256-465891ca76e751fe4c6f840f15b08a4f3a11e1c26f2b8fa1a5d3e45324ba3849)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.2.3.2/images/sha256-465891ca76e751fe4c6f840f15b08a4f3a11e1c26f2b8fa1a5d3e45324ba3849)

## General Changes

### Apache Pulsar

\[improve]\[broker] Remove ClassLoaderSwitcher to avoid objects allocations and consistent the codestyle

\[improve]\[broker] avoid creating new objects when intercepting

\[fix]\[broker] EntryFilters fix NoClassDefFoundError due to closed classloader

\[improve]\[broker] Clear thread local BrokerEntryMetadata instance before reuse

\[improve]\[broker] Close protocol handlers before unloading namespace bundles

### KoP

Reduce the default partitions for system topics

### Snowflake Connector

Add connection string identifier

### StreamNative Tiered storage

Allow to overwrite the consumer configuration

Export the count of fatal errors caused the service shutdown

Fix AddFileAction miss stats field problem.


# V3.2.3.3
Source: https://docs.streamnative.io/release-notes/pulsar/v3.2/v3.2.3.3



# StreamNative Weekly Release Notes v3.2.3.3

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.2.3.3](https://github.com/streamnative/pulsar/releases/tag/v3.2.3.3)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.2.3.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.2.3.3/images/sha256-61d00ea0acd89c7f52e8963164bf82249acc6655944a54ccb888dd01d860c0c4)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.2.3.3/images/sha256-7ab210a2318f7623add2cf0848823a094f469a66e099080b7f53934aa7aa5ab3)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.2.3.3/images/sha256-7ab210a2318f7623add2cf0848823a094f469a66e099080b7f53934aa7aa5ab3)

## General Changes

### KoP

Prevent the possible Netty TooLongFrameException with the default entry format

### StreamNative Pulsar Plugins

detector: revert test dockerfile to avoid breaking

c710fa65 detector: tidy mod

detector: replace cloud image detector for a quick test

detector: fix the unacked messages causes high e2e latency

detector: tidy dependencies

detector: refine detector doc

detector: remove legacy code

detector: refactor pulsar detector

### StreamNative Tiered storage

Iceberg support tabular catalog service


# V3.3.0.2
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.0.2



# StreamNative Weekly Release Notes v3.3.0.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.3.0.2](https://github.com/streamnative/pulsar/releases/tag/v3.3.0.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.0.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.0.2/images/sha256-b2c7bae1606a81342f3e7b9349efb00eac03e9b177d0c689c2584f0c54427c25)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.0.2/images/sha256-7a6c1440b6eceaf11fffeb0794a958b53550bbdeac85040a6560df97f016c446)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.0.2/images/sha256-7a6c1440b6eceaf11fffeb0794a958b53550bbdeac85040a6560df97f016c446)

## General Changes

### Apache Pulsar

\[improve] \[broker] PIP-355: Enhancing Broker-Level Metrics for Pulsar

\[improve] \[client] improve the class GetTopicsResult

\[improve] Upgrade IPAddress to 5.5.0

\[fix]\[cli] Fix Pulsar standalone "--wipe-data"

\[improve]\[ci] Migrate from Gradle Enterprise to Develocity

\[fix]\[misc] Add proper nslookup (included in bind-tools) to docker image

\[improve]\[broker] Reuse topic OpenTelemetry attributes

\[fix]\[cli] Fix healthcheck script pulsar-zookeeper-ruok.sh

\[fix]\[cli] Fix Pulsar standalone shutdown - bkCluster wasn't closed

\[fix]\[cli] Fix the shell script parameter passthrough syntax

\[improve]\[ci] Add arm64 image build

\[improve]\[broker] Reduce number of OpenTelemetry consumer attributes

\[feat]\[broker] PIP-264: Add OpenTelemetry consumer metrics

\[fix] \[broker] fix topic partitions was expanded even if disabled topic level replication

\[fix] \[broker] fix deadlock when disable topic level Geo-Replication

6e3adc8175 Bump version to 3.3.1-SNAPSHOT

\[fix]\[misc] Disable JFR based telemetry collection since it's not used

\[improve]\[build] Support git worktree working directory while building docker images

\[fix]\[broker] fix replicated subscriptions for transactional messages

\[improve]\[broker] Remove ClassLoaderSwitcher to avoid objects allocations and consistent the codestyle

\[fix]\[cli] Fix expiration of tokens created with "pulsar tokens create"

\[improve] \[test] Add a test to guarantee the TNX topics will not be replicated

\[fix]\[broker] EntryFilters fix NoClassDefFoundError due to closed classloader

\[improve]\[broker] avoid creating new objects when intercepting

\[fix] \[broker] maintain last active info in memory only.

\[fix] \[broker] disable loadBalancerDirectMemoryResourceWeight by default

\[fix] \[conf] fix configuration name and typo.

\[improve] Validate range of argument before long -> int conversion

\[fix]\[meta] Check if metadata store is closed in RocksdbMetadataStore

\[improve] \[broker] Do not call cursor.isCursorDataFullyPersistable if disabled dispatcherPauseOnAckStatePersistentEnabled

\[fix]\[ml]: subscription props could be lost in case of missing ledger during recovery

\[improve]\[broker] Close protocol handlers before unloading namespace bundles

\[improve]\[offload] Allow to disable the managedLedgerOffloadDeletionLagInMillis

\[fix]\[broker] usedLocallySinceLastReport should always be reset

\[fix] \[broker] rename to changeMaxReadPositionCount

\[fix]\[sec] Upgrade postgresql version to avoid CVE-2024-1597

\[improve] \[broker] Add additionalSystemCursorNames ignore list for TTL check

\[improve] Refactored BK ClientFactory to return futures

\[fix] Remove blocking calls from BookieRackAffinityMapping

### AoP

\[branch-3.3] Bump pulsar 3.3.0.2

\[fix] Release `EntryImpl` while reading exchange topic

### MoP

9b2f122 fix checkstyle issue

9b49858 Fix compile issue.

### KoP

Bump Pulsar to 3.3.0.2 and fix some tests due to upstream changes

### StreamNative Pulsar Plugins

Bump Pulsar 3.3.0.2

Update k8s client sdk to 0.30.1

\[cve] Upgrade Kerby to 2.0.3

### Cloud Pulsar Plugins

bump pulsar 3.3.0.2

### Function Mesh Worker Service

Use stg oauth2 parameters

### Snowflake Connector

Add connection string identifier


# V3.3.0.3
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.0.3



# StreamNative Weekly Release Notes v3.3.0.3

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.3.0.3](https://github.com/streamnative/pulsar/releases/tag/v3.3.0.3)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.0.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.0.3/images/sha256-e20fb9c14e8909b323a973cfd46e2b6b21066b5bf8c8ffd4d6ac9766a1edb926)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.0.3/images/sha256-9abaaa9ecd45eea1ca40dabfd696cdb394f1de1ccdac96adacc1f875523a67a7)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.0.3/images/sha256-9abaaa9ecd45eea1ca40dabfd696cdb394f1de1ccdac96adacc1f875523a67a7)

## General Changes

### Apache Pulsar

([#22867)](https://github.com/apache/pulsar/pull/22867))) \[fix] Revert "\[fix]\[cli] Fix the shell script parameter passthrough syntax

\[improve] Upgrade to Oxia client 0.3.0

\[fix]\[broker] Fix topic status for oldestBacklogMessageAgeSeconds continuously increases even when there is no backlog.

\[fix]\[cli] Fix the pulsar-daemon parameter passthrough syntax

e75f6ba1d4 \[fix] Fix cherry-pick for #22892

\[fix]\[broker] The topic might reference a closed ledger

\[improve]\[misc] Upgrade to Netty 4.1.111.Final and switch to use grpc-netty-shaded

### KoP

\[improve] Optimize take producer state snapshot logic

### StreamNative Pulsar Plugins

Update xnio-api to 3.8.14 to fix CVE-2023-5685

Bump oxia to 0.3.0

### Lakehouse Connector

Update the snappy download link in the Dockerfile

Install the snappy lib in the alpine image

Make seprate workflow for the release

Add dockerfile for the pulsar-io-lakehouse

### StreamNative Tiered storage

fix slf4j conflict


# V3.3.0.4
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.0.4



# StreamNative Weekly Release Notes v3.3.0.4

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.3.0.4](https://github.com/streamnative/pulsar/releases/tag/v3.3.0.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.0.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.0.4/images/sha256-693edd47603e78ad1b11019f714a1514315d0aeb17cee42bf67b6f1f9d96e03c)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.0.4/images/sha256-0e1fc995aa776797e795af7b4a06f4bf844475812e989becdf729c6866a8fbad)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.0.4/images/sha256-0e1fc995aa776797e795af7b4a06f4bf844475812e989becdf729c6866a8fbad)

## General Changes

### Apache Pulsar

\[feat]\[broker] PIP-321 Introduce allowed-cluster at the namespace level

\[fix]\[broker] Fix updatePartitionedTopic when replication at ns level and topic policy is set

\[improve]\[fn] Make producer cache bounded and expiring in Functions/Connectors

\[fix]\[client] Fix orphan consumer when reconnection and closing are concurrency executing

\[fix]\[ci] Fix jacoco code coverage report aggregation

\[improve]\[misc] Upgrade Bookkeeper to 4.17.1

\[improve]\[misc] Replace rename-netty-native-libs.sh script with renaming with maven-shade-plugin

\[fix]\[ci] Replace removed macos-11 with macos-latest in GitHub Actions

([#22908)](https://github.com/apache/pulsar/pull/22908))) Revert "\[improve]\[broker] Optimize `ConcurrentOpenLongPairRangeSet` by RoaringBitmap

\[improve] \[broker] make system topic distribute evenly.

\[fix]\[misc] Rename netty native libraries in pulsar-client-admin-shaded

\[improve]\[misc] Set Alpine base image to 3.20 instead of 3.19.1

\[cleanup]\[misc] Remove classifier from netty-transport-native-unix-common dependency

\[improve]\[broker] Optimize `ConcurrentOpenLongPairRangeSet` by RoaringBitmap

\[fix]\[broker] Check the markDeletePosition and calculate the backlog

\[fix]\[fn] Support compression type and crypto config for all producers in Functions and Connectors

\[fix] \[broker] broker log a full thread dump when a deadlock is detected in healthcheck every time

\[fix] \[client] Fix resource leak in Pulsar Client since HttpLookupService doesn't get closed

\[fix]\[test] Fix TableViewBuilderImplTest NPE and infinite loop

\[fix]\[fn] Enable optimized Netty direct byte buffer support for Pulsar Function runtimes

\[fix]\[misc] Topic name from persistence name should decode local name

\[fix] \[broker] Messages lost on the remote cluster when using topic level replication

### KoP

Log lookup data of all brokers when the topic lookup failed

### AWS SQS Connector

make source queue size configurable

af20c84 fix maunl workflow

d9fb023 feat: Support cutomize trigger a release

### pulsarctl

Fix json marshal error for Secrets and UserConfigs when creating/updating functions

### StreamNative Pulsar Plugins

rbac: support primary delegator for fully compatible

rbac: support patch/delete the role of role binding

rbac: refine document for all the components

### StreamNative Tiered storage

\[bugfix] fix lose data when get data from partitioned table

Iceberg cast utf8 error

Translate pulsar tenant and namespace to Iceberg namespace

Support configure offload policy

remove credentials in print configurations

Create namespace in rest catalog when namespace not exist

make iceberg expire snapshot use catalog

Add topic name in the log message

Remove credentials in log

Remove storage path check


# V3.3.0.5
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.0.5



# StreamNative Weekly Release Notes v3.3.0.5

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.3.0.5](https://github.com/streamnative/pulsar/releases/tag/v3.3.0.5)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.0.5/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.0.5/images/sha256-f0b634c132f1b05220ce0daa280e1415394e7e9cc5d446a39d6419cea01b0a72)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.0.5/images/sha256-897835fff0133bd3051e8a5271a0e6a3c2b148d73cbc7979ff020e7c18737e7c)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.0.5/images/sha256-897835fff0133bd3051e8a5271a0e6a3c2b148d73cbc7979ff020e7c18737e7c)

## General Changes

### Apache Pulsar

\[fix]\[broker] Can't connecte to non-persist topic when enable broker client tls

\[fix]\[test] Update OpenTelemetry receiver endpoint in integration test

\[fix]\[broker] Fix broker OOM when upload a large package.

\[improve]\[broker] Improve exception for topic does not have schema to check

\[improve] \[broker] PIP-356 Support Geo-Replication starts at earliest position

\[fix] \[broker] response not-found error if topic does not exist when calling getPartitionedTopicMetadata

\[improve] \[client] PIP-344 support feature flag supportsGetPartitionedMetadataWithoutAutoCreation

\[fix] \[client] PIP-344 Do not create partitioned metadata when calling pulsarClient.getPartitionsForTopic(topicName)

\[fix]\[broker] Ensure that PulsarService is ready for serving incoming requests

\[fix]\[broker] Update init and shutdown time and other minor logic (ExtensibleLoadManagerImpl only)

\[fix]\[broker] Asynchronously return brokerRegistry.lookupAsync when checking if broker is active(ExtensibleLoadManagerImpl only)

\[fix]\[broker] Support advertised listeners when gracefully transferring topics (ExtensibleLoadManagerImpl only)

\[fix]\[broker] Fix NPE after publishing a tombstone to the service unit channel

\[fix]\[broker] Immediately tombstone Deleted and Free state bundles

\[improve]\[broker]Ensure namespace deletion doesn't fail

### KoP

Fix the TopicExistsInfo object not recycled

### Cloud Storage Connector

\[fix]: fix Parquet/Avro format with separated key value avro-avro messages

### StreamNative Pulsar Plugins

Add Pulsar OIDC plugin

bump toolchain 1.22.4

Support multiple private keys token AuthenticationProvider

### Cloud Pulsar Plugins

Compatible changes for Auth0 Actions migration

Fix oidcIssuers not update after restart broker

### Function Mesh Worker Service

bump sn-operator to v0.5.0-rc.15

add kafka connect apis

Bump function mesh to 0.21.0

### Snowflake Connector

Improve json schema conversion

### StreamNative Tiered storage

Fix PB repeated nested field issue

Support protobuf native schema in the lakehouse storage

Create namespace if it supports and not exists

Upgrade iceberg to 1.5.2

Move the iceberg api call to one place

Enable the brokerMetadataInterceptro in the IT


# V3.3.0.6
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.0.6



# StreamNative Weekly Release Notes v3.3.0.6

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.3.0.6](https://github.com/streamnative/pulsar/releases/tag/v3.3.0.6)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.0.6/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.0.6/images/sha256-a30f7a8b3f06e64ba6df689df7588c36464452062069a4498020092f46d610d0)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.0.6/images/sha256-ebbd9d7feb4f9167b3aae9784609be3eea3c2c3ca67a066ddf8347a57120c550)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.0.6/images/sha256-ebbd9d7feb4f9167b3aae9784609be3eea3c2c3ca67a066ddf8347a57120c550)

## General Changes

### Apache Pulsar

\[improve] \[broker] Trigger offload on topic load

\[fix]\[admin] Fix half deletion when attempt to topic with a incorrect API

\[improve]\[build] Upgrade dependency-check-maven-plugin to 10.0.2

\[fix]\[misc] Remove RoaringBitmap dependency from pulsar-common

\[fix]\[broker] PulsarStandalone started with error if --stream-storage-port is not 4181

\[improve]\[broker] Use RoaringBitmap in tracking individual acks to reduce memory usage

\[fix]\[broker] Fix MessageDeduplication replay timeout cause topic loading stuck

\[fix] Make operations on `individualDeletedMessages` in lock scope

\[fix]\[ci] Fix OWASP Dependency Check download by using NVD API key

### KoP

Fix producer state snapshot not taken during shutdown

### pulsarctl

Bump pulsar version to 3.3.0.6 & Fix TestDeleteNonExistPartitionedTopic

### StreamNative Pulsar Plugins

\[improve] \[log] Change log level of consumer not found to WARN

### Snowflake Connector

\[feat] Support metadata field mapping

### StreamNative Tiered storage

Fix the the issue with the kop format message and kop schema registry

Add create namespace tests for the iceberg catalog

Using iceberg bom to import the iceberg dependency


# V3.3.0.7
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.0.7



# StreamNative Weekly Release Notes v3.3.0.7

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.3.0.7](https://github.com/streamnative/pulsar/releases/tag/v3.3.0.7)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.0.7/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.0.7/images/sha256-c9206df5a6120ef241821bd64f2df4e1f5563e8d45ca487d49bda300f79d4823)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.0.7/images/sha256-cc18bc74955ff41bbe967145ab1cfded8f26a2a2546608f4239681b59b18f0a6)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.0.7/images/sha256-cc18bc74955ff41bbe967145ab1cfded8f26a2a2546608f4239681b59b18f0a6)

## General Changes

### Apache Pulsar

\[fix] Upgrade to Oxia 0.3.1

\[fix]\[broker] Replication stuck when partitions count between two clusters is not the same

\[fix]\[broker] Fix stuck when enable topic level replication and build remote admin fails

\[fix]\[broker] Fix geo-replication admin client url

\[fix]\[broker]Fix lookupService.getTopicsUnderNamespace can not work with a quote pattern

\[fix]\[client] Fix pattern consumer create crash if a part of partitions of a topic have been deleted

### KoP

Fix producer state snapshot not taken during shutdown

### Function Mesh Worker Service

allow passing javaopts to kafka connect

support jwt token fallback for oauth2 handler

add integration tests for kafka connect

### StreamNative Tiered storage

Fix the bytes type handle error

Fix the running tests missing dependency issue


# V3.3.0.9
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.0.9



# StreamNative Weekly Release Notes v3.3.0.9

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.3.0.9](https://github.com/streamnative/pulsar/releases/tag/v3.3.0.9)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.0.9/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.0.9/images/sha256-f1f5e70b5bfd249e8f3740d09aa08061362a26cf90031e0b2b7f40e6fa394650)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.0.9/images/sha256-88836e98f39707d5e2f77eecb87c49da98a6080815436ba64550222ccd8ecd6d)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.0.9/images/sha256-88836e98f39707d5e2f77eecb87c49da98a6080815436ba64550222ccd8ecd6d)

## General Changes

### KoP

\[test] Adjust test about transaction timeout

### pulsarctl

ce80e93 fix ci

### Google Pub / Sub Connector

9dbdfbd fix ci

### Lakehouse Connector

Fix the get-version.sh file permission

### Google BigQuery Sink Connector

c0e1f56 fix ci

### StreamNative Tiered storage

Enable compact pulsar meta fields by default

Iceberg offloader add HMS support

fix data loss in iceberg


# V3.3.1.1
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.1.1



# StreamNative Weekly Release Notes v3.3.1.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.3.1.1](https://github.com/streamnative/pulsar/releases/tag/v3.3.1.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.1.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.1.1/images/sha256-a51a843a9896afc1d64307b2334cebf209c3e4d1c6d035207df1a88e0be7f509)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.1.1/images/sha256-ddedc9f2cc518ccd09367f60caf966a61726b8e3837a38df45456f2ba4fe49eb)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.1.1/images/sha256-ddedc9f2cc518ccd09367f60caf966a61726b8e3837a38df45456f2ba4fe49eb)

## General Changes

### Apache Pulsar

\[improve]\[fn] Add support for overriding additionalJavaRuntimeArguments with PF\_additionalJavaRuntimeArguments env

\[fix]\[build] Remove unnecessary Oracle maven repository from pom.xml

\[fix]\[client] Fix timeout handling in Pulsar Admin client

\[improve]\[misc] Optimize TLS performance by omitting extra buffer copies

\[fix]\[broker] Handle the case when `getOwnedServiceUnits` fails gracefully

\[fix]\[test] Fixed many tests of pulsar-proxy are not running

\[improve]\[misc] Improve AES-GCM cipher performance

\[improve]\[pip] PIP-366: Support to specify different config for Configuration and Local Metadata Store

\[improve]\[broker]Reuse method getAvailableBrokersAsync

\[fix] \[broker] fix replicated namespaces filter in filterAndUnloadMatchedNamespaceAsync

\[fix]\[client] TransactionCoordinatorClient support retry

\[fix]\[broker] Fix authenticate order in AuthenticationProviderList

\[fix]\[broker]A failed consumer/producer future in ServerCnx can never be removed

\[improve]\[broker] Support to specify auth-plugin, auth-parameters and tls-enable arguments when init cluster metadata

### KoP

\[test] Adjust test about transaction timeout

### StreamNative Pulsar Plugins

Optimizing OIDC provider

Using `authenticationService` to authenticate token

### Function Mesh Worker Service

a1e05871 use bitnami/kafka:3.4.1 for testing kafka

Make generic runtime fallback to using base image and add tests for bentos

### Lakehouse Connector

Fix the release ci

### Google BigQuery Sink Connector

fix integration test

### StreamNative Tiered storage

Update grafana dashboard

## Security Fixes


# V3.3.1.2
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.1.2



# StreamNative Weekly Release Notes v3.3.1.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.3.1.2](https://github.com/streamnative/pulsar/releases/tag/v3.3.1.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.1.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.1.2/images/sha256-c4b7ff31ee15f001cf2fc98248dd93e811e2063b284b785576c3374cb027c961)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.1.2/images/sha256-1257e47241467940079af810f96b5b10cc3c1c2841c53c681b11c979c8878da0)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.1.2/images/sha256-1257e47241467940079af810f96b5b10cc3c1c2841c53c681b11c979c8878da0)

## General Changes

### Apache Pulsar

\[improve] \[broker] Avoid subscription fenced error with consumer.seek whenever possible

\[improve] \[client]Add new ServiceUrlProvider implementation: SameAuthParamsAutoClusterFailover

\[fix] \[broker] Let Pending ack handler can retry to init when encounters a metadata store error

\[fix]\[client] Create the retry producer async

\[improve]\[proxy] Reuse authentication instance in pulsar-proxy

\[fix]\[client] Fix for early hit `beforeConsume` for MultiTopicConsumer

\[fix] \[meta] Oxia metadta store: Convert error to MetadataStoreException if operation failed

\[fix]\[broker] Fix 'Disabled replicated subscriptions controller' logic and logging

\[improve]\[broker] Explicitly close LB internal topics when playing a follower (ExtensibleLoadManagerImpl only)

\[fix] \[broker] Fix compatibility issues for PIP-344

\[fix]\[metadata] Upgrade Oxia to 0.3.2

\[improve]\[client] Add maxConnectionsPerHost and connectionMaxIdleSeconds to PulsarAdminBuilder

\[fix]\[broker] Fix the bug that elected leader thinks it's a follower

### KoP

Fix getOwnedServiceUnits has exception cause broker shutdown

### pulsarctl

update pulsar-client-go to master latest commit

### StreamNative Pulsar Plugins

bump pulsar 3.3.1.2

rbac: avoid delegation exception breaking the chain

rest: make header be case-insensitive

### Function Mesh Worker Service

cab53ed7 Fix kafka-client permission error

Support multiple tasks

## Security Fixes


# V3.3.1.3
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.1.3



# StreamNative Weekly Release Notes v3.3.1.3

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.3.1.3](https://github.com/streamnative/pulsar/releases/tag/v3.3.1.3)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.1.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.1.3/images/sha256-37041b012d65ef934272feb2b9aea3c84454d908215846b4833ccc9492838125)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.1.3/images/sha256-9bb2324d65392f4cbbdbf29400c94472affe167cc5156775841fb2926f3ac877)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.1.3/images/sha256-9bb2324d65392f4cbbdbf29400c94472affe167cc5156775841fb2926f3ac877)

## General Changes

### Apache Pulsar

\[fix] \[log] Do not print warn log when concurrently publishing and switching ledgers

\[fix]\[client] the nullValue in msgMetadata should be true by default

\[improve] \[client]Add new ServiceUrlProvider implementation: SameAuthParamsAutoClusterFailover

\[improve]\[client] Don't print info logs for each schema loaded by client

\[improve]\[broker] Improve pulsar\_topic\_load\_failed metric to record correct failed time

\[improve]\[broker] Optimize high CPU usage when consuming from topics with ongoing txn

\[fix] \[broker] Topic can never be loaded up due to broker maintains a failed topic creation future

\[fix]\[broker] Skip reading entries from closed cursor.

\[improve] \[broker] Optimize performance for checking max topics when the topic is a system topic

\[improve]\[broker] Should notify bundle ownership listener onLoad event when ServiceUnitState start (ExtensibleLoadManagerImpl only)

\[improve]\[broker] Support customized shadow managed ledger implementation

\[fix]\[test] Fix flaky SubscriptionSeekTest.testSeekIsByReceive

\[feat] Add scripts for updating BK RocksDB ini files

\[fix]\[client] Copy orderingKey to retry letter topic and DLQ messages and fix bug in copying

\[fix] DLQ to handle bytes key properly

\[fix]\[broker] Fix shadow topics cannot be consumed when the entry is not cached

### AoP

\[branch-3.3]\[fix] Change read max position to earliest position

### KoP

Add timeout for metadata recovery operation

### StreamNative Pulsar Plugins

Update oidc test token

rbac: migrate snrbac to unified rbac

rbac: fix the async exception chain

### Function Mesh Worker Service

f5878bc1 Increase broker memory in ci

Enhance error handling

Filter out empty configs when put kafka connect

fix kafka connect builtin connector use java-opts

support kafka connect builtin worker config

Fix function stats not correct error

Return annotations when get functions/connectors/kafka connects

Read kafka connect config from file

Support pause resume restart for kafka connect

## Security Fixes


# V3.3.1.4
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.1.4



# StreamNative Weekly Release Notes v3.3.1.4

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.3.1.4](https://github.com/streamnative/pulsar/releases/tag/v3.3.1.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.1.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.1.4/images/sha256-b0220502a8061ea3cecb92be7cb8b28ee5b90c07c353859311cd006745a8af8f)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.1.4/images/sha256-146303887c7441f5416f5813b591eec21e7ae4c39f300d9f447cc0d578f17c2f)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.1.4/images/sha256-146303887c7441f5416f5813b591eec21e7ae4c39f300d9f447cc0d578f17c2f)

## General Changes

### Apache Pulsar

\[improve]\[broker] Add msgInReplay subscription stat and metric to improve Key\_Shared observability

\[fix] StatsOutputStream: add string write function

### KoP

Support using metadata store to store offset and group metadata

### StreamNative Pulsar Plugins

detector: set isolation level to be read uncommitted

Implement new init method for OxiaStateStoreProviderImpl

### Cloud Pulsar Plugins

Add billing metrics for Ursa

\[SN-RBAC] added oauthSnRBACSkipApiList

### Function Mesh Worker Service

Update trace for OverProvisioned

Fix OverProvisioned's trace

Enhance error handling for kafka connect

## Security Fixes


# V3.3.1.5
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.1.5



# StreamNative Weekly Release Notes v3.3.1.5

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.3.1.5](https://github.com/streamnative/pulsar/releases/tag/v3.3.1.5)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.1.5/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.1.5/images/sha256-16235a23008a8c56b016d4201d0faabeec5d616ee01cfdefc7522f93ad6d4ff0)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.1.5/images/sha256-e68afb0503b8f76c2c591ee7872b1305604cfecb094c53817eeafcf6a2999454)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.1.5/images/sha256-e68afb0503b8f76c2c591ee7872b1305604cfecb094c53817eeafcf6a2999454)

## General Changes

### Apache Pulsar

\[fix]\[broker] Fix retry backoff for PersistentDispatcherMultipleConsumers

\[feat]\[meta] Bump oxia java version from 0.3.2 to 0.4.5

\[fix] Bump io.grpc from 1.56.0 to 1.56.1

\[fix]\[broker] fix pulsar-admin topics stats-internal caused a BK client thread a deadlock

\[improve]\[broker] Optimize message payload traffic for ShadowReplicator

\[fix]\[broker] Execute the pending callbacks in order before ready for incoming requests

\[fix]\[test] Fix flaky UnloadSubscriptionTest.testMultiConsumer

\[improve]\[admin] PIP-369 Introduce `unload` flag in `ns-isolation-policy set` call

\[improve]\[misc] Upgrade Netty to 4.1.113 and netty-tcnative to 2.0.66

\[improve]\[broker] Add retry for start service unit state channel (ExtensibleLoadManagerImpl only)

\[fix]\[broker] Fix brokers still retry start replication after closed the topic

\[improve]\[broker] Reschedule reads with increasing backoff when no messages are dispatched

### MoP

Support mTLS authentication for MoP

Fix workflow

Implement AuthenticationProviderMTls

### KoP

Redesign the metrics related to Fetch requests

### pulsarctl

Setup go version to 1.22 fix cve

### StreamNative Pulsar Plugins

7e5f3dac delete rbac CI since it has been moved to unified-rbac repo

Add authType field for IdentityPool

### Cloud Pulsar Plugins

fix kafka connect skip api regex

### Function Mesh Worker Service

Update oxia to latest and always pull image

Support alter/reset kafka connect offsets

Use `defaultNamespace` to check kafka connect permission

### Google BigQuery Sink Connector

Support batch max bytes limit to 10Mb

## Security Fixes


# V3.3.1.6
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.1.6



# StreamNative Weekly Release Notes v3.3.1.6

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.3.1.6](https://github.com/streamnative/pulsar/releases/tag/v3.3.1.6)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.1.6/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.1.6/images/sha256-1d23479dc60b3d7fbe952ab03b8c1514d4aa9efa91fe7e60229a6443b27acf06)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.1.6/images/sha256-3557bf71c5c2d69e0f5d28391bf5b00c3f3817b470ed4e6b23ccaa3d84b876d3)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.1.6/images/sha256-3557bf71c5c2d69e0f5d28391bf5b00c3f3817b470ed4e6b23ccaa3d84b876d3)

## General Changes

### Apache Pulsar

\[fix] \[broker] Fix system topic can not be loaded up if it contains data offloaded

### MoP

Fix mTls authorize bug

Fix mTLS authorization bug

### KoP

Redesign the metrics related to Fetch requests

### StreamNative Pulsar Plugins

Add mTLS expression verification for identity pool servlet

f41166bc Remove rbac from oidc test & Improve metrics and logs(#1895)

### Function Mesh Worker Service

Support set terminationGracePeriodSeconds

## Security Fixes


# V3.3.1.7
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.1.7



# StreamNative Weekly Release Notes v3.3.1.7

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.3.1.7](https://github.com/streamnative/pulsar/releases/tag/v3.3.1.7)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.1.7/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.1.7/images/sha256-6a8dc96a1f229f7284332122666aa6c80a91c3c1e1620902ab3e5bcc8ffb693f)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.1.7/images/sha256-a7cf0093fbf238dd7fe1ea70f35074a394d7b656fa8bad40e3ffdb1ce7e17a26)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.1.7/images/sha256-a7cf0093fbf238dd7fe1ea70f35074a394d7b656fa8bad40e3ffdb1ce7e17a26)

## General Changes

### Apache Pulsar

([#23226)](https://github.com/apache/pulsar/pull/23226))) Revert "\[improve]\[broker] Reschedule reads with increasing backoff when no messages are dispatched

([#23284)](https://github.com/apache/pulsar/pull/23284))) Revert "\[fix]\[broker] Fix retry backoff for PersistentDispatcherMultipleConsumers

([#23340)](https://github.com/apache/pulsar/pull/23340))) Revert "\[fix] Key\_Shared mode consumption latency when low traffic

\[fix] Key\_Shared mode consumption latency when low traffic

\[fix]\[build] Fix problem where git.commit.id.abbrev is missing in image tagging

\[fix]\[test] Fix flaky test LeaderElectionTest.revalidateLeaderWithinSameSession

\[fix]\[broker] Fix incomplete NAR file extraction which prevents broker from starting

\[improve]\[broker] Register the broker to metadata store without version id compare

\[fix]\[broker] Fail fast if the extensible load manager failed to start

\[fix]\[io] Upgrade mssql server docker tag in DebeziumMsSqlContainer

### MoP

Support returning user subject with variables for AuthenticationProviderMTls

### KoP

Abstract the schema storage

### pulsarctl

fix: upgrade golang version to fix CVE

### StreamNative Pulsar Plugins

Update Go version for kube plugin for pulsarctl

### Function Mesh Worker Service

89291d7c Remove tune-runner-vm step

32e3aaa3 Use large runner for ci

Return image for kafka-connect when extend is set

Increase the default resource to 0.25 CU

### StreamNative Unified RBAC

feat(provider): support variable based user info for provider

## Security Fixes

### Apache Pulsar

\[fix]\[sec] Upgrade vertx to 4.5.10 to address CVE-2024-8391


# V3.3.1.8
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.1.8



# StreamNative Weekly Release Notes v3.3.1.8

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.3.1.8](https://github.com/streamnative/pulsar/releases/tag/v3.3.1.8)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.1.8/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.1.8/images/sha256-e1a5d2da3ddcdad33918172dfeccc58f808bac8d7e0989d42d3980c8a891544c)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.1.8/images/sha256-450bd1cb83a0c6a5e463fca26177c8c8cf2658c1a4e596eb0faedb573df3d93f)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.1.8/images/sha256-450bd1cb83a0c6a5e463fca26177c8c8cf2658c1a4e596eb0faedb573df3d93f)

## General Changes

### Apache Pulsar

\[fix] Bump commons-io:commons-io from 2.8.0 to 2.14.0

### KoP

Upgrade the apicurio dependency to 2.5.3.Final

### Cloud Storage Connector

Fix prootbuf CVE-2024-7254

### AWS Lambda Connector

Fix prootbuf CVE-2024-7254

### StreamNative Pulsar Plugins

Upgrade Avro from 1.11.3 to 1.11.4 for CVE-2024-47561

7ff1b589 Exclude dnsjava to avoid cve(#1920)

Fix prootbuf CVE-2024-7254

### Google BigQuery Sink Connector

Fix prootbuf CVE-2024-7254

### Aws EventBridge Connector

Fix prootbuf CVE-2024-7254

## Security Fixes

### Apache Pulsar

\[fix]\[sec] Upgrade Avro to 1.11.4 to address CVE-2024-47561

\[fix]\[sec]\[branch-3.3] Upgrade protobuf-java to 3.25.5


# V3.3.1.9
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.1.9



# StreamNative Weekly Release Notes v3.3.1.9

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.3.1.9](https://github.com/streamnative/pulsar/releases/tag/v3.3.1.9)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.1.9/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.1.9/images/sha256-ae0bcbf8d6a9b82d3fc6edf86b65e4ec5cac31ec951272627f0716fce94e5c0d)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.1.9/images/sha256-d47fad76454bfaa09aae2329aea75e7be4b76e1aa45064f5f9fb63a5184f4b4b)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.1.9/images/sha256-d47fad76454bfaa09aae2329aea75e7be4b76e1aa45064f5f9fb63a5184f4b4b)

## General Changes

### Apache Pulsar

\[fix]\[build] Remove duplicate dependencies in pom.xml

\[fix]\[broker] normalize path

\[improve]\[ci] Continue Pulsar CI build even when Trivy scanner fails

\[fix]\[broker] Avoid orphan ledgers in BucketDelayedDeliveryTracker

\[improve]\[client] Increase default Java client connectionMaxIdleSeconds to 60 seconds

\[fix]\[ci] Pin aquasecurity/trivy-action\@0.26.0 since master is broken

\[improve]\[build] Update maven-wrapper (mvnw) to recent stable version 3.3.2

\[improve]\[misc] Upgrade Jetty to 9.4.56.v20240826

\[fix]\[ml] Managed ledger should recover after open ledger failed

\[improve]\[broker] PIP-383: Support granting/revoking permissions for multiple topics

\[fix]\[broker] Fix AvgShedder strategy check

\[improve]\[build] Require Java 17 or Java 21 for building Pulsar

8fa42ca824 Bump version to next snapshot version

\[fix]\[broker] Fix out-of-order issues with ConsistentHashingStickyKeyConsumerSelector

\[fix]\[broker] Cancel possible pending replay read in cancelPendingRead

\[improve] Upgrade Pulsar Python client in docker image to 3.5.0

\[improve]\[ci] Switch to Java 21 as default JVM version for CI

\[improve]\[build] Use amazoncorretto:21-alpine image instead of apk installation

\[improve] Install openssl in the docker image to fix compatibility with Apache Pulsar Helm chart

\[fix]\[misc] Log Conscrypt security provider initialization warnings at debug level

\[fix]\[broker] Fix the broker registery cannot recover from the metadata node deletion

\[fix] \[log] Do not print error log if tenant/namespace does not exist when calling get topic metadata

### KoP

Disable metadata compaction for Ursa

### StreamNative Pulsar Plugins

Exclude dnsjava to avoid CVE

Exclude dnsjava to avoid CVE

### Function Mesh Worker Service

baf5d6c8 Make oxia state tests stable

validate function-mesh v0.22.0

Support pause rollout

fix kafka-connect owner reference convert

### StreamNative Tiered storage

Upgrade avro version to fix CVE

## Security Fixes

### Apache Pulsar

\[fix]\[sec] Drop hdfs2 support, Upgrade hadoop3 to 3.4.0 and dnsjava to 3.6.2 to address CVE-2024-25638


# V3.3.2.1
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.2.1



# StreamNative Weekly Release Notes v3.3.2.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.3.2.1](https://github.com/streamnative/pulsar/releases/tag/v3.3.2.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.2.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.2.1/images/sha256-2ba1260e8b1fa8c5b7ba09b5c17b98ae402a8d5454b79111f7f8758562b15d20)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.2.1/images/sha256-79550533f992f3f1f6019883256f3f0dddc67c27a506d321a3b9a0365f801487)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.2.1/images/sha256-79550533f992f3f1f6019883256f3f0dddc67c27a506d321a3b9a0365f801487)

## General Changes

### Apache Pulsar

\[fix]\[broker] Fix namespace unload might be blocked too long with extensible load manager

\[fix] \[broker] Topics failed to delete after remove cluster from replicated clusters set and caused OOM

\[fix]\[broker] timeout when broker registry hangs and monitor broker registry (ExtensibleLoadManagerImpl only)

\[fix]\[broker] Fix the broker registering might be blocked for long time

\[fix]\[client] Fix producer/consumer stop to reconnect or Pub/Sub due to IO thread race-condition

\[fix]\[test] Fix running ClusterMetadataSetupTest in IDE

\[fix]\[broker] Fix unloadNamespaceBundlesGracefully can be stuck with extensible load manager

\[fix]\[client] Use dedicated executor for requests in BinaryProtoLookupService

\[fix] \[proxy] Fix pattern consumer does not work when using Proxy

\[fix]\[test] Fix memory leak via OTel shutdown hooks in tests

\[improve]\[test] Added message properties tests for batch and non-batch messages

\[fix]\[client] Prevent embedding protobuf-java class files in pulsar-client-admin and pulsar-client-all

\[improve]\[io] Upgrade Spring version to 6.1.13 in IO Connectors

\[improve]\[broker] Add log to track issue when `handleGetTopicsOfNamespace`

\[fix]\[test] Address flaky GetPartitionMetadataMultiBrokerTest

\[fix]\[client] Fix the javadoc for ConsumerBuilder.isAckReceiptEnabled

\[fix]\[build] Add basic support for vscode-java and Eclipse IDE

\[fix]\[test] Fix flaky test ManagedLedgerTest.testDeleteCurrentLedgerWhenItIsClosed

\[fix]\[test] Fix flaky GetPartitionMetadataMultiBrokerTest.testCompatibilityDifferentBrokersForNonPersistentTopic

\[improve]\[broker] Make cluster metadata init command support metadata config path

### AoP

Fix the publish latency unit

### KoP

Re-implement the EventManager

### Google BigQuery Sink Connector

Remove unnecessary version define

Add table type validation logic

Add google partner header

Upgrade google lib version

\[feat] Support sync properties to biguqery

## Security Fixes


# V3.3.2.2
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.2.2



# StreamNative Weekly Release Notes v3.3.2.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.3.2.2](https://github.com/streamnative/pulsar/releases/tag/v3.3.2.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.2.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.2.2/images/sha256-d31025cbff351bc2e562eeb6f55cff36717a73bf5c52e21fd967ddefd4e53667)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.2.2/images/sha256-a480ebcd194194551d36316c4dbd5e5b4dca9fd1e35508a38d0ad58dc08294e4)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.2.2/images/sha256-a480ebcd194194551d36316c4dbd5e5b4dca9fd1e35508a38d0ad58dc08294e4)

## General Changes

### Apache Pulsar

\[fix] \[admin] Fix lookup get a null result if uses proxy

\[improve]\[io] Support update subscription position for sink connector

\[fix]\[broker] Increase readBuffer size for bookkeeper.DLOutputStream

\[improve]\[broker] Make cluster metadata teardown command support metadata config path

\[fix]\[client] Fix Reader.hasMessageAvailable return wrong value after seeking by timestamp with startMessageIdInclusive

\[fix]\[client] Fix ReaderBuilder doest not give illegalArgument on connection failure retry

\[fix] \[broker] Fix race-condition causing repeated delete topic

### KoP

Adopt a more efficient and reliable approach for compacted topic replay

### Function Mesh Worker Service

0b635d7d Use pulsarctl java runner in ci

### Google BigQuery Sink Connector

Support auto update table with pulsar system filed

If model is null will use NULLABLE

## Security Fixes


# V3.3.2.3
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.2.3



# StreamNative Weekly Release Notes v3.3.2.3

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.3.2.3](https://github.com/streamnative/pulsar/releases/tag/v3.3.2.3)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.2.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.2.3/images/sha256-2d5d6b7b0d7e4a04c5e2dcc66b76dd2cc4b5c4ce33c29dcbbceb623d243395c5)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.2.3/images/sha256-4d121b2c7b0a14d24b30e45daada80164ab6ffd1bd04faef1d550bf5ac295c00)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.2.3/images/sha256-4d121b2c7b0a14d24b30e45daada80164ab6ffd1bd04faef1d550bf5ac295c00)

## General Changes

### Apache Pulsar

\[fix]\[broker] Fix failed TokenAuthenticatedProducerConsumerTest

\[improve]\[offload] Use filesystemURI as the storage path

\[fix]\[misc] Unable to connect an etcd metastore with recent releases due to jetc-core sharding problem

Enabling DNS retryOnTimeout with TCP in DnsNameResolver

\[improve] \[broker] replace HashMap with inner implementation ConcurrentLongLongPairHashMap in Negative Ack Tracker.

\[fix]\[client] The partitionedProducer maxPendingMessages always is 0

\[improve]\[broker] Support cleanup `replication cluster` and `allowed cluster` when cluster metadata teardown

\[fix]\[broker] Broker is failing to create non-durable sub if topic is fenced

\[fix]\[client] fix the beforeConsume() method earlier hit with message listener

\[fix]\[test] Fix DeadLetterTopicTest.testDeadLetterTopicWithInitialSubscriptionAndMultiConsumers

\[fix]\[broker] Fix currently client retries until operation timeout if the topic does not exist

\[fix]\[test] Fix SimpleProducerConsumerTest.testMultiTopicsConsumerImplPauseForManualSubscription

\[fix]\[broker] fix logging with correct error message while loading the topic

\[improve]\[test] Disable OTel autoconfigured exporters in tests

\[fix]\[broker] Fix print cluster migration state response

\[fix]\[broker] Fix Broker migration NPE while broker tls url not configured

\[improve]\[broker] re-elect the channel owner if no channel owner is found

\[improve]\[broker] Exclude system topics from namespace level publish and dispatch rate limiting

\[improve]\[admin] Print error log if handle http response fails

\[fix]\[broker] Fix ownership loss

### MoP

367595c fix checkstyle

4cc93c3 fix branch-3.3

Fix the auth data is NPE error

Fix broker enable dedup cause client publish failed

Seperate proxy and broker a single module

Refactor MoP to prepare for split Proxy to seperate module

Support returning user subject with variables for AuthenticationProviderMTls

Fix mTls authorize bug

Fix mTLS authorization bug

Support mTLS authentication for MoP

Fix workflow

Fix authentication metrics for 4.0

Implement AuthenticationProviderMTls

Fix TLS initialization

Fix test for 3.4.0-SNAPSHOT

Upgrade Pulsar from 2.10.0 to 3.4.0

### KoP

Disable bundle ownership transferring for bundles in shadow namespaces

### Cloud Storage Connector

Upgrade netty to fix CVE-2024-47535

### Cloud Pulsar Plugins

Upgrade netty to fix CVE-2024-47535

### StreamNative Tiered storage

Configure ksn entryformat in test

Increase the kop test waiting time

### StreamNative Unified RBAC

fix: fix build script typo

fix: avoid spotless format pom

fix(sdk-go-cloud): return nil when not found role

feat: upgrade sdk-go to 0.1.7

feat: downgrade k8s client version to 0.24

feat: use CEL to instead CElExpression

feat(sdk-go-cloud): upgrade sdk-go version to v0.1.5

feat(sdk-go): introduce sdk-go-cloud for control plane components

feat(schema): use protobuf defined pojo for all the sdk.

feat(provider): validate superuser along with permission check

feat: support new permission clusterrole describe

feat(sdk-go): make data source interface more general

fix: fix wrong permission name

feat: support mock cloud image

feat: support variables for CEL expression

feat: support an error to indicate empty role

feat(endpoint): change apply endpoint success code to 200

feat(sdk): support apply for role and role binding

feat(metadata): add missing verbs

feat(authorizer): support CEL condition for role binding

feat: application superuser support

feat(generator): change APIGroups to APIVersion

fix(k8s-generator): fix wrong group key causes wrong result

Support cluster role

## Security Fixes

### Apache Pulsar

\[fix]\[sec] Upgrade to Netty 4.1.115.Final to address CVE-2024-47535

\[fix]\[sec] Upgrade Zookeeper to 3.9.3 to address CVE-2024-51504

\[fix]\[sec] Replace bcprov-jdk15on dependency with bcprov-jdk18-on


# V3.3.2.4
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.2.4



# StreamNative Weekly Release Notes v3.3.2.4

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.3.2.4](https://github.com/streamnative/pulsar/releases/tag/v3.3.2.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.2.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.2.4/images/sha256-6f861e1a61d25963c6daa76ffbe8ef45e92294c46b4ca4c697205c9677ddf1d7)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.2.4/images/sha256-1bd120bfe156ac05a1d4a5a3b09c9787092e654d273a70593932ca05bef8235c)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.2.4/images/sha256-1bd120bfe156ac05a1d4a5a3b09c9787092e654d273a70593932ca05bef8235c)

## General Changes

### Apache Pulsar

a7107f97b8 \[fix]\[test]\[branch-3.3] Fix OneWayReplicatorUsingGlobalZKTest#testRemoveCluster

\[improve]\[broker] Clear thread local BrokerEntryMetadata instance before reuse

\[fix]\[client] fix incomingMessageSize and client memory usage is negative

\[fix]\[fn] ack messages for window function when its result is null

\[improve] Improve logic for enabling Netty leak detection

\[improve]\[ml] Avoid repetitive nested lock for isMessageDeleted in ManagedCursorImpl

\[improve]\[broker] PIP-392: Add configuration to enable consistent hashing to select active consumer for partitioned topic

### MoP

ace5d00 Revert AuthenticationProviderMTls

### KoP

\[Ursa] Don't fail with OFFSET\_OUT\_OF\_RANGE when LEO is less than the fetch offset

### AWS Lambda Connector

Support include publish time to metadata

### StreamNative Pulsar Plugins

Upgrade aws sdk dependency version to v2

exlude netty for aws-jdk

### Function Mesh Worker Service

Set processingGuarantee for window functions

Bump function-mesh to v0.23.0

### Google BigQuery Sink Connector

Set partitionedTable and ClusterTables to false when disable auto create table

Remove verify logic for autoCreateTable

Improve auto update logic

## Security Fixes


# V3.3.2.5
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.2.5



# StreamNative Weekly Release Notes v3.3.2.5

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.3.2.5](https://github.com/streamnative/pulsar/releases/tag/v3.3.2.5)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.2.5/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.2.5/images/sha256-a470fa0aebc7f298febd7855fa2ed75c952c07829f5b63479b29cdd238ede143)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.2.5/images/sha256-936912dbcd44f63750c25cb7d23de558ecfabd99492b3b51ba2420adb4c4f5bc)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.2.5/images/sha256-936912dbcd44f63750c25cb7d23de558ecfabd99492b3b51ba2420adb4c4f5bc)

## General Changes

### Apache Pulsar

\[improve]\[admin] Opt-out of topic-existence check

\[fix]\[broker] Catch exception for entry payload interceptor processor

\[fix]\[cli] Fix set-retention with >2GB size value for topic policy

\[fix]\[cli] Fix set topic retention policy failed

\[fix] \[broker] Add consumer name for subscription stats

\[improve] Install coreutils in docker image to improve compatibility

\[fix]\[broker] Invoke custom BrokerInterceptor's `onFilter` method if it's defined

\[fix]\[broker] support missing cluster level fine-granted permissions

\[feat]\[broker] Implement allowBrokerOperationAsync in PulsarAuthorizationProvider to avoid exception thrown

\[fix]\[broker] support missing tenant level fine-granted permissions

\[fix]\[broker]: support missing broker level fine-granted permissions

f792b02794 Bump version to next snapshot version

\[improve]\[broker] Skip unloading when bundle throughput is zero (ExtensibleLoadManagerImpl only)

\[improve]\[client] Enhance error handling for non-exist subscription in consumer creation

\[fix]\[client] Fix race-condition causing doReconsumeLater to hang when creating retryLetterProducer has failed

\[improve]\[client] Reduce unshaded dependencies and shading warnings in shaded Java client modules

\[improve] Upgrade OpenTelemetry library to 1.44.1 version

\[improve]\[client] Replace NameUtil#generateRandomName with RandomStringUtils#randomAlphanumeric

\[fix]\[build] Fix error "Element encoding is not allowed here" in pom.xml

\[fix]\[client] Fix DLQ producer name conflicts when there are same name consumers

\[improve] Upgrade oxia-java to 0.4.10 and fix closing of OxiaMetadataStore

\[fix]\[client] Fix deadlock of NegativeAcksTracker

\[improve]\[broker] Decouple pulsar\_storage\_backlog\_age\_seconds metric with backlogQuota check

\[fix]\[client] Make protobuf-java dependency optional in java client libraries

\[improve] Use single buffer for metrics when noUnsafe use

\[fix]\[broker] fix null lookup result when brokers are starting

\[fix]\[client] Fixed an issue where a cert chain could not be used in TLS authentication

\[improve]\[misc] Disable OTel by default when running the pulsar-perf tool

\[cleanup]\[build] skip generating pom.xml.versionsBackup

\[fix]\[client] Initializing client-authentication using configured auth params

\[fix]\[misc] Class conflict during jetcd-core-shaded shading process

\[fix]\[ws] Implement missing http header data functions in AuthenticationDataSubscription

### KoP

Add pulsar-kafka-schema-registry jar to image

### Cloud Storage Connector

aeb2c91 code format

479f01b \[fix] fix wrong call to `bulkHandleFailedRecords`

Exist connector process when encounter exception

### StreamNative Pulsar Plugins

bump pulsar 3.3.2.5

Change authentication failed log level to warn

### Cloud Pulsar Plugins

Allow accepting token from query parameters

### Function Mesh Worker Service

Add `extraDependency` field to FunctionMeshConnectorDefinition

Support set pod annotations via CustomRuntimeOptions

### Google BigQuery Sink Connector

Optimize getGoogleCredentials exception

### StreamNative Tiered storage

iceberg catalog suit the polaris catalog.

Upgrade aws dependency to 2.x

### StreamNative Unified RBAC

feat(sdk-go-cloud): upgrade sdk-go to 0.1.16

feat(sdk-go): schema cel support

disable permission check to get better compatibility

feat(sdk-java): upgrade version to 1.3.0

upgrade sdk to 0.1.15

feat: support schema permissions

Add schema permissions

feat(sdk-go-cloud): support role cache to avoid io call

feat(sdk-java-pulsar): support missing computing component permissions

feat(sdk-go-cloud): upgrade sdk-go to version 0.1.14

fix(sdk-go): avoid filter the permissions for application role

feat(cv): support binding with cel

build(deps): bump nanoid from 3.3.7 to 3.3.8 in /sdk/sdk-js

fix(cv): fix the private repo visibility

feat(cv): support integration test

feat(sdk-go-cloud): upgrade sdk-go to 0.1.12

ci: fix CI to make sure test passed

fix(sdk-go-x): filter pulsar service admin permission

feat(sdk-apiserver): support sdk-apiserver

feat(sdk-java-pulsar): Implement broker rbac filter

feat(sdk-go-cloud): upgrade sdk

fix(ci): fix CI failed by wrong packet name

feat: upgrade the pulsar to snapshot repo

fix(sdk-js): fix the JSON format

feat(sdk-js): support permission cube

feat: support cluster,tenant,broker level permissions

upgrade the project version to 1.2.0

fix(sdk-js): upgrade the version to 0.0.4

refine the metadata specification

fix(sdk-java): fix failed integration test

feat(java): release 1.1.0

feat(pom): upgrade version to 1.1.0-snaphsot

Feat.improve.publish

feat(sdk-java): publish java to github packages

feat(sdk-java): improve the condition authorization interface

fix(sdk-go-cloud): fix wrong subject comparing

fix(sdk-go-cloud): fix undecoded service account subject

feat(metadata): support new permissions for cloud metrics

upgrade sdk js version

fix types import

feat: support CLI for unified rbac read endpoints

fix: fix proto decode issue

feat(sdk-js): support NewAuthorizerWithPrivilegesString

feat: upgrade sdk-go proto

feat(sdk-js): update proto definition

feat(sdk-go-cloud): upgrade sdk-go to 0.1.8

feat(sdk-go): upgrade schema

feat(sdk-js): support privileges authorizer

feat: make proto pojo json to camel case

fix: fix sdk-js name

feat: upgrade node version to 20

feat(doc): add document for unified-rbac

feat: support sdk-js

## Security Fixes

### Apache Pulsar

\[fix]\[sec] Bump commons-io version to 2.18.0


# V3.3.2.6
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.2.6



# StreamNative Weekly Release Notes v3.3.2.6

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.3.2.6](https://github.com/streamnative/pulsar/releases/tag/v3.3.2.6)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.2.6/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.2.6/images/sha256-e05d67951d57bf27bc72fbac553ab3a82ac3af045554a62ca4e1fa7420054356)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.2.6/images/sha256-96365f39f2d40550cfd114f60c977662669a0edb4107ab305324613bfe3d0c25)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.2.6/images/sha256-96365f39f2d40550cfd114f60c977662669a0edb4107ab305324613bfe3d0c25)

## General Changes

### Apache Pulsar

\[fix]\[ml] Topic load timeout due to ml data ledger future never finishes

\[fix]\[broker] System topic should not be migrated during blue-green cluster migration

\[fix]\[admin] Fix exception loss in getMessageId method

\[fix] Fix issues with Pulsar Alpine docker image stability: remove glibc-compat

\[fix]\[client] Fix enableRetry for consumers using legacy topic naming where cluster name is included

\[fix]\[client] Fix reader message filtering issue during blue-green cluster switch

\[fix]\[broker] Fix bug causing loss of migrated information when setting other localPolicies in namespace

\[Fix]\[client] Fix pending message not complete when closeAsync

\[improve]\[monitor] Upgrade OTel to 1.45.0

\[fix] \[client] Fix memory leak when publishing encountered a corner case error

\[improve]\[fn] Improve closing of producers in Pulsar Functions ProducerCache invalidation

\[improve]\[fn] Improve implementation for maxPendingAsyncRequests async concurrency limit when return type is `CompletableFuture<Void>`

\[improve] Upgrade lombok to 1.18.36

\[fix]\[common] TopicName: Throw IllegalArgumentException if localName is whitespace only

\[fix] \[broker] fix NPE when calculating a topic's backlogQuota

\[fix] \[broker] Fix config replicationStartAt does not work when set it to earliest

### AoP

remove useless check in tests

### KoP

Update pulsar consumer stats from ksn

### StreamNative Pulsar Plugins

Add the missing dependency for package-storage

### Function Mesh Worker Service

Replace the deprecated `getZooKeeperSessionTimeoutMillis`

## Security Fixes

### Apache Pulsar

\[fix]\[sec] Upgrade golang.org/x/crypto from 0.21.0 to 0.31.0 in pulsar-function-go

\[fix]\[sec] Upgrade async-http-client to 2.12.4 to address CVE-2024-53990

\[fix]\[sec] Mitigate CVE-2024-53990 by disabling AsyncHttpClient CookieStore


# V3.3.2.7
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.2.7



# StreamNative Weekly Release Notes v3.3.2.7

## Download

### Distributions

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.2.7/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.2.7/images/sha256-3d0929fbcec9d1e5a97852a231d54cde34f6f43f26d7a981dfb1f7d4467289fc)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.2.7/images/sha256-8773edc70eda20d5d881b02d9c27e301206722d57e6afb2e8b25a88de6d861dd)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.2.7/images/sha256-8773edc70eda20d5d881b02d9c27e301206722d57e6afb2e8b25a88de6d861dd)

## General Changes

### Apache Pulsar

([#23813](https://github.com/apache/pulsar/pull/23813)) \[improve] Upgrade to Netty 4.1.116.Final and io\_uring to 0.0.26.Final

([#23784](https://github.com/apache/pulsar/pull/23784)) \[fix]\[admin] Fix exception thrown in getMessageId method

([#23666](https://github.com/apache/pulsar/pull/23666)) \[fix]\[test]: Flaky-test: GetPartitionMetadataMultiBrokerTest.testCompatibilityDifferentBrokersForNonPersistentTopic

([#23802](https://github.com/apache/pulsar/pull/23802)) \[fix] \[broker] Fix items in dispatcher.recentlyJoinedConsumers are out-of-order, which may cause a delivery stuck

([#23795](https://github.com/apache/pulsar/pull/23795)) Msg delivery is stuck due to items in the collection recentlyJoinedConsumers are out-of-order

([#23615](https://github.com/apache/pulsar/pull/23615)) \[fix]\[broker] Skip to persist cursor info if it failed by cursor closed

([#23791](https://github.com/apache/pulsar/pull/23791)) \[fix]\[client] Cannot access message data inside ProducerInterceptor#onSendAcknowledgement

([#23718](https://github.com/apache/pulsar/pull/23718)) \[fix]\[client] Make DeadLetterPolicy & KeySharedPolicy serializable

([#23797](https://github.com/apache/pulsar/pull/23797)) \[fix]\[broker] Continue using the next provider for authentication if one fails

([#23781](https://github.com/apache/pulsar/pull/23781)) \[fix]\[broker] Fix enableReplicatedSubscriptions

([#23757](https://github.com/apache/pulsar/pull/23757)) \[improve]\[client] Make replicateSubscriptionState nullable

### MoP

([#1590](https://github.com/streamnative/mop/pull/1590)) Remove yahoo dependency

### KoP

Add a simple partition index based load balancer implementation

### AMQP1\_0 Connector

([#1092](https://github.com/streamnative/pulsar-io-amqp-1-0/pull/1092)) Use new clean disk job

### StreamNative Pulsar Plugins

Revert "Azure Blob Storage backed Package Management Service

Azure Blob Storage backed Package Management Service

### Cloud Pulsar Plugins

Use new clean disk job

### Function Mesh Worker Service

3684aa96 Fix ci

### StreamNative Unified RBAC

fix: fix the license of generator

feat: support new permission `pulsar.packages.admin`

feat: format generator license

feat: improve the document

Add User, ClusterRole, RoleBinding tests

fix(sdk-java): fix empty cel passed authorization

feat(sdk-go-cloud): upgrade the sdk-go to v0.2.0

fix(\*): fix license

fix(\*): fix license issue

feat(\*): upgrade the project version to 1.4.0

feat(sdk-js): upgrade version to 0.0.8

all: basic SRN supprot.

Add application tests

Optimize integration tests

feat(sdk-js): resource admin support

Add more integration tests

## Security Fixes


# V3.3.5.10
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.5.10



# StreamNative Weekly Release Notes v3.3.5.10

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.3.5.10](https://github.com/streamnative/pulsar/releases/tag/v3.3.5.10)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.5.10/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.5.10/images/sha256-9bf7a5e1df47bfe5b589b0496a8f1899ee82aaee5b8e55691083c0e6422efd85)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.5.10/images/sha256-8ddd5e1c15dbd5d6a73849882acdc2d02e8d4826e8233aa29956aaece69b07ed)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.5.10/images/sha256-8ddd5e1c15dbd5d6a73849882acdc2d02e8d4826e8233aa29956aaece69b07ed)

## General Changes

### Apache Pulsar

([#24512](https://github.com/apache/pulsar/pull/24512)) \[fix]\[broker] Fix NPE when getting delayed delivery policy

([#24453](https://github.com/apache/pulsar/pull/24453)) \[fix]\[broker] replication does not work due to the mixed and repetitive sending of user messages and replication markers

([#24424](https://github.com/apache/pulsar/pull/24424)) \[fix]\[broker] Fix the non-persistenttopic's replicator always get error "Producer send queue is full" if set a small value of the config replicationProducerQueueSize

([#24189](https://github.com/apache/pulsar/pull/24189)) \[fix]\[broker]excessive replication speed leads to error: Producer send queue is full

([#23213](https://github.com/apache/pulsar/pull/23213)) \[improve] \[broker] Part 2 of PIP-370: add metrics "pulsar\_replication\_disconnected\_count"

([#23169](https://github.com/apache/pulsar/pull/23169)) \[improve] \[broker] Phase 1 of PIP-370 support disable create topics on remote cluster through replication

([#22674](https://github.com/apache/pulsar/pull/22674)) \[Fix]\[broker] Limit replication rate based on bytes

([#20931](https://github.com/apache/pulsar/pull/20931)) \[fix]\[broker] Fix ack hole in cursor for geo-replication

([#24443](https://github.com/apache/pulsar/pull/24443)) \[fix]\[txn] Fix negative unacknowledged messages in transactions by ensuring that the batch size is added into CommandAck

([#24421](https://github.com/apache/pulsar/pull/24421)) \[fix]\[build] Add missing `<name>` to submodules

### KoP

Increase timeout for testTwoTopicsGroupState to improve reliability

### StreamNative Tiered storage

Unity catalog support update table schema.

Fix pulsar offload format npe issue.

## Security Fixes


# V3.3.5.11
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.5.11



# StreamNative Weekly Release Notes v3.3.5.11

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.3.5.11](https://github.com/streamnative/pulsar/releases/tag/v3.3.5.11)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.5.11/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.5.11/images/sha256-8468c727e959830d2624419bb49825b9c6296cfab5695885928e645417fd9a84)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.5.11/images/sha256-47f0c66c0377e626b7fe91799c7a5412adae621d324cc7254b67509ff757632a)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.5.11/images/sha256-47f0c66c0377e626b7fe91799c7a5412adae621d324cc7254b67509ff757632a)

## General Changes

### Apache Pulsar

([#24542)](https://github.com/apache/pulsar/pull/24542))) Revert "\[fix]\[broker]Fix thread safety issues in BucketDelayedDeliveryTracker with StampedLock optimistic reads

([#24554](https://github.com/apache/pulsar/pull/24554)) ([#24571](https://github.com/apache/pulsar/pull/24571)) \[fix]\[client]\[branch-4.0] Partitioned topics are unexpectedly created by client after deletion

([#24576](https://github.com/apache/pulsar/pull/24576)) \[fix]\[test] fix flaky GrowableArrayBlockingQueueTest.testPollBlockingThreadsTermination

([#24569](https://github.com/apache/pulsar/pull/24569)) \[fix]\[broker] Fix ManagedCursor state management race conditions and lifecycle issues

([#24550](https://github.com/apache/pulsar/pull/24550)) \[improve]\[client] Terminate consumer.receive() when consumer is closed

([#24560](https://github.com/apache/pulsar/pull/24560)) \[fix]\[broker] Fix maxTopicsPerNamespace might report a false failure

([#24505](https://github.com/apache/pulsar/pull/24505)) \[fix]\[test]fix flaky test BrokerServiceAutoTopicCreationTest.testDynamicConfigurationTopicAutoCreationPartitioned

([#24472](https://github.com/apache/pulsar/pull/24472)) \[fix] Prevent IllegalStateException: Field 'message' is not set

([#24542](https://github.com/apache/pulsar/pull/24542)) \[fix]\[broker]Fix thread safety issues in BucketDelayedDeliveryTracker with StampedLock optimistic reads

([#24551](https://github.com/apache/pulsar/pull/24551)) \[fix]\[broker] Fix Broker OOM due to too many waiting cursors and reuse a recycled OpReadEntry incorrectly

([#24511](https://github.com/apache/pulsar/pull/24511)) \[fix]\[broker] Fix deduplication replay might never complete for exceptions

([#24557](https://github.com/apache/pulsar/pull/24557)) \[fix]\[broker]\[branch-3.3] Disable broken ExtensibleLoadManager tests and add closeInternalTopics in follower monitor

([#24522](https://github.com/apache/pulsar/pull/24522)) \[fix]\[ml] Fix the possibility of message loss or disorder when ML PayloadProcessor processing fails

([#24552](https://github.com/apache/pulsar/pull/24552)) \[improve]\[test] Remove EntryCacheCreator from ManagedLedgerFactoryImpl

([#24544](https://github.com/apache/pulsar/pull/24544)) \[improve] Upgrade pulsar-client-python to 3.8.0 in Docker image

([#24516](https://github.com/apache/pulsar/pull/24516)) \[fix]\[broker] Fix exclusive producer creation when last shared producer closes

([#24506](https://github.com/apache/pulsar/pull/24506)) \[fix]\[broker] Fix duplicate increment of ADD\_OP\_COUNT\_UPDATER in OpAddEntry

([#24543](https://github.com/apache/pulsar/pull/24543)) \[fix]\[broker] Fix matching of topicsPattern for topic names which contain non-ascii characters

([#24539](https://github.com/apache/pulsar/pull/24539)) \[fix]\[client] Close orphan producer or consumer when the creation is interrupted

([#24517](https://github.com/apache/pulsar/pull/24517)) \[fix]\[client] Fix ClientCnx handleSendError NPE

([#24515](https://github.com/apache/pulsar/pull/24515)) \[fix]\[ml] Fix asyncReadEntries might never complete if empty entries are read from BK

([#24530](https://github.com/apache/pulsar/pull/24530)) \[improve]\[misc] Upgrade RE2/J to 1.8

([#24518](https://github.com/apache/pulsar/pull/24518)) \[fix]\[broker] Fix wrong backlog age metrics when the mark delete position point to a deleted ledger

([#24468](https://github.com/apache/pulsar/pull/24468)) \[improve]\[broker] Upgrade bookkeeper to 4.17.2/commons-configuration to 2.x/grpc to 1.72.0 and enable ZooKeeper client to establish connection in read-only mode

([#24473](https://github.com/apache/pulsar/pull/24473)) \[improve]\[build] replace org.apache.commons.lang to org.apache.commons.lang3

([#24525](https://github.com/apache/pulsar/pull/24525)) \[improve]\[misc] Optimize topic list hashing so that potentially large String allocation is avoided

([#24528](https://github.com/apache/pulsar/pull/24528)) \[fix]\[client] Fix issue in auto releasing of idle connection with topics pattern consumer

([#24529](https://github.com/apache/pulsar/pull/24529)) \[fix]\[proxy] Fix default value of connectionMaxIdleSeconds in Pulsar Proxy

([#24476](https://github.com/apache/pulsar/pull/24476)) \[fix]\[client] NPE in MultiTopicsConsumerImpl.negativeAcknowledge

([#24465](https://github.com/apache/pulsar/pull/24465)) \[fix]\[proxy] Fix proxy OOM by replacing TopicName with a simple conversion method

([#24434](https://github.com/apache/pulsar/pull/24434)) \[improve]\[broker] Improve the log when namespace bundle is not available

### MoP

Fix MQTT message error handling and improve connection responses

### KoP

Do not set setReplicationClusters on createTopicIfNotExist

### StreamNative Pulsar Plugins

fix export duplicated JVM metrics on AuditLogMetrics

Upgrade delta kernel to 4.0.0

### Cloud Pulsar Plugins

Change to use commons-lang3

Only append jwk when kty is rsa

### Function Mesh Worker Service

0a27de1b Update version when release

Implement agent function

Set minReplicas to parallelism when HPA is enabled

### StreamNative Tiered storage

Fix build failure

### StreamNative Unified RBAC

Use `GET_BUNDLE` operation to check the "get" permission for namespace

## Security Fixes

### Apache Pulsar

([#24562](https://github.com/apache/pulsar/pull/24562)) \[fix]\[sec] Remove dependency on out-dated commons-configuration 1.x

([#24564](https://github.com/apache/pulsar/pull/24564)) \[fix]\[sec] Upgrade Kafka connector and clients version to 3.9.1 to address CVE-2025-27818

([#24547](https://github.com/apache/pulsar/pull/24547)) \[fix]\[sec] Upgrade pulsar-function-go dependencies to address CVE-2025-22868


# V3.3.5.12
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.5.12



## StreamNative Weekly Release Notes v3.3.5.12

#### General Changes

### Apache Pulsar

([#24626](https://github.com/apache/pulsar/pull/24626)) \[fix]\[proxy] Fix TooLongFrameException with Pulsar Proxy

([#24621](https://github.com/apache/pulsar/pull/24621)) \[fix]\[broker] Fix duplicate watcher registration after SessionReestablished

([#24610](https://github.com/apache/pulsar/pull/24610)) \[fix]\[client]Prevent ZeroQueueConsumer from receiving batch messages when using MessagePayloadProcessor

([#24606](https://github.com/apache/pulsar/pull/24606)) \[improve]\[broker]Remove block calling that named cursor.asyncGetNth when expiring messages

([#24604](https://github.com/apache/pulsar/pull/24604)) \[improve]\[io] Add dependency file name information to error message when .nar file validation fails with ZipException

([#24601](https://github.com/apache/pulsar/pull/24601)) \[improve]\[doc] Improve the JavaDocs of sendAsync to avoid improper use

([#24599](https://github.com/apache/pulsar/pull/24599)) \[fix]\[client] Retry for unknown exceptions when creating a producer or consumer

([#24450](https://github.com/apache/pulsar/pull/24450)) \[fix]\[broker] Fix REST API to produce messages to single-partitioned topics

([#24595](https://github.com/apache/pulsar/pull/24595)) \[fix]\[ci] Fix code coverage metrics in Pulsar CI

([#24582](https://github.com/apache/pulsar/pull/24582)) \[improve]\[client] Support load RSA PKCS#8 private key

([#24535](https://github.com/apache/pulsar/pull/24535)) \[improve]\[test] Add test for dead letter topic with max unacked messages blocking

([#24532](https://github.com/apache/pulsar/pull/24532)) \[fix]\[misc] Upgrade dependencies to fix critical security vulnerabilities

([#24514](https://github.com/apache/pulsar/pull/24514)) \[improve]\[build] Bump org.apache.commons:commons-lang3 from 3.17.0 to 3.18.0

([#24586](https://github.com/apache/pulsar/pull/24586)) \[improve]\[test] Refactor the way way pulsar-io-debezium-oracle nar file is patched when building the test image

([#24590](https://github.com/apache/pulsar/pull/24590)) \[fix]\[broker] Fix flaky testReplicatorsInflightTaskListIsEmptyAfterReplicationFinished

### KoP

451a59b88 \[branch-3.3] Bump version to 3.3.5.12

Fix possible deadlock of system topic access due to blocking call when holding the lock

Fix incorrect ListOffsets result on a compacted topic

### Function Mesh Worker Service

747a4e68 Update MeshWorkerServer

bb788d1e Fix version

Support set agent tools config

Make MeshWorker able to run standalone and load additional servlets

Support load ConnectorCatalog using label

Update error msg in status

### StreamNative Tiered storage

Run tests

### StreamNative Unified RBAC

fix(misc): add some logs and missing output

Update package.json

feat: upgrade sdk-go to 0.13

feat: support Acls on Pulsar

feat: support `allPartition` macros and fix a compatibility issue

feat: upgrade dependencies

feat: support ACL specification

feat(sdk-go-cloud): upgrade the sdk-go to 0.11.0

\[feat] Extend functions\&connectors permissions

feat(sdk-java): add some debugging logs

Update package.json

fix(wasm): fix csp

fix(sdk-go): fix compatibility issue as #466

fix: upgrade dependencies && fix project CI

fix(sdk-java): compatible with legacy request condition

Update package.json

feat(wasm): refine result code

fix(wasm): fix compile error

Update package.json

fix(wasm): remove debugging logs

feat(wasm): standardise API

Update package.json

feat(wasm): upgrade the dependencies to avoid wasm panic

feat(sdk-go): move k8s related dependencies to cloud

feat: move document to wiki

Update package.json

Update ci-publish-js.yaml

feat(sdk-js): refine the public API

fixes(sdk-js): fixes the missing dependency protobuf

fix(sdk-js): fix CI workflow 2

fix: fix the CI golang building

feat: support sdk-js based on wasm

fix(test): fix testing docker image build script

fix(sdk-java): fix empty condition missing SRN validation

feat: upgrade sdk-go version to 0.9.0

fix: avoid empty cel mess up authorization logic

feat: upgrade dependencies

fix(sdk-go): fix the empty condition ignore SRN

Use `GET_BUNDLE` operation to check the "get" permission for namespace

feat(sdk-go-\*): upgrade dependencies

fix(sdk-go): support authorizer missing method

feat: downgrade golang version

feat(sdk-go-cloud): upgrade dependencies

feat: support macros on condition

feat: update dependencies

feat: support resource name based conditions

Update ci: service-accout-admin -> account-admin

feat(ci): Notify in slack when CI failed

feat(sdk-go-cloud): upgrade sdk-go to 0.7.2

Add Kafka topics delete permission

feat(sdk-go-cloud): upgrade sdk-go to 0.7.1

feat(sdk-go): kafka cel support

Update test cluster location

feat(sdk-go-cloud): upgrade sdk-go to 0.7.0

Reduce rbac filter timeout to 3 seconds by default and make it configurable

Fix kafka permission format

Add permission format rule

Use RoleBinding name to save bindings in metadata

Add KSN permissions

Add cv tests for service-account-admin and secret-viewer

Upgrade project and sdk/java to 1.6.0

Add kafka condition

Support batch apply role bindings

Run CV tests in parallel mode

Create test clusters dynamically for CI

feat(sdk-go-cloud): upgrade sdk-go to 0.6.0

Add ServiceAccount, ApiKey and Secret to SRN

Use Github package wildcard repository url

Change the cluster used in integration test

Optimize logic of list filter interceptor

fix: remove annoying logs

Do not create temp cloud resources in CI

Add lack sdk jar package and fix integration test

Add more cloud tests

Use SN bom

Add packages, produce, consume tests

feat(sdk-go-cloud): upgrade sdk-go to 0.5.0

feat(metadata): support cloud resource volumes and catalogs

Add more cloud tests

feat(sdk-go-cloud): upgrade the sdk-go to 0.4.1

feat(metadata): generate metadata for all sdk

feat:  `GetPrivileges` support for sdk-go-cloud

feat(sdk-go): support pivileges validation

feat: introduce new permission `cloud.selfOrganizations.alter`

feat(sdk-js): upgrade sdk to 0.0.9 to support new permission

feat: upgrade project and sdk-java, sdk-pulsar to 1.5.0

feat(sdk-go-cloud): upgrade sdk-go to v0.3.0

Skip rbac reconcile for PulsarInstances create in CI

Add Pool/PoolOption/PoolMember tests


# V3.3.5.13
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.5.13



## StreamNative Weekly Release Notes v3.3.5.13

#### General Changes

### Apache Pulsar

([#24633](https://github.com/apache/pulsar/pull/24633)) \[fix]\[test]fix flaky ZeroQueueSizeTest.testZeroQueueGetExceptionWhenReceiveBatchMessage

([#24632](https://github.com/apache/pulsar/pull/24632)) \[fix]\[test] Fix ConcurrentModificationException in Ipv4Proxy

([#24630](https://github.com/apache/pulsar/pull/24630)) \[fix]\[test]fix flaky ZeroQueueSizeTest.testZeroQueueGetExceptionWhenReceiveBatchMessage

### MoP

23da6ad0 upgrade depenedecy

### KoP

Fix Kafka Connect's topic replay loop might be stuck when all messages have been compacted out

### Function Mesh Worker Service

be141f7f Cleanup disk

### StreamNative Unified RBAC

fix(acl): avoid parsing token from data source


# V3.3.5.14
Source: https://docs.streamnative.io/release-notes/pulsar/v3.3/v3.3.5.14



# StreamNative Weekly Release Notes v3.3.5.14

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.3.5.14](https://github.com/streamnative/pulsar/releases/tag/v3.3.5.14)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.3.5.14/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.3.5.14/images/sha256-ed31a9bc852e575519a5a84f75c35eb79b32a2476cea4fb61ba0f07f58f1176f)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.3.5.14/images/sha256-e5988b48ba49db04cb7f8a5bd4786c07540a0b0256fe7da7c60ccc99c4e29bcd)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.3.5.14/images/sha256-e5988b48ba49db04cb7f8a5bd4786c07540a0b0256fe7da7c60ccc99c4e29bcd)

## General Changes

### Apache Pulsar

([#24741](https://github.com/apache/pulsar/pull/24741)) \[fix]\[broker] Prevent unexpected recycle failure in dispatcher's read callback

([#24752](https://github.com/apache/pulsar/pull/24752)) \[fix]\[client] rollback TopicListWatcher retry behavior

([#24698](https://github.com/apache/pulsar/pull/24698)) \[fix]\[client]TopicListWatcher not closed when calling PatternMultiTopicsConsumerImpl.closeAsync() method

([#24634](https://github.com/apache/pulsar/pull/24634)) Dispatcher did unnecessary sort for recentlyJoinedConsumers and printed noisy error logs

([#24730](https://github.com/apache/pulsar/pull/24730)) \[fix]\[broker] Ensure KeyShared sticky mode consumer respects assigned ranges

([#24743](https://github.com/apache/pulsar/pull/24743)) \[fix]\[client] Fix receiver queue auto-scale without memory limit

([#24742](https://github.com/apache/pulsar/pull/24742)) \[improve]\[build] Upgrade Apache Parent POM to version 35

([#24731](https://github.com/apache/pulsar/pull/24731)) \[fix]\[broker] Fix cannot shutdown broker gracefully by admin api

([#24654](https://github.com/apache/pulsar/pull/24654)) \[fix]\[io] Improve Kafka Connect source offset flushing logic

([#24725](https://github.com/apache/pulsar/pull/24725)) \[fix]\[client] Avoid recycling the same ConcurrentBitSetRecyclable among different threads

([#24721](https://github.com/apache/pulsar/pull/24721)) \[feat]\[fn] Fallback to using `STATE_STORAGE_SERVICE_URL` in `PulsarMetadataStateStoreProviderImpl.init`

([#24719](https://github.com/apache/pulsar/pull/24719)) \[fix]\[broker] Fix memory leak when metrics are updated in a thread other than FastThreadLocalThread

([#24594](https://github.com/apache/pulsar/pull/24594)) \[improve]\[build] Disable javadoc build failure

([#23336](https://github.com/apache/pulsar/pull/23336)) \[fix]\[client] Fix ArrayIndexOutOfBoundsException when using SameAuthParamsLookupAutoClusterFailover

([#24665](https://github.com/apache/pulsar/pull/24665)) \[fix]\[meta] Use `getChildrenFromStore` to read children data to avoid lost data

([#23977](https://github.com/apache/pulsar/pull/23977)) \[fix]\[broker] Invalid regex in PulsarLedgerManager causes zk data notification to be ignored

([#24663](https://github.com/apache/pulsar/pull/24663)) \[fix]\[client] Skip schema validation when sending messages to DLQ to avoid infinite loop when schema validation fails on an incoming message

([#24669](https://github.com/apache/pulsar/pull/24669)) \[improve]\[io] Support specifying Kinesis KPL native binary path with 1.0 version specific path

([#24668](https://github.com/apache/pulsar/pull/24668)) \[improve]\[build] Use org.apache.nifi:nifi-nar-maven-plugin:2.1.0 with skipDocGeneration=true

([#24661](https://github.com/apache/pulsar/pull/24661)) \[improve]\[io] Upgrade AWS SDK v1 & v2, Kinesis KPL and KPC versions

([#24666](https://github.com/apache/pulsar/pull/24666)) \[improve]\[build] Increase maven resolver's sync context timeout

([#24662](https://github.com/apache/pulsar/pull/24662)) \[fix]\[client] fix ArrayIndexOutOfBoundsException in SameAuthParamsLookupAutoClusterFailover

([#24639](https://github.com/apache/pulsar/pull/24639)) \[fix]\[broker] Fix race condition in MetadataStoreCacheLoader causing inconsistent availableBroker list caching

([#24649](https://github.com/apache/pulsar/pull/24649)) \[fix]\[offload] Exclude unnecessary dependencies from tiered storage provider / offloader nar files

([#24643](https://github.com/apache/pulsar/pull/24643)) \[fix]\[broker] Add double-check for non-durable cursor creation

### KoP

6743e673b Bump Pulsar version to 3.3.5.14

Use new CompactedTopicUtils.asyncReadCompactedEntries API from apache/pulsar#24725

\[branch-3.3] Add latency metrics for ManagedLedgerImpl#internalAsyncAddEntry

Add more info to logs when the connection is closed

Handle unexpected exception in decode for safe producer state recovery

Avoid blocking when the previous consumer closed without sending SyncGroup requests

### StreamNative Pulsar Plugins

\[detector] Print more info when a corrupted value is received by Kafka consumer

\[detector] Enable idempotence for Kafka detector

Update x/net and go 1.24

### pulsarctl

Preserve tenant fields on partial update

upgrade client go version to 0.16.0

Update stable version

Upgrade go to 1.24.6 to fix CVE-2025-47907

### Function Mesh Worker Service

feat: support multiple mcp servers

Support trigger agent function with properties

Update function-mesh version to v0.25.0 in pom.xml

Remove ConnectRestException from mesh-worker-common module

Support input-type-class and output-type-class arguments for Functions

Support set extra env for kafka connect

Support streamable http for AgentFunction and make trigger timeout value configurable

Create a new sub module mesh-worker-common

### StreamNative Tiered storage

\[Branch 3.3] Cherry pick 1102

### StreamNative Unified RBAC

feat: treat NotFound exception as success for deleting

feat: upgrade sdk-go version to v0.14.0

feat: support extract variable claim from token

## Security Fixes

### Apache Pulsar

([#24717](https://github.com/apache/pulsar/pull/24717)) \[fix]\[sec] Upgrade Netty to 4.1.127.Final to address CVEs

([#24650](https://github.com/apache/pulsar/pull/24650)) \[fix]\[sec] Upgrade bouncycastle bcpkix-fips version to 1.79 to address CVE-2025-8916


# V4.0.0.1
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.0.1



# StreamNative Weekly Release Notes v4.0.0.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.0.1](https://github.com/streamnative/pulsar/releases/tag/v4.0.0.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.0.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.0.1/images/sha256-6afa0ecd618c99eccb454bc9d4c8bff545041dff48ede8ca3a77d5a3453ac5f5)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.0.1/images/sha256-b24a1a362f4342a9a1af939b8a13e38e8afafd1584387b4834af3a96a92a2092)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.0.1/images/sha256-b24a1a362f4342a9a1af939b8a13e38e8afafd1584387b4834af3a96a92a2092)

## General Changes

### Apache Pulsar

\[improve]\[io] Upgrade Spring version to 6.1.14 in IO Connectors

\[improve]\[io] Upgrade Spring version to 6.1.13 in IO Connectors

### KoP

Refactor the transaction implementation to make it align with Kafka

### StreamNative Ursa storage

Parquet read perfomance improvement

Delay metadata update task for managedLedger

Introduce compact cordinator.

Fix flaky test `testReadAfterTrimmed`

Decrease s3OpsRateLimitPerSecond to 100

Fix delete topic failed when not load

Fix compaction otel paramter

## Security Fixes


# V4.0.0.10
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.0.10



# StreamNative Weekly Release Notes v4.0.0.10

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.0.10](https://github.com/streamnative/pulsar/releases/tag/v4.0.0.10)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.0.10/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.0.10/images/sha256-01201029716997770e88824f26e738aaa68ff5c5d770d0316f8ebcc7ae1d4378)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.0.10/images/sha256-765e6109bb45d551b6991140505792d5955d8dc22ee854b63f6db9b70765394e)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.0.10/images/sha256-765e6109bb45d551b6991140505792d5955d8dc22ee854b63f6db9b70765394e)

## General Changes

### Apache Pulsar

\[fix]\[broker] Avoid block markDeletePosition forward when skip lost entries

\[improve] \[test] Add more test for the case that client receives a SendError, which relates to the PR #23038

\[fix]\[doc] Refine ClientBuilder#memoryLimit and ConsumerBuilder#autoScaledReceiverQueueSizeEnabled javadoc

\[fix]\[client] Fix wrong start message id when it's a chunked message id

\[fix] \[broker] fix NPE when calculating a topic's backlogQuota

\[improve]\[client] Print consumer stats log if prefetched messages are not zero

\[fix]\[broker] Fix the retry mechanism in `MetadataCache#readModifyUpdateOrCreate`

\[fix] \[broker] Fix config replicationStartAt does not work when set it to earliest

\[improve]\[admin] Opt-out of topic-existence check

\[fix]\[admin] Listen partitioned topic creation event

\[fix]\[broker] Catch exception for entry payload interceptor processor

### KoP

Handle schema registry authorization compatibility issue

### Cloud Storage Connector

Exist connector process when encounter exception

### StreamNative Pulsar Plugins

Add pulsar-oxia-state-store jar package

Just add non fat jar to image

Use SN bom

### Cloud Pulsar Plugins

Use SN bom

### Function Mesh Worker Service

Add `extraDependency` field to FunctionMeshConnectorDefinition

### Google BigQuery Sink Connector

Optimize getGoogleCredentials exception

### Aws EventBridge Connector

Remove require annotation for assessKey and secretKey

### StreamNative Unified RBAC

feat(sdk-go-cloud): upgrade sdk-go to 0.1.16

feat(sdk-go): schema cel support

disable permission check to get better compatibility

feat(sdk-java): upgrade version to 1.3.0

upgrade sdk to 0.1.15

feat: support schema permissions

Add schema permissions

feat(sdk-go-cloud): support role cache to avoid io call

feat(sdk-java-pulsar): support missing computing component permissions

feat(sdk-go-cloud): upgrade sdk-go to version 0.1.14

fix(sdk-go): avoid filter the permissions for application role

feat(cv): support binding with cel

build(deps): bump nanoid from 3.3.7 to 3.3.8 in /sdk/sdk-js

fix(cv): fix the private repo visibility

feat(cv): support integration test

feat(sdk-go-cloud): upgrade sdk-go to 0.1.12

ci: fix CI to make sure test passed

fix(sdk-go-x): filter pulsar service admin permission

feat(sdk-apiserver): support sdk-apiserver

feat(sdk-java-pulsar): Implement broker rbac filter

feat(sdk-go-cloud): upgrade sdk

### StreamNative Ursa storage

Add core and lakehouse jars to image

Support unity catalog.

Add iceberg support

Remove jar with dependencies when release

Switch to sn-bom pom dependency

\[cleanup] Remove the duplicated code

Separate the integration tests and ut in workflows

Enable all the primitive types tests

Limit the read request from compaction to the storage api

Get the compact task in the start offset order

## Security Fixes

### Apache Pulsar

\[fix]\[sec] Upgrade async-http-client to 2.12.4 to address CVE-2024-53990

\[fix]\[sec] Mitigate CVE-2024-53990 by disabling AsyncHttpClient CookieStore


# V4.0.0.2
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.0.2



# StreamNative Weekly Release Notes v4.0.0.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.0.2](https://github.com/streamnative/pulsar/releases/tag/v4.0.0.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.0.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.0.2/images/sha256-fdb2ea6ca8def121d091f7e430be3b8cbaf746dd745d7783d05603249821eca6)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.0.2/images/sha256-8423d57d2afc0bf9aa5bc75ad2f3afdd231ed3ba35a1b73d52cc1c7221cafb3a)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.0.2/images/sha256-8423d57d2afc0bf9aa5bc75ad2f3afdd231ed3ba35a1b73d52cc1c7221cafb3a)

## General Changes

### Apache Pulsar

\[feat]\[meta] Bump oxia java version from 0.4.5 to 0.4.7

\[fix]\[client] Fix producer/consumer stop to reconnect or Pub/Sub due to IO thread race-condition

\[fix]\[broker] Key\_Shared subscription: Reject consumers with incompatible policy

\[fix]\[test] Fix running ClusterMetadataSetupTest in IDE

\[fix] \[proxy] Fix pattern consumer does not work when using Proxy

\[fix]\[test] Prevent OOM in test by not spying invocations in SimpleProducerConsumerTest

\[improve]\[monitor] Upgrade OTel to 1.41.0

\[feat]\[monitor] Add offloader stats grafana dashboard

\[fix]\[client] Prevent embedding protobuf-java class files in pulsar-client-admin and pulsar-client-all

\[fix]\[client] Fix ConsumerStats.getRateMsgsReceived javadoc

\[fix]\[client] Use dedicated executor for requests in BinaryProtoLookupService

8c20e64651 Bump version to 4.0.1-SNAPSHOT

### KoP

Remove all coordinator epoch usages

### Cloud Storage Connector

feat: Support new batch model BLEND/PARTITIONED

### Google BigQuery Sink Connector

Upgrade google lib version

\[feat] Support sync properties to biguqery

### StreamNative Ursa storage

Fix the flaky test testOperationRejection

Fix wrong metadata updates for first and last entry headers

Increase commit runner concurrency

Fix som cve

feat(distributed-lock): use metadata oxia client as distributed lock client

Use concurrent map for the cache strings in storageApi

\[cleanup] Move the metrics part out of the implementation

Make the maxPendingAddRequest dynamic

Correct the dockerfile used file location

Make the sync method call the async method to get result

Fix the request rejection of SimpleStorageImpl

Fix the missed exception handling for getting the next read index

Fix prepared publish task may failed issue.

Fix the ledger deletion for PersistCache

## Security Fixes


# V4.0.0.3
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.0.3



# StreamNative Weekly Release Notes v4.0.0.3

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.0.3](https://github.com/streamnative/pulsar/releases/tag/v4.0.0.3)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.0.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.0.3/images/sha256-08cf7fad11eee635b6ff34798e73677184d84b163b286144e29ee31519fb4f54)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.0.3/images/sha256-9adf05cd0ed26923deea58b260e962374c09d76b48eceb9b92c1dc544a759024)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.0.3/images/sha256-9adf05cd0ed26923deea58b260e962374c09d76b48eceb9b92c1dc544a759024)

## General Changes

### Apache Pulsar

\[fix] \[admin] Fix lookup get a null result if uses proxy

\[improve]\[io] Support update subscription position for sink connector

\[fix]\[broker] Increase readBuffer size for bookkeeper.DLOutputStream

### KoP

Enable geo replication test

### Cloud Storage Connector

Use messageId instead of sequence as file name by default

### Cloud Pulsar Plugins

Fix the wrong README of how to build sn-broker-interceptors and possible build failure locally

### Google BigQuery Sink Connector

Support auto update table with pulsar system filed

If model is null will use NULLABLE

Remove unnecessary version define

Add table type validation logic

Add google partner header

### StreamNative Ursa storage

Use ConcurrentHashMap to reduce risks

\[metrics] Add metrics for the write buffer

Filter topics with not support schema

Support write entry for the StorageApi

Optimize logs in compaction service

Optimize compaction service default configurations

Make the stream map evict by time.

\[cleanup] move the write cache out of the storage impl

\[cleanup]Remove the unused code in the persistStorageApi

Refactor compaction metrics

Bump org.apache.hadoop:hadoop-common from 3.3.6 to 3.4.0

## Security Fixes


# V4.0.0.4
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.0.4



# StreamNative Weekly Release Notes v4.0.0.4

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.0.4](https://github.com/streamnative/pulsar/releases/tag/v4.0.0.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.0.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.0.4/images/sha256-98f93e7bd929c1ebda2288af53faf8e4d18a66fb2d1f2b13f6e95e57a03ced21)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.0.4/images/sha256-4a0543b40cc8a07ddf5d0303945a8f7d3a428f3afccc3b4b12253f4025f4f7b5)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.0.4/images/sha256-4a0543b40cc8a07ddf5d0303945a8f7d3a428f3afccc3b4b12253f4025f4f7b5)

## General Changes

### Apache Pulsar

\[improve]\[broker] re-elect the channel owner if no channel owner is found

\[fix]\[broker] Fix ownership loss

\[improve]\[broker] Make cluster metadata teardown command support metadata config path

### KoP

Fix Kafka headers are not converted correctly when entryFormat is pulsar

### Cloud Storage Connector

Support include publish time to metadata

### StreamNative Pulsar Plugins

Reduce pulsar-rollout-plugin nar package size

### Google BigQuery Sink Connector

Remove verify logic for autoCreateTable

Improve auto update logic

### StreamNative Unified RBAC

fix: fix build script typo

### StreamNative Ursa storage

Fix aws credential not match issue

Reduce package size

Fix CVE-2024-7254

Upgrade aws sdk version to fix cve

Add error log when put entry failed in the managedLedgder

Trigger publish task by self, not wait all the topics.

Add json support check in publish task

Bump org.scala-lang:scala-library from 2.13.0 to 2.13.9

Update the Docker resource and readme to the latest stat

Optimize quarantine logs in publish tasks

CompactService support json schema

## Security Fixes


# V4.0.0.5
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.0.5



# StreamNative Weekly Release Notes v4.0.0.5

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.0.5](https://github.com/streamnative/pulsar/releases/tag/v4.0.0.5)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.0.5/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.0.5/images/sha256-73e7fd15bfe6be474b1514a013035b252c70dafd4ac6ec1d876b105e3ba0c632)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.0.5/images/sha256-86ee48f98e0e8362d428c36d0fa6f28038f63b8667012f79c5fa88525895207d)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.0.5/images/sha256-86ee48f98e0e8362d428c36d0fa6f28038f63b8667012f79c5fa88525895207d)

## General Changes

### Apache Pulsar

\[feat]\[misc] Upgrade oxia version to 0.4.9

\[fix]\[broker] Fix failed TokenAuthenticatedProducerConsumerTest

\[improve]\[offload] Use filesystemURI as the storage path

\[fix]\[misc] Unable to connect an etcd metastore with recent releases due to jetc-core sharding problem

\[improve]\[test] Clarify method signatures in Bookkeeper mock client

Enabling DNS retryOnTimeout with TCP in DnsNameResolver

\[improve] \[broker] replace HashMap with inner implementation ConcurrentLongLongPairHashMap in Negative Ack Tracker.

\[fix]\[client] The partitionedProducer maxPendingMessages always is 0

\[improve]\[broker] Support cleanup `replication cluster` and `allowed cluster` when cluster metadata teardown

\[fix]\[broker] Broker is failing to create non-durable sub if topic is fenced

\[fix]\[client] fix the beforeConsume() method earlier hit with message listener

\[fix]\[test] Fix DeadLetterTopicTest.testDeadLetterTopicWithInitialSubscriptionAndMultiConsumers

\[fix]\[test]Flaky-test: SchemaServiceTest.testSchemaRegistryMetrics

\[fix]\[broker] Fix currently client retries until operation timeout if the topic does not exist

\[fix]\[test] Fix SimpleProducerConsumerTest.testMultiTopicsConsumerImplPauseForManualSubscription

\[fix]\[broker] fix logging with correct error message while loading the topic

\[fix]\[test] Fix ManagedCursorTest.testForceCursorRecovery

\[improve]\[test] Disable OTel autoconfigured exporters in tests

\[fix]\[test] Fix memory leak via OTel shutdown hooks in tests

\[fix]\[broker] Fix print cluster migration state response

\[fix]\[broker] Fix Broker migration NPE while broker tls url not configured

\[fix]\[client] Fix the javadoc for startMessageIdInclusive

\[fix] \[broker] Fix race-condition causing repeated delete topic

\[fix]\[standalone] correctly delete bookie registration znode

\[fix]\[client] Fix Reader.hasMessageAvailable return wrong value after seeking by timestamp with startMessageIdInclusive

\[improve]\[broker] Exclude system topics from namespace level publish and dispatch rate limiting

\[improve]\[admin] Print error log if handle http response fails

### KoP

Disable bundle ownership transferring for bundles in shadow namespaces

### Cloud Storage Connector

Upgrade netty to fix CVE-2024-47535

### Cloud Pulsar Plugins

Upgrade netty to fix CVE-2024-47535

### StreamNative Tiered storage

Configure ksn entryformat in test

### StreamNative Ursa storage

Support creating the managedLedger in different brokers

\[fix]\[ml] fixed npe in getLastIndividualDeletedRange

Fix trivy downloading db rate limitation

Introduce task manager to avoid acquire lock between threads (Re-Commit)

Make mockito test scope

\[fix]\[tests] Enable ursa-storage-test tests

\[improve]\[tests] cover Pulsar messaging tests

([#401)](https://github.com/streamnative/ursa-storage/pull/401))) Revert "Introduce compaction task provider to avoid acquire lock between multiple threads

LakehouseKafkaReader add prefetch cache support

Fix the integer key leak issue

Improve the WAL catch up read throughput by prefetching cache

Introduce compaction task provider to avoid acquire lock between multiple threads

Add GCS FileStorage implementation

## Security Fixes

### Apache Pulsar

\[fix]\[sec] Upgrade to Netty 4.1.115.Final to address CVE-2024-47535

\[fix]\[sec] Upgrade Zookeeper to 3.9.3 to address CVE-2024-51504

\[fix]\[sec] Replace bcprov-jdk15on dependency with bcprov-jdk18-on


# V4.0.0.6
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.0.6



# StreamNative Weekly Release Notes v4.0.0.6

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.0.6](https://github.com/streamnative/pulsar/releases/tag/v4.0.0.6)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.0.6/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.0.6/images/sha256-8d25f6efbe4c2dc1f359834d9e35324613a91f5a9902e10b2baa5db32b301623)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.0.6/images/sha256-9c8ca48a4b63b150f66b2817c3cd2ebb8e8900066a13280d7d14ca8319db089c)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.0.6/images/sha256-9c8ca48a4b63b150f66b2817c3cd2ebb8e8900066a13280d7d14ca8319db089c)

## General Changes

### Apache Pulsar

\[improve]\[broker] PIP-392: Add configuration to enable consistent hashing to select active consumer for partitioned topic

### KoP

Handle Kafka multi-tenant format topic name in schema registry

### Google BigQuery Sink Connector

Set partitionedTable and ClusterTables to false when disable auto create table

## Security Fixes


# V4.0.0.7
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.0.7



# StreamNative Weekly Release Notes v4.0.0.7

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.0.7](https://github.com/streamnative/pulsar/releases/tag/v4.0.0.7)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.0.7/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.0.7/images/sha256-1de5a47bf08daf9d6befca88c7034588f26b29d96da80540f818f12832401d82)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.0.7/images/sha256-c54233c56a107edc2386d8ec45118274d450f8ec1cd359e656e3802d159add36)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.0.7/images/sha256-c54233c56a107edc2386d8ec45118274d450f8ec1cd359e656e3802d159add36)

## General Changes

### Apache Pulsar

\[improve] Upgrade oxia-java to 0.4.10 and fix closing of OxiaMetadataStore

\[fix]\[client] Fix deadlock of NegativeAcksTracker

\[improve]\[broker] Decouple pulsar\_storage\_backlog\_age\_seconds metric with backlogQuota check

\[fix]\[client] Make protobuf-java dependency optional in java client libraries

\[improve] Use single buffer for metrics when noUnsafe use

\[fix]\[broker] fix null lookup result when brokers are starting

\[fix]\[client] Fixed an issue where a cert chain could not be used in TLS authentication

\[improve]\[misc] Disable OTel by default when running the pulsar-perf tool

\[cleanup]\[build] skip generating pom.xml.versionsBackup

\[fix]\[client] Initializing client-authentication using configured auth params

\[fix]\[ws] Implement missing http header data functions in AuthenticationDataSubscription

\[fix]\[misc] Class conflict during jetcd-core-shaded shading process

\[improve]\[test] Reduce OneWayReplicatorUsingGlobalZKTest.testRemoveCluster execution time

\[improve]\[broker] Skip unloading when bundle throughput is zero (ExtensibleLoadManagerImpl only)

\[improve]\[broker] Reduce memory occupation of the delayed message queue

\[fix]\[client] fix incomingMessageSize and client memory usage is negative

\[fix]\[fn] ack messages for window function when its result is null

\[improve] Improve logic for enabling Netty leak detection

\[improve]\[ml] Avoid repetitive nested lock for isMessageDeleted in ManagedCursorImpl

### KoP

\[Ursa] Don't fail with OFFSET\_OUT\_OF\_RANGE when LEO is less than the fetch offset

### AWS Lambda Connector

Support include publish time to metadata

### StreamNative Pulsar Plugins

Change authentication failed log level to warn

Upgrade aws sdk dependency version to v2

exlude netty for aws-jdk

### Cloud Pulsar Plugins

Allow accepting token from query parameters

### Function Mesh Worker Service

Set processingGuarantee for window functions

Bump function-mesh to v0.23.0

### StreamNative Ursa storage

Fix the commit runner race condition issue.

Update streamId if Shadow Managed Ledger stream ID is invalid

Trigger the metadata update when calling getLastConfirmed entry

\[improve]\[tests] covered Pulsar protocol retention and backlog quota

\[improve]\[tests] covers Pulsar protocol encryption and compression tests

Remove awssdk bundle dependency

Avoiding the risk of pending add buffer release

fix netty and json cve

Fix the stream id is duplicated

Do not reject the entries bigger than write buffer

Support the azure file storage

\[improve]\[tests] enabled ExtensibleLoadBalancer in integ tests and added delayed messaging test

\[improve]\[tests] enabled concurrent tests in integ and enabled new load balancer

## Security Fixes


# V4.0.0.8
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.0.8



# StreamNative Weekly Release Notes v4.0.0.8

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.0.8](https://github.com/streamnative/pulsar/releases/tag/v4.0.0.8)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.0.8/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.0.8/images/sha256-edd7bca0c80f1838dec227d6a7e767bd8f23a2698124d84fd0d9c7034edf3757)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.0.8/images/sha256-ff965d222b7fc11dbdb3484646c5f62afa81afb93af01063e9c523f8ed14eacd)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.0.8/images/sha256-ff965d222b7fc11dbdb3484646c5f62afa81afb93af01063e9c523f8ed14eacd)

## General Changes

### Apache Pulsar

\[fix]\[broker] Invoke custom BrokerInterceptor's `onFilter` method if it's defined

\[fix]\[broker] support missing cluster level fine-granted permissions

0aa04368a9 Bump version to next snapshot version

\[fix]\[broker] support missing tenant level fine-granted permissions

\[fix]\[broker] Revert "\[improve]\[client] Add log when can't add message to the container

\[improve]\[broker] Reduce memory occupation of InMemoryRedeliveryTracker.

\[feat]\[broker] Implement allowBrokerOperationAsync in PulsarAuthorizationProvider to avoid exception thrown

\[fix]\[broker]: support missing broker level fine-granted permissions

\[improve]\[client] Enhance error handling for non-exist subscription in consumer creation

\[fix]\[client] Fix race-condition causing doReconsumeLater to hang when creating retryLetterProducer has failed

\[improve]\[client] Reduce unshaded dependencies and shading warnings in shaded Java client modules

\[improve] Upgrade OpenTelemetry library to 1.44.1 version

\[improve] \[pip] PIP-373: Add a topic's system prop that indicates whether users have published TXN messages in before.

\[improve]\[client] Replace NameUtil#generateRandomName with RandomStringUtils#randomAlphanumeric

\[fix]\[build] Fix error "Element encoding is not allowed here" in pom.xml

\[fix]\[client] Fix DLQ producer name conflicts when there are same name consumers

### MoP

Make the proxy adapter worker thread configrable

Fix the auth data is NPE error

Fix broker enable dedup cause client publish failed

Seperate proxy and broker a single module

Refactor MoP to prepare for split Proxy to seperate module

### KoP

Add metric for consumer lag

### StreamNative Pulsar Plugins

bump pulsar 4.0.0.8

### StreamNative Unified RBAC

fix(ci): fix CI failed by wrong packet name

feat: upgrade the pulsar to snapshot repo

fix(sdk-js): fix the JSON format

feat(sdk-js): support permission cube

feat: support cluster,tenant,broker level permissions

upgrade the project version to 1.2.0

fix(sdk-js): upgrade the version to 0.0.4

refine the metadata specification

fix(sdk-java): fix failed integration test

feat(java): release 1.1.0

feat(pom): upgrade version to 1.1.0-snaphsot

Feat.improve.publish

feat(sdk-java): publish java to github packages

feat(sdk-java): improve the condition authorization interface

fix(sdk-go-cloud): fix wrong subject comparing

fix(sdk-go-cloud): fix undecoded service account subject

feat(metadata): support new permissions for cloud metrics

upgrade sdk js version

fix types import

feat: support CLI for unified rbac read endpoints

fix: fix proto decode issue

feat(sdk-js): support NewAuthorizerWithPrivilegesString

feat: upgrade sdk-go proto

feat(sdk-js): update proto definition

feat(sdk-go-cloud): upgrade sdk-go to 0.1.8

feat(sdk-go): upgrade schema

feat(sdk-js): support privileges authorizer

feat: make proto pojo json to camel case

fix: fix sdk-js name

feat: upgrade node version to 20

feat(doc): add document for unified-rbac

feat: support sdk-js

### StreamNative Ursa storage

\[WIP] Add primitive type support

Fix the null filed value can't decode issue.

Introduce lakehouse read prefetch cache manager

Speed up get all task

Separte the maxRequest config and write buffer segment config

## Security Fixes


# V4.0.0.9
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.0.9



# StreamNative Weekly Release Notes v4.0.0.9

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.0.9](https://github.com/streamnative/pulsar/releases/tag/v4.0.0.9)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.0.9/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.0.9/images/sha256-25eeae6d7c8ec19febaef9378e6485e6b3d95da00352970e30cfa35966fc0f9d)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.0.9/images/sha256-47065fa9d11ae619fc4d8b1e6a1b16a0662569b22b9954e772dd89df1a29fbf8)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.0.9/images/sha256-47065fa9d11ae619fc4d8b1e6a1b16a0662569b22b9954e772dd89df1a29fbf8)

## General Changes

### Apache Pulsar

\[fix]\[cli] Fix set-retention with >2GB size value for topic policy

\[fix]\[cli] Fix set topic retention policy failed

\[improve]\[io] Bump io.lettuce:lettuce-core from 5.0.2.RELEASE to 6.5.1.RELEASE in /pulsar-io/redis

\[fix] \[broker] Add consumer name for subscription stats

\[improve] Install coreutils in docker image to improve compatibility

### MoP

Fix mop producer publish metric

### KoP

Add pulsar-kafka-schema-registry jar to image

### Function Mesh Worker Service

Support set pod annotations via CustomRuntimeOptions

### StreamNative Tiered storage

iceberg catalog suit the polaris catalog.

Upgrade aws dependency to 2.x

### StreamNative Unified RBAC

feat(sdk-java-pulsar): Implement broker rbac filter

### StreamNative Ursa storage

Grouping pending add requests by stream id

Add read request limitation for the persistStorageApi read

Guarantee managed ledger's stream id is always valid and never modified

Avoid concurrently update metadata for ShadowManagedLedger

Provided storage endpoint for azure storage

## Security Fixes

### Apache Pulsar

\[fix]\[sec] Bump commons-io version to 2.18.0


# V4.0.1.1
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.1.1



# StreamNative Weekly Release Notes v4.0.1.1

## Download

### Distributions

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.1.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.1.1/images/sha256-730e429d6f2f737fa01b670e17483e895065d039ef58a455d91ac9e2f2dc286b)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.1.1/images/sha256-bef14be1a4e927e1ed418a3de24c8290a2dfb92d15a6e82935acef9a0adee22b)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.1.1/images/sha256-bef14be1a4e927e1ed418a3de24c8290a2dfb92d15a6e82935acef9a0adee22b)

## General Changes

### Apache Pulsar

([#23824](https://github.com/apache/pulsar/pull/23824)) \[fix]\[client] Prevent retry topic and dead letter topic producer leaks when sending of message fails

([#22792](https://github.com/apache/pulsar/pull/22792)) \[improve]\[broker] Optimize subscription seek (cursor reset) by timestamp

([#23823](https://github.com/apache/pulsar/pull/23823)) \[fix]\[test] Remove useless test code

([#23817](https://github.com/apache/pulsar/pull/23817)) \[fix]\[broker] Remove failed OpAddEntry from pendingAddEntries

([#23813](https://github.com/apache/pulsar/pull/23813)) \[improve] Upgrade to Netty 4.1.116.Final and io\_uring to 0.0.26.Final

([#23810](https://github.com/apache/pulsar/pull/23810)) \[improve]\[ci] Move ZkSessionExpireTest to flaky group to unblock CI

([#23784](https://github.com/apache/pulsar/pull/23784)) \[fix]\[admin] Fix exception thrown in getMessageId method

([#23776](https://github.com/apache/pulsar/pull/23776)) \[improve]\[ml] Optimize BlobStoreManagedLedgerOffloader.getOffloadPolicies

([#23779](https://github.com/apache/pulsar/pull/23779)) \[improve]\[txn] Improve Reader in TransactionBuffer to reduce GC pressure

([#23804](https://github.com/apache/pulsar/pull/23804)) \[improve]\[client] PIP-393: Support configuring NegativeAckPrecisionBitCnt while building consumer.

([#23666](https://github.com/apache/pulsar/pull/23666)) \[fix]\[test]: Flaky-test: GetPartitionMetadataMultiBrokerTest.testCompatibilityDifferentBrokersForNonPersistentTopic

([#23780](https://github.com/apache/pulsar/pull/23780)) \[improve]\[broker] Improve SystemTopicBasedTopicPoliciesService reader to reduce GC pressure

([#23802](https://github.com/apache/pulsar/pull/23802)) \[fix] \[broker] Fix items in dispatcher.recentlyJoinedConsumers are out-of-order, which may cause a delivery stuck

([#23600](https://github.com/apache/pulsar/pull/23600)) \[improve]\[client] PIP-393: Improve performance of Negative Acknowledgement

([#23795](https://github.com/apache/pulsar/pull/23795)) \[fix]\[broker] Msg delivery is stuck due to items in the collection recentlyJoinedConsumers are out-of-order

([#23791](https://github.com/apache/pulsar/pull/23791)) \[fix]\[client] Cannot access message data inside ProducerInterceptor#onSendAcknowledgement

([#23786](https://github.com/apache/pulsar/pull/23786)) \[fix]\[broker] topic policy deadlock block metadata thread.

([#23652](https://github.com/apache/pulsar/pull/23652)) \[improve]\[log] Print ZK path if write to ZK fails due to data being too large to persist

([#23718](https://github.com/apache/pulsar/pull/23718)) \[fix]\[client] Make DeadLetterPolicy & KeySharedPolicy serializable

([#23798](https://github.com/apache/pulsar/pull/23798)) \[fix]\[client] Fix compatibility between kerberos and tls

([#23797](https://github.com/apache/pulsar/pull/23797)) \[fix]\[broker] Continue using the next provider for authentication if one fails

([#23781](https://github.com/apache/pulsar/pull/23781)) \[fix]\[broker] Fix enableReplicatedSubscriptions

([#23757](https://github.com/apache/pulsar/pull/23757)) \[improve]\[client] Make replicateSubscriptionState nullable

([#23772](https://github.com/apache/pulsar/pull/23772)) \[fix]\[ml] Topic load timeout due to ml data ledger future never finishes

([#23767](https://github.com/apache/pulsar/pull/23767)) \[fix]\[broker] System topic should not be migrated during blue-green cluster migration

([#23766](https://github.com/apache/pulsar/pull/23766)) \[fix]\[admin] Fix exception loss in getMessageId method

([#23762](https://github.com/apache/pulsar/pull/23762)) \[fix] Fix issues with Pulsar Alpine docker image stability: remove glibc-compat

([#23753](https://github.com/apache/pulsar/pull/23753)) \[fix]\[client] Fix enableRetry for consumers using legacy topic naming where cluster name is included

([#23693](https://github.com/apache/pulsar/pull/23693)) \[fix]\[client] Fix reader message filtering issue during blue-green cluster switch

([#23764](https://github.com/apache/pulsar/pull/23764)) \[fix]\[broker] Fix bug causing loss of migrated information when setting other localPolicies in namespace

([#23747](https://github.com/apache/pulsar/pull/23747)) \[fix]\[test] Fix flaky KeySharedSubscriptionTest.testNoKeySendAndReceiveWithHashRangeAutoSplitStickyKeyConsumerSelector

([#23761](https://github.com/apache/pulsar/pull/23761)) \[Fix]\[Client] Fix pending message not complete when closeAsync

([#23756](https://github.com/apache/pulsar/pull/23756)) \[improve]\[monitor] Upgrade OTel to 1.45.0

([#23738](https://github.com/apache/pulsar/pull/23738)) \[fix] \[client] Fix memory leak when publishing encountered a corner case error

([#23734](https://github.com/apache/pulsar/pull/23734)) \[improve]\[fn] Improve closing of producers in Pulsar Functions ProducerCache invalidation

([#23708](https://github.com/apache/pulsar/pull/23708)) \[improve]\[fn] Improve implementation for maxPendingAsyncRequests async concurrency limit when return type is CompletableFuture

([#23752](https://github.com/apache/pulsar/pull/23752)) \[improve] Upgrade lombok to 1.18.36

([#23691](https://github.com/apache/pulsar/pull/23691)) \[fix]\[common] TopicName: Throw IllegalArgumentException if localName is whitespace only

([#23730](https://github.com/apache/pulsar/pull/23730)) \[fix]\[admin] Verify is policies read only before revoke permissions on topic

### AoP

c11cb32 Fix build script

([#1433](https://github.com/streamnative/aop/pull/1433)) Use SN bom

([#1471](https://github.com/streamnative/aop/pull/1471)) remove useless check in tests

### MoP

([#1590](https://github.com/streamnative/mop/pull/1590)) Remove yahoo dependency

### KoP

Revert "Revert using sn-bom (#957)"

### Cloud Storage Connector

([#1172](https://github.com/streamnative/pulsar-io-cloud-storage/pull/1172)) Add configuration to exclude topic on file path

([#1142](https://github.com/streamnative/pulsar-io-cloud-storage/pull/1142)) Refactor thread mode and batch memory control

### AMQP1\_0 Connector

([#1092](https://github.com/streamnative/pulsar-io-amqp-1-0/pull/1092)) Use new clean disk job

### StreamNative Pulsar Plugins

fix azure package failed when list non-exists directory

Support Topic Level Tracing and LogTraceRecorder

Azure Blob Storage backed Package Management Service

Exclude commons-io to fix cve

Fix license commons plugin jar package

Use WebIdentity way when AWS\_WEB\_IDENTITY\_TOKEN\_FILE exists

fix test in BrokerUnloadJobResourcesTest

### Cloud Pulsar Plugins

Use new clean disk job

### Function Mesh Worker Service

Use large runner to avoid disk full issue

Fix free disk job of ubuntu-latest runner

bump kafka dep to 3.9.0

c4fd73a7 Use separate image for kafka connect in CI

Replace the deprecated `getZooKeeperSessionTimeoutMillis`

### Google BigQuery Sink Connector

Stop executor when connector closed

### StreamNative Unified RBAC

fix: fix the license of generator

feat: support new permission `pulsar.packages.admin`

feat: format generator license

feat: improve the document

Add User, ClusterRole, RoleBinding tests

fix(sdk-java): fix empty cel passed authorization

feat(sdk-go-cloud): upgrade the sdk-go to v0.2.0

fix(\*): fix license

fix(\*): fix license issue

feat(\*): upgrade the project version to 1.4.0

feat(sdk-js): upgrade version to 0.0.8

all: basic SRN supprot.

Add application tests

Optimize integration tests

feat(sdk-js): resource admin support

Add more integration tests

### StreamNative Ursa storage

Fix flaky test TestCompactionServiceBaseFileStorage#simple

Fix ManagedCursorTest#testReadEntriesWithSkipDeletedEntries flaky test.

Fix StorageWalManagedCursor#internalReadEntries only read one entry at each round.

Fix s3 compact flaky test.

\[TEST] Run integration test with image ksn 3.9.0

Fix parquet prefetch bug

Add iceberg azure dependency

Use separate thread pool for parquet reading

Test master CI

Fix default compaction thread bug

Support load credential from file

add cloud region

bdb91385 \[Bug] fix duplicated entry id put in PersistCache.index

Adjust the compaction default configuration according to the performance test

Fix flaky test in TestGarbageCollection

Add the ProfileCredentialsProvider into the auth chain

Disable nonblocking dns and get rid of request limitation of S3 prefix

improvement in dispatch logic

Fix the default write buffer segment configuration

Unity catalog support config User-agent.

Adding stress-ng to CI

Fix oxia read failed with small range

Improve CI by running S3 and GCS integration tests In parallel

Optimize PersistCache serialization to only persist used segments

Support disable read cache expire by time

Compaction service support GCS

Delta table support partition column using topic partition index

Compact service support azure blob protocol

Parse the storage account name and blob container name from bucket for azure

Check the table whether register to the unity catalog when commit action. If not, register the table to unity catalog.

Compaction service support Azure storage.

Modify delta table mapping name in unity catalog

Rename azure to azureblob to make it more precise

fix azure make data in heap

Quarantine the compact task if it read WAL data fialed.

Add the storage metrics for all the file storage type

Allow to disable the lakehouse reader in the managedledger

Enable ChecksumCRC32C for getting object from S3

## Security Fixes

### Apache Pulsar

([#23743](https://github.com/apache/pulsar/pull/23743)) \[fix]\[sec] Upgrade golang.org/x/crypto from 0.21.0 to 0.31.0 in pulsar-function-go


# V4.0.1.2
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.1.2



# StreamNative Weekly Release Notes v4.0.1.2

## Download

### Distributions

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.1.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.1.2/images/sha256-82d0b4d50327db370af66977bd77085a44ef1d6a8bc685176911c7a34d9e256a)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.1.2/images/sha256-79a5d3ce04593c39880b91dff3a5324c4cc4a03e9976c335beb68fea5a50cd5b)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.1.2/images/sha256-79a5d3ce04593c39880b91dff3a5324c4cc4a03e9976c335beb68fea5a50cd5b)

## General Changes

### Apache Pulsar

([#23871](https://github.com/apache/pulsar/pull/23871)) \[feat]\[misc] upgrade oxia java client to 0.5.0

([#23864](https://github.com/apache/pulsar/pull/23864)) \[improve]\[broker] Improve Consumer.equals performance

([#23863](https://github.com/apache/pulsar/pull/23863)) \[improve] Upgrade to Netty 4.1.117.Final

([#23862](https://github.com/apache/pulsar/pull/23862)) \[improve]\[broker] Remove spamming logs for customized managed ledger

([#23857](https://github.com/apache/pulsar/pull/23857)) \[fix]\[test] Add reconsumeLater call in RetryTopicTest#testRetryTopicWithMultiTopic.

([#23853](https://github.com/apache/pulsar/pull/23853)) \[fix]\[client] Orphan producer when concurrently calling producer closing and reconnection

([#23384)](https://github.com/apache/pulsar/pull/23384))) ([#23855](https://github.com/apache/pulsar/pull/23855)) \[fix]\[broker] Revert "\[fix]\[broker] Cancel possible pending replay read in cancelPendingRead

([#23854](https://github.com/apache/pulsar/pull/23854)) \[fix]\[broker] Fix deadlock in Key\_Shared PIP-379 implementation

([#23851](https://github.com/apache/pulsar/pull/23851)) \[improve]\[ci] Publish build scans to develocity.apache.org

([#23852](https://github.com/apache/pulsar/pull/23852)) \[fix]\[test]Fix flaky test testTopicUnloadAfterSessionRebuild

([#23846](https://github.com/apache/pulsar/pull/23846)) \[improve] Support overriding java.net.preferIPv4Stack with OPTS

([#23712](https://github.com/apache/pulsar/pull/23712)) \[fix]\[broker] PIP-399: Fix Metric Name for Delayed Queue

([#23833](https://github.com/apache/pulsar/pull/23833)) \[fix]\[broker] Fix possible mark delete NPE when batch index ack is enabled

([#23841](https://github.com/apache/pulsar/pull/23841)) \[fix] \[broker] Fix acknowledgeCumulativeAsync block when ackReceipt is enabled

([#23847](https://github.com/apache/pulsar/pull/23847)) \[fix]\[misc] Honor dynamic log levels in log4j2.yaml

([#23832](https://github.com/apache/pulsar/pull/23832)) \[fix]\[broker] Remove blocking calls from internalGetPartitionedStats

([#23842](https://github.com/apache/pulsar/pull/23842)) \[fix]\[broker] Continue using the next provider for http authentication if one fails

([#23839](https://github.com/apache/pulsar/pull/23839)) \[improve]\[broker] Reduce unnecessary REPLICATED\_SUBSCRIPTION\_SNAPSHOT\_REQUEST

### AoP

([#1519](https://github.com/streamnative/aop/pull/1519)) Fix SN bom version

([#1501](https://github.com/streamnative/aop/pull/1501)) Fix build script

### MoP

([#1609](https://github.com/streamnative/mop/pull/1609)) Fix build script

([#1606](https://github.com/streamnative/mop/pull/1606)) Use SN bom

### KoP

Add a simple consistent hashing based load balancer implementation

### pulsarctl

([#1711](https://github.com/streamnative/pulsarctl/pull/1711)) fix code check

([#1704](https://github.com/streamnative/pulsarctl/pull/1704)) feat: Subscription get message by id json output

([#1699](https://github.com/streamnative/pulsarctl/pull/1699)) Update subscription get message by id typo lederId to ledgerId

### StreamNative Pulsar Plugins

Fix deploy workflow

Fix deploy package workflow

Add plugin module deploy workflow

### Function Mesh Worker Service

Use SN bom SNAPSHOT version

Do not allow using system topics when creating sink\&source

Reject request when kafka connect's name is longer than 29

Use SN bom to reduce artifact size

### StreamNative Unified RBAC

Add lack sdk jar package and fix integration test

Use SN bom

### StreamNative Ursa storage

\[cleanup] Move the read cache into a single class

Clean up the unused code of ursa-storage-core

Remove the unused code

Use ubuntu-latest to run ci

Unity Catalog support clientId and clientSecret to authenticate.

## Security Fixes


# V4.0.1.3
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.1.3



# StreamNative Weekly Release Notes v4.0.1.3

## Download

### Distributions

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.1.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.1.3/images/sha256-85ae7df05e1f69f60342c54dcc61408f75dac4d0ff3cf370e86c4a9b6b2151f8)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.1.3/images/sha256-fc15792acab687cfd57f12647a857d13e1a46cb7c1df79c5ee402e74e53f993d)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.1.3/images/sha256-fc15792acab687cfd57f12647a857d13e1a46cb7c1df79c5ee402e74e53f993d)

## General Changes

### Apache Pulsar

([#23914](https://github.com/apache/pulsar/pull/23914)) \[fix] Initialize UrlServiceProvider before trying to use transaction coordinator

([#23911](https://github.com/apache/pulsar/pull/23911)) \[fix] Avoid NPE when closing an uninitialized SameAuthParamsLookupAutoClusterFailover

([#23901](https://github.com/apache/pulsar/pull/23901)) \[fix]\[broker] Make InflightReadsLimiter asynchronous and apply it for replay queue reads

([#23909](https://github.com/apache/pulsar/pull/23909)) \[fix]\[ci] Configure Docker data-root to /mnt/docker to avoid running out of disk space

([#23903](https://github.com/apache/pulsar/pull/23903)) \[fix]\[broker Fix bug in RangeCache where different instance of the key wouldn't ever match

([#23893](https://github.com/apache/pulsar/pull/23893)) \[fix]\[test] Fix flaky DelayedDeliveryTest.testEnableTopicDelayedDelivery

([#23869](https://github.com/apache/pulsar/pull/23869)) \[fix]\[broker] Fix repeatedly acquired pending reads quota

([#23898](https://github.com/apache/pulsar/pull/23898)) \[fix]\[build] Use amazoncorretto:21-alpine3.20 JDK build for Alpine 3.20

([#23894](https://github.com/apache/pulsar/pull/23894)) \[fix]\[broker] Apply dispatcherMaxReadSizeBytes also for replay reads for Shared and Key\_Shared subscriptions

([#23892](https://github.com/apache/pulsar/pull/23892)) \[improve]\[test] Support decorating topic, subscription, dispatcher, ManagedLedger and ManagedCursors instances in tests

([#23881](https://github.com/apache/pulsar/pull/23881)) \[improve]\[fn] Set default tenant and namespace for ListFunctions cmd

([#23886](https://github.com/apache/pulsar/pull/23886)) \[fix]\[client] Fix LoadManagerReport not found

([#23759](https://github.com/apache/pulsar/pull/23759)) \[fix] \[ml] Fix cursor metadata compatability issue when switching the config unackedRangesOpenCacheSetEnabled

([#23878](https://github.com/apache/pulsar/pull/23878)) \[improve]\[broker] Support values up to 2^32 in ConcurrentBitmapSortedLongPairSet

([#23883](https://github.com/apache/pulsar/pull/23883)) \[improve]\[ci] Increase Maven max heap size to 2048M and tune GCLockerRetryAllocationCount

([#23874](https://github.com/apache/pulsar/pull/23874)) \[fix]\[broker] PIP-379 Key\_Shared implementation race condition causing out-of-order message delivery

([#23876](https://github.com/apache/pulsar/pull/23876)) \[fix]\[test] Fix quiet time implementation in BrokerTestUtil.receiveMessages

([#23875](https://github.com/apache/pulsar/pull/23875)) \[improve]\[test] Add solution to PulsarMockBookKeeper for intercepting reads

87674c4b8e Bump version to next snapshot version

### AoP

([#1525](https://github.com/streamnative/aop/pull/1525)) Upgrade artifact github action version

([#1525](https://github.com/streamnative/aop/pull/1525)) Upgrade artifact github action version

### MoP

([#1626](https://github.com/streamnative/mop/pull/1626)) Upgrade artifact github action version

### KoP

Add a simple consistent hashing based load balancer implementation

### Google BigQuery Sink Connector

Upgrade artifact version

Upgrade artifact version

### StreamNative Ursa storage

## Security Fixes


# V4.0.1.4
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.1.4



# StreamNative Weekly Release Notes v4.0.1.4

## Download

### Distributions

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.1.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.1.4/images/sha256-3cc73b7c17a3d2ef98f56bc9b146d66ba8094b0fa8414e2ec066fd497b1fa992)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.1.4/images/sha256-19c4eb1cc65fe1d75b006a77f2f653083c4cb52921c483c7ae21a8f61b8a1c96)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.1.4/images/sha256-19c4eb1cc65fe1d75b006a77f2f653083c4cb52921c483c7ae21a8f61b8a1c96)

## General Changes

### Apache Pulsar

([#23331](https://github.com/apache/pulsar/pull/23331)) \[fix]\[broker] fix broker may lost rack information

([#23984](https://github.com/apache/pulsar/pull/23984)) \[fix]\[meta] Fix ephemeral Zookeeper put which creates a persistent znode

([#23796](https://github.com/apache/pulsar/pull/23796)) \[fix]\[broker] Fix incorrect blockedConsumerOnUnackedMsgs value when maxUnackedMessagesPerConsumer is 1

([#23981](https://github.com/apache/pulsar/pull/23981)) \[improve]\[proxy] Make keep-alive interval configurable in Pulsar Proxy

([#23979](https://github.com/apache/pulsar/pull/23979)) \[fix]\[io] Fix pulsar-io:pom not found

([#23969](https://github.com/apache/pulsar/pull/23969)) \[improve]\[client] Update TypedMessageBuilder deliverAfter and deliverAt api comment

([#23967](https://github.com/apache/pulsar/pull/23967)) \[fix]\[client] Fix memory leak when message size exceeds max message size and batching is enabled

([#23971](https://github.com/apache/pulsar/pull/23971)) \[fix]\[client] Fix memory leak in ClientCnx.newLookup when there's TooManyRequestsException

([#23899](https://github.com/apache/pulsar/pull/23899)) \[fix] Bump org.apache.solr:solr-core from 8.11.3 to 9.8.0 in /pulsar-io/solr

([#23970](https://github.com/apache/pulsar/pull/23970)) \[improve]\[ci] Skip "OWASP dependency check" when data wasn't found in cache

([#23966](https://github.com/apache/pulsar/pull/23966)) \[fix]\[build] Upgrade json-smart to 2.5.2

([#23957](https://github.com/apache/pulsar/pull/23957)) \[improve]\[broker] Avoid PersistentReplicator.expireMessages logic compute backlog twice

([#23960](https://github.com/apache/pulsar/pull/23960)) \[fix]\[ml] Fix memory leaks in ManagedCursorInfo and ManagedLedgerInfo decompression and compression

([#23964](https://github.com/apache/pulsar/pull/23964)) \[fix] Use Alpine 3.21 in base image

([#23958](https://github.com/apache/pulsar/pull/23958)) \[fix]\[ml] Fix deadlock in PendingReadsManager

([#23915](https://github.com/apache/pulsar/pull/23915)) \[improve]\[broker] Refactor a private method to eliminate an unnecessary parameter

([#23919](https://github.com/apache/pulsar/pull/23919)) \[fix]\[broker] Fix seeking by timestamp can be reset the cursor position to earliest

([#23951](https://github.com/apache/pulsar/pull/23951)) \[fix] \[ml] incorrect non-durable cursor's backlog due to concurrently trimming ledger and non-durable cursor creation

([#23615](https://github.com/apache/pulsar/pull/23615)) \[fix]\[broker] Skip to persist cursor info if it failed by cursor closed

([#23955](https://github.com/apache/pulsar/pull/23955)) \[fix]\[ml] Fix memory leak due to duplicated RangeCache value retain operations

([#23940](https://github.com/apache/pulsar/pull/23940)) \[improve]\[ml] Do not switch thread to execute asyncAddEntry's core logic

([#23943](https://github.com/apache/pulsar/pull/23943)) \[fix] \[client] call redeliver 1 msg but did 2 msgs

([#23930](https://github.com/apache/pulsar/pull/23930)) \[fix]\[broker] Fix rate limiter token bucket and clock consistency issues causing excessive throttling and connection timeouts

([#23947](https://github.com/apache/pulsar/pull/23947)) \[feat]\[client] Support forward proxy for the ZTS server in pulsar-client-auth-athenz

([#23932](https://github.com/apache/pulsar/pull/23932)) \[improve]\[io] Allow skipping connector deployment

([#23938](https://github.com/apache/pulsar/pull/23938)) \[improve]\[broker] Avoid printing log for IncompatibleSchemaException in ServerCnx

([#23939](https://github.com/apache/pulsar/pull/23939)) \[improve]\[broker] Avoid logging errors when there is a connection issue during subscription.

([#23928](https://github.com/apache/pulsar/pull/23928)) \[improve]\[broker] Do not print error logs for NotFound or Conflict errors when using the Admin API

([#23929](https://github.com/apache/pulsar/pull/23929)) \[improve]\[broker] Don't print error logs for ProducerBusyException

([#23935](https://github.com/apache/pulsar/pull/23935)) \[improve]\[client] Avoid logging errors for retriable errors when creating producer

([#23884](https://github.com/apache/pulsar/pull/23884)) \[fix]\[broker] Closed topics won't be removed from the cache

### KoP

Support wildcard characters for kopAllowedNamespaces

### StreamNative Pulsar Plugins

feat(detector): support pprof

fix: discard superuser flag to avoid deadlock

fix(detector): compatible with non-partitioned aliveness topic

Change error log to warn when topic closed

### Function Mesh Worker Service

Fix update error for functions/sinks/sources with secrets injected

Support set log topic from configs for Sinks\&Sources

Support set resources for kafka connect

### StreamNative Ursa storage

Support nested protobuf schema for the compaction

Move the create table to the writer instead committer

Add test case to cover msg payload schema content

Remove copied class

Remove the ursa-lakehouse dependency from the ursa-ml

Add iceberg external table support

Support kafka protobuf schema for the compaction

Fix BrokerInterceptorTest.testAddBrokerEntryMetadataEntryRef flaky test.

Add test unit to cover get wrong streamId issue.

Fix CI

Fix the json schema convert issue.

Fix always use the wrong streamId when publish stream task.

Set partitionkey for the rangeScan in the persistStorageApi

Add the persistCache size into the meta

Improve PersistCache with a simpler and thread-safe implementation

Add EntryIndex protobuf

feat: move the info log to debug to prevent server keep logging

\[cleanup] Remove unused api

Refactor compaction worker interface

Replace guava cache with caffeine cache

\[cleanup] Move the read cache into a single class

Clean up the unused code of ursa-storage-core

Remove the unused code

Unity Catalog support clientId and clientSecret to authenticate.

Use ubuntu-latest to run ci

Fix Github Packages publish workflow

Remove nexus snapshot distribution

Use `____` to replace '-' for table name when create unity catalog table.

Support publishing to GitHub Packages

## Security Fixes

### Apache Pulsar

([#23965](https://github.com/apache/pulsar/pull/23965)) \[fix]\[sec] Upgrade to Netty 4.1.118


# V4.0.3.2
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.3.2



# StreamNative Weekly Release Notes v4.0.3.2

## Download

### Distributions

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.3.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.3.2/images/sha256-7756fe68d9c347b034c711911665f6a9a48f11dcf7e0b54ac4f73dc28474e1ca)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.3.2/images/sha256-1855949fd0608acc0b68b4f27e18e7c6eb7649e0a7a6f4cc76b2f3bc91c11192)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.3.2/images/sha256-1855949fd0608acc0b68b4f27e18e7c6eb7649e0a7a6f4cc76b2f3bc91c11192)

## General Changes

### Apache Pulsar

([#24126](https://github.com/apache/pulsar/pull/24126)) \[improve]\[meta] Change log level from error to warn for unknown notification types in OxiaMetadataStore

([#24131](https://github.com/apache/pulsar/pull/24131)) \[fix]\[ml] Return 1 when bytes size is 0 or negative for entry count estimation

([#24128](https://github.com/apache/pulsar/pull/24128)) \[improve]\[io] Enhance Kafka connector logging with focused bootstrap server information

([#24125](https://github.com/apache/pulsar/pull/24125)) \[fix]\[ml] Don't estimate number of entries when ledgers are empty, return 1 instead

([#24123](https://github.com/apache/pulsar/pull/24123)) \[improve]\[client] Prevent NullPointException when closing ClientCredentialsFlow

([#24124](https://github.com/apache/pulsar/pull/24124)) \[improve]\[io] Remove sleep when sourceTask.poll of kafka return null

([#24116](https://github.com/apache/pulsar/pull/24116)) \[improve]\[broker] Change topic exists log to warn

([#24111](https://github.com/apache/pulsar/pull/24111)) \[improve]\[broker]\[branch-4.0] PIP-406: Introduce metrics related to dispatch throttled events

### KoP

Support content negotiation for schema registry HTTP service

### StreamNative Pulsar Plugins

Add environment to e2e pipeline

### Function Mesh Worker Service

add security schemas to kafka connect openapi

better error responses

bump function-mesh to 0.24.0

masking the sensitive data in logs

support insecure auth secret override

### StreamNative Tiered storage

Fix AWS authentication order

### StreamNative Ursa storage

\[fix] write api support for v2 format

Make the entry read instance in compaction not shared betweet the different task

Fix the IllegalReferenceCountException by duplicated method execute

Fix npe when check lakehouse commit table failed.

Do ursa GCS performance and improvement

Fix GCS deadlock issue

## Security Fixes


# V4.0.4.1
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.4.1



# StreamNative Weekly Release Notes v4.0.4.1

## Download

### Distributions

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.4.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.4.1/images/sha256-27c8e603e27d0afd45bf826b670f9248613e5417027a63d16d4e432baf5fce8f)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.4.1/images/sha256-532a0ef7d4e94268865e2a5f8b5cb25310d88f36cf1b21d2e68525fdb6e4c52a)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.4.1/images/sha256-532a0ef7d4e94268865e2a5f8b5cb25310d88f36cf1b21d2e68525fdb6e4c52a)

## General Changes

### Apache Pulsar

([#24156](https://github.com/apache/pulsar/pull/24156)) \[fix]\[broker] fix ExtensibleLoadManager to override the ownerships concurrently without blocking load manager thread

([#24161](https://github.com/apache/pulsar/pull/24161)) \[fix]\[test] Fix flaky BrokerServiceChaosTest.testFetchPartitionedTopicMetadataWithCacheRefresh

([#24162](https://github.com/apache/pulsar/pull/24162)) \[fix]\[test] Fix flaky BrokerServiceChaosTest

([#24150](https://github.com/apache/pulsar/pull/24150)) \[fix]\[broker] The feature brokerDeleteInactivePartitionedTopicMetadataEnabled leaves orphan topic policies and topic schemas

([#24158](https://github.com/apache/pulsar/pull/24158)) \[fix]\[proxy] Propagate client connection feature flags through Pulsar Proxy to Broker

78d51f91ea Bump version to next snapshot version

([#24155](https://github.com/apache/pulsar/pull/24155)) \[fix]\[build] Fix docker image building by replacing deprecated and removed compress argument

([#23831](https://github.com/apache/pulsar/pull/23831)) \[fix]\[misc]: ignore deleted ledger when tear down cluster

([#24097](https://github.com/apache/pulsar/pull/24097)) \[fix] \[broker] topics infinitely failed to delete after remove cluster from replicated clusters modifying when using partitioned system topic

([#24117](https://github.com/apache/pulsar/pull/24117)) \[improve]\[broker] extract getMaxEntriesInThisBatch into a method and add unit test for it

([#24122](https://github.com/apache/pulsar/pull/24122)) \[improve]\[fn] Introduce NewOutputMessageWithError to enable error handling

([#24134](https://github.com/apache/pulsar/pull/24134)) \[fix]\[test] Fix flaky NonPersistentTopicTest.testMsgDropStat

([#24132](https://github.com/apache/pulsar/pull/24132)) \[fix]\[io] Fix KinesisSink json flattening for AVRO's SchemaType.BYTES

### KoP

Fix DeadLock issue when resolving the reference schema

### StreamNative Pulsar Plugins

fix(detector): enable pprof by default

\[graceful-rollout] set unload retry max

feat(detector): support disable pulsar protocol detection.

make integration test for all release branch

### Cloud Pulsar Plugins

Make sure the custom dynamic configuration interceptor can handle the multiple auth providers

### StreamNative Unified RBAC

Add ServiceAccount, ApiKey and Secret to SRN

### StreamNative Ursa storage

Skip topics in pulsar tenant in compaction service

fix deadlock from EntryIndexCache.invalidate

Fix compaction service generates a lot of small parquet files

Add Iceberg bigquery metastore catalog support

Fix publish thread executor block issue.

Skip some tests to speed up integration test

Support streamnative Delta External table.

Fix s3Table name format

Remove sn-bom build in CI

## Security Fixes

### Apache Pulsar

([#24140](https://github.com/apache/pulsar/pull/24140)) \[fix]\[sec] Upgrade jwt/v5 to 5.2.2 to address CVE-2025-30204

([#24135](https://github.com/apache/pulsar/pull/24135)) \[fix]\[sec] Upgrade pulsar-function-go dependencies to address CVE-2025-22870


# V4.0.4.2
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.4.2



# StreamNative Weekly Release Notes v4.0.4.2

## Download

### Distributions

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.4.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.4.2/images/sha256-7df5a4efcf5018b8a31a8e106049736ec8deb1609d1eb2084f77851d532a264a)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.4.2/images/sha256-ddba48fb9adb9e343b287c7db2aaa27f68706a73f7b2d78ab2998c017f7de4fb)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.4.2/images/sha256-ddba48fb9adb9e343b287c7db2aaa27f68706a73f7b2d78ab2998c017f7de4fb)

## General Changes

### Apache Pulsar

([#24246](https://github.com/apache/pulsar/pull/24246)) \[fix]\[build] Fix errorprone maven profile configuration

([#24243](https://github.com/apache/pulsar/pull/24243)) \[improve]\[build] Upgrade SpotBugs to 4.9.x

([#24239](https://github.com/apache/pulsar/pull/24239)) \[improve]\[misc] Migrate from multiple nullness annotation libraries to JSpecify annotations

([#24240](https://github.com/apache/pulsar/pull/24240)) \[improve]\[build] Upgrade to jacoco 0.8.13

([#24236](https://github.com/apache/pulsar/pull/24236)) \[improve] Adapt startup scripts for Java 24 changes

([#24242](https://github.com/apache/pulsar/pull/24242)) \[improve]\[build] Upgrade errorprone to 2.38.0

([#24237](https://github.com/apache/pulsar/pull/24237)) \[improve]\[build] Upgrade Lombok to 1.18.38 to support JDK 24

([#24235](https://github.com/apache/pulsar/pull/24235)) \[fix]\[test] Fix resource leaks in PulsarBrokerStarterTest

([#24221](https://github.com/apache/pulsar/pull/24221)) \[improve]\[io] support kafka connect transforms and predicates

([#24230](https://github.com/apache/pulsar/pull/24230)) \[improve]\[client]Improve transaction log when a TXN command timeout

([#24223](https://github.com/apache/pulsar/pull/24223)) \[fix]\[broker] Orphan schema after disabled a cluster for a namespace

([#24228](https://github.com/apache/pulsar/pull/24228)) \[fix]\[broker] Fix ByteBuf memory leak in REST API for publishing messages

([#24184](https://github.com/apache/pulsar/pull/24184)) \[fix]\[client] Fix incorrect producer.getPendingQueueSize due to incomplete queue implementation

([#24219](https://github.com/apache/pulsar/pull/24219)) \[improve]\[broker]Improve the feature "Optimize subscription seek (cursor reset) by timestamp": search less entries

([#24214](https://github.com/apache/pulsar/pull/24214)) \[improve] Upgrade Netty to 4.1.121.Final

([#24212](https://github.com/apache/pulsar/pull/24212)) \[fix]\[test] Fix flaky BatchMessageWithBatchIndexLevelTest.testBatchMessageAck

([#24218](https://github.com/apache/pulsar/pull/24218)) \[fix]\[test] Fix multiple resource leaks in tests

([#24187](https://github.com/apache/pulsar/pull/24187)) \[improve]\[client] validate ClientConfigurationData earlier to avoid resource leaks

([#24216](https://github.com/apache/pulsar/pull/24216)) \[fix]\[broker] Fix HealthChecker deadlock in shutdown

([#24209](https://github.com/apache/pulsar/pull/24209)) \[fix]\[broker] Fix tenant creation and update with null value

([#24192](https://github.com/apache/pulsar/pull/24192)) \[fix]\[admin] Backlog quota's policy is null which causes a NPE

([#24205](https://github.com/apache/pulsar/pull/24205)) \[improve] Upgrade Apache Commons library versions to compatible versions

([#24210](https://github.com/apache/pulsar/pull/24210)) \[fix]\[broker] Fix broker shutdown delay by resolving hanging health checks

([#24207](https://github.com/apache/pulsar/pull/24207)) \[fix]\[broker] Fix compaction service log's wrong condition

([#24204](https://github.com/apache/pulsar/pull/24204)) \[fix]\[test] Fix resource leaks in ProxyTest and fix invalid tests

([#24201](https://github.com/apache/pulsar/pull/24201)) \[improve]\[io] Upgrade Kafka client and compatible Confluent platform version

([#24118)](https://github.com/apache/pulsar/pull/24118))) Revert "\[fix]\[broker] Add topic consistency check

([#24154)](https://github.com/apache/pulsar/pull/24154))) Revert "\[fix]\[broker] Directly query single topic existence when the topic is partitioned

([#24194](https://github.com/apache/pulsar/pull/24194)) \[fix]\[test] Fix flaky NamespacesTest.testNamespacesApiRedirects

([#24196](https://github.com/apache/pulsar/pull/24196)) \[fix]\[broker] Fix NPE from the wrong iterator in the ownership cleanup job(ExtensibleLoadManagerImpl only)

([#24151](https://github.com/apache/pulsar/pull/24151)) \[fix]\[broker] Fix cluster level OffloadedReadPriority to bookkeeper-first does not work

([#24186](https://github.com/apache/pulsar/pull/24186)) \[fix]\[broker] Fixes Inconsistent ServiceUnitStateData View (ExtensibleLoadManagerImpl only)

([#24103](https://github.com/apache/pulsar/pull/24103)) \[fix]\[schema] Reject unsupported Avro schema types during schema registration

([#24181](https://github.com/apache/pulsar/pull/24181)) \[fix]\[proxy] Fix incorrect client error when calling get topic metadata

([#24091](https://github.com/apache/pulsar/pull/24091)) \[fix]\[broker] Fix some problems in calculate totalAvailableBookies in method getExcludedBookiesWithIsolationGroups when some bookies belongs to multiple isolation groups.

([#24174](https://github.com/apache/pulsar/pull/24174)) \[fix]\[test] Fix remaining UnfinishedStubbingException issue with AuthZTests

([#24171](https://github.com/apache/pulsar/pull/24171)) \[improve]\[test] Use configured session timeout for MockZooKeeper and TestZKServer in PulsarTestContext

([#24172](https://github.com/apache/pulsar/pull/24172)) \[fix]\[test] Improve reliability of IncrementPartitionsTest

([#24168](https://github.com/apache/pulsar/pull/24168)) \[fix]\[build] Fix skipTag and use explicit tag for image name

([#24170](https://github.com/apache/pulsar/pull/24170)) \[fix]\[test]flaky-test:ManagedLedgerInterceptorImplTest.testManagedLedgerPayloadInputProcessorFailure

([#23980](https://github.com/apache/pulsar/pull/23980)) \[fix]\[broker] Consumer stuck when delete subscription \_\_compaction failed

([#24167](https://github.com/apache/pulsar/pull/24167)) \[fix]\[ml] Fix ML thread blocking issue in internalGetPartitionedStats API

([#24166](https://github.com/apache/pulsar/pull/24166)) \[fix]\[test] Fix invalid test CompactionTest.testDeleteCompactedLedgerWithSlowAck

([#24165](https://github.com/apache/pulsar/pull/24165)) \[fix]\[test] Fix UnfinishedStubbing issue in AuthZTests

([#24098](https://github.com/apache/pulsar/pull/24098)) \[fix]\[ml] Skip deleting cursor if it was already deleted before calling unsubscribe

([#24154](https://github.com/apache/pulsar/pull/24154)) \[fix]\[broker] Directly query single topic existence when the topic is partitioned

([#24118](https://github.com/apache/pulsar/pull/24118)) \[fix]\[broker] Add topic consistency check

([#24056](https://github.com/apache/pulsar/pull/24056)) \[fix]\[test] Update partitioned topic subscription assertions in IncrementPartitionsTest

([#24033](https://github.com/apache/pulsar/pull/24033)) \[cleanup]\[misc] Add override annotation

([#24180](https://github.com/apache/pulsar/pull/24180)) \[improve]\[build] Build apachepulsar/pulsar-io-kinesis-sink-kinesis\_producer with Alpine 3.21

### AoP

255d5d8 Add back qpid-test-utils package

### MoP

([#1731](https://github.com/streamnative/mop/pull/1731)) Fix user properties lost when enable authorization

### KoP

Support handle DESCRIBE\_LOG\_DIRS request

### StreamNative Pulsar Plugins

Fix the time ticker leak issue which caused high CPU usage

\[cluster-rollout] add orphan unload job cleanup logic

Update Go SDK rbac & oidc dependencies

Update x/net and go 1.24

Fix jlink command's compress argument

### pulsarctl

([#1790](https://github.com/streamnative/pulsarctl/pull/1790)) Add release workflow

([#1784](https://github.com/streamnative/pulsarctl/pull/1784)) Update to Go 1.24 to avoid CVEs

### Function Mesh Worker Service

exclude lz4-java for CVE reasons

Unified resource scale for all Objects and make it configurable

### StreamNative Unified RBAC

Support batch apply role bindings

### StreamNative Ursa storage

Fix topic quarantine bug

Implement new interface for Pulsar

Add default timeout for all integration tests

External table support wal compact

Disable expire Iceberg snapshot by default

Support fencing a managed ledger after closing

a4a8e721 Fix pom

Use nonRetriableQuaratine for task publish

Improve the compaction worker handle stream task logic.

Flaky test in SimpleStorageImplTest.java

\[improve] Use ReentrantReadWriteLock in  EntryCache

Move out the integration test containers to a new module

Add metrics for the read cache size in bytes

Fix the NPE when getting the non-exists blob

Fix the aws config socket time not work issue.

Add more netty config

Reduce primitive schema retry times

Format compaction service quarantine logic

Support configure iceberg table properties with topic properties

Throw exception when read empty entries in lakehouse worker.

\[fix] throw exception upon EntryCache apis calls after EntryCache is closed

Move the persistStorageApi initialize in the common place

Fix real offset incorrect

Fix semaphore not release bug

Fix read lock not release bug

Fix build failure

GCS supports delete with lifecycle

Throw exceptions when handle the recursive schema

throw exception when hitting non-binary index while processing RAW type

Support azure to delete object using lifecycle rules

Delete the compact task when compaction worker read compacted wal file.

Quarantine topic when get topic failed or get topic metadata failed

## Security Fixes

### Apache Pulsar

([#24232](https://github.com/apache/pulsar/pull/24232)) \[fix]\[sec] Upgrade Jetty to 9.4.57.v20241219 to mitigate CVE-2024-6763


# V4.0.4.3
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.4.3



# StreamNative Weekly Release Notes v4.0.4.3

## Download

### Distributions

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.4.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.4.3/images/sha256-41cf5efafce6fba56d0e60d6c2a00a49ef74cf3161932f9426c6ce9bf922a3cb)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.4.3/images/sha256-ec1c6fcfb9553d06f7c5dc61f09b913243bdb249bbeadc260e398008b102b643)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.4.3/images/sha256-ec1c6fcfb9553d06f7c5dc61f09b913243bdb249bbeadc260e398008b102b643)

## General Changes

### Apache Pulsar

([#23594](https://github.com/apache/pulsar/pull/23594)) \[fix] \[broker] No longer allow creating subscription that contains slash

([#24384](https://github.com/apache/pulsar/pull/24384)) \[fix]\[ml]Revert a behavior change of releasing idle offloaded ledger handle: only release idle BlobStoreBackedReadHandle

([#24397](https://github.com/apache/pulsar/pull/24397)) \[improve]\[misc] Upgrade Netty to 4.1.122.Final and tcnative to 2.0.72.Final

([#24391](https://github.com/apache/pulsar/pull/24391)) \[improve]\[broker] Add managedCursor/LedgerInfoCompressionType settings to broker.conf

([#24392](https://github.com/apache/pulsar/pull/24392)) \[improve]\[broker] Make maxBatchDeletedIndexToPersist configurable and document other related configs

([#24356](https://github.com/apache/pulsar/pull/24356)) \[fix]\[client] Fix consumer not returning encrypted messages on decryption failure with compression enabled

([#24386](https://github.com/apache/pulsar/pull/24386)) \[improve]\[broker] Added synchronized for sendMessages in Non-Persistent message dispatchers

([#24381](https://github.com/apache/pulsar/pull/24381)) \[improve]\[ml]Release idle offloaded read handle only the ref count is 0

([#24366](https://github.com/apache/pulsar/pull/24366)) \[fix]\[broker]Fix deadlock when compaction and topic deletion execute concurrently

([#19783](https://github.com/apache/pulsar/pull/19783)) \[improve]\[offloaders] Automatically evict Offloaded Ledgers from memory

([#24360](https://github.com/apache/pulsar/pull/24360)) \[fix]\[broker] expose consumer name for partitioned topic stats

([#24350](https://github.com/apache/pulsar/pull/24350)) \[fix]\[broker] Fix issue that topic policies was deleted after a sub topic deleted, even if the partitioned topic still exists

([#24354](https://github.com/apache/pulsar/pull/24354)) \[fix]\[io] Acknowledge RabbitMQ message after processing the message successfully

([#24352](https://github.com/apache/pulsar/pull/24352)) \[fix]\[broker] Ignore metadata changes when broker is not in the Started state

([#24346](https://github.com/apache/pulsar/pull/24346)) \[improve]\[broker] Enable concurrent processing of pending read Entries to avoid duplicate Reads

([#24190](https://github.com/apache/pulsar/pull/24190)) \[fix]\[broker] Resolve the issue of frequent updates in message expiration deletion rate

([#24338](https://github.com/apache/pulsar/pull/24338)) \[fix]\[ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent issue

([#24331](https://github.com/apache/pulsar/pull/24331)) \[fix]\[offload] Complete the future outside of the reading loop in BlobStoreBackedReadHandleImplV2.readAsync

([#24320](https://github.com/apache/pulsar/pull/24320)) \[fix]\[cli] Fix pulsar-shell cannot produce message with quotes and space

([#24324](https://github.com/apache/pulsar/pull/24324)) \[fix]\[test] Fix flaky AutoScaledReceiverQueueSizeTest.testNegativeClientMemory

([#24316](https://github.com/apache/pulsar/pull/24316)) \[fix]\[io] Fix kinesis avro bytes handling

([#24365](https://github.com/apache/pulsar/pull/24365)) \[improve] Enable metrics for all broker caches

([#24359](https://github.com/apache/pulsar/pull/24359)) \[improve]\[broker]Improve the log when encountered in-flight read limitation

([#24344](https://github.com/apache/pulsar/pull/24344)) \[improve]\[ml] Offload ledgers without check ledger length

([#24286](https://github.com/apache/pulsar/pull/24286)) \[fix]\[broker]Non-global topic policies and global topic policies overwrite each other

([#24279](https://github.com/apache/pulsar/pull/24279)) \[fix]\[broker]Global topic policies do not affect after unloading topic and persistence global topic policies never affect

([#24314](https://github.com/apache/pulsar/pull/24314)) \[fix]\[test] Fix more resource leaks in tests

([#24315](https://github.com/apache/pulsar/pull/24315)) \[cleanup] Remove unused config `autoShrinkForConsumerPendingAcksMap`

([#24313](https://github.com/apache/pulsar/pull/24313)) \[fix]\[broker] Fix potential deadlock when creating partitioned topic

([#24293](https://github.com/apache/pulsar/pull/24293)) \[fix]\[broker] fix wrong method name checkTopicExists.

([#24309](https://github.com/apache/pulsar/pull/24309)) \[improve]\[cli] Make pulsar-perf termination more responsive by using Thread interrupt status

([#24307](https://github.com/apache/pulsar/pull/24307)) \[fix]\[build] Ensure that buildtools is Java 8 compatible and fix remaining compatibility issue

([#24304](https://github.com/apache/pulsar/pull/24304)) \[fix]\[test] Simplify BetweenTestClassesListenerAdapter and fix issue with BeforeTest/AfterTest annotations

([#24289](https://github.com/apache/pulsar/pull/24289)) \[improve]\[io] Add configuration parameter for disabling aggregation for Kinesis Producers

([#24302](https://github.com/apache/pulsar/pull/24302)) \[improve] Upgrade pulsar-client-python to 3.7.0 in Docker image

([#24299](https://github.com/apache/pulsar/pull/24299)) \[fix]\[test] Fix more Netty ByteBuf leaks in tests

([#24297](https://github.com/apache/pulsar/pull/24297)) \[fix]\[io] Fix SyntaxWarning in Pulsar Python functions

([#24282](https://github.com/apache/pulsar/pull/24282)) \[fix]\[client] Fix producer publishing getting stuck after message with incompatible schema is discarded

([#24283](https://github.com/apache/pulsar/pull/24283)) \[cleanup]\[test] Remove unused parameter from deleteNamespaceWithRetry method in MockedPulsarServiceBaseTest

([#24272](https://github.com/apache/pulsar/pull/24272)) \[improve]\[ci] Add Netty leak detection reporting to Pulsar CI

([#24277](https://github.com/apache/pulsar/pull/24277)) \[improve]\[build] Improve thread leak detector by ignoring "Attach Listener" thread

([#24263](https://github.com/apache/pulsar/pull/24263)) \[improve]\[build] Upgrade zstd version from 1.5.2-3 to 1.5.7-3

([#24281](https://github.com/apache/pulsar/pull/24281)) \[fix]\[test] Fix multiple ByteBuf leaks in tests

([#24278](https://github.com/apache/pulsar/pull/24278)) \[improve]\[build] Suppress JVM class sharing warning when running tests

([#24275](https://github.com/apache/pulsar/pull/24275)) \[fix]\[broker] Fix HashedWheelTimer leak in PulsarService by stopping it in shutdown

([#24274](https://github.com/apache/pulsar/pull/24274)) \[fix]\[misc] Fix ByteBuf leak in SchemaUtils

([#24273](https://github.com/apache/pulsar/pull/24273)) \[fix]\[misc] Fix ByteBuf leaks in tests by making ByteBufPair.coalesce release the input ByteBufPair

([#24270](https://github.com/apache/pulsar/pull/24270)) \[improve]\[build] Upgrade commons-compress version from 1.27.0 to 1.27.1

([#24268](https://github.com/apache/pulsar/pull/24268)) \[improve]\[build] Allow building and running tests on JDK 24 and upcoming JDK 25 LTS

([#24254](https://github.com/apache/pulsar/pull/24254)) \[fix]\[broker]Fix incorrect priority between topic policies and global topic policies

([#24266](https://github.com/apache/pulsar/pull/24266)) \[improve]\[ci] Disable detailed console logging for integration tests in CI

([#24241](https://github.com/apache/pulsar/pull/24241)) \[improve]\[build] Upgrade Mockito to 5.17.0 and byte-buddy to 1.15.11

([#24261](https://github.com/apache/pulsar/pull/24261)) \[fix]\[test] Fix flaky ManagedCursorTest.testLastActiveAfterResetCursor and disable failing SchemaTest

([#24244](https://github.com/apache/pulsar/pull/24244)) \[fix]\[test] Fix flaky ManagedCursorTest.testSkipEntriesWithIndividualDeletedMessages

([#24248](https://github.com/apache/pulsar/pull/24248)) \[improve]\[io]\[kca] support fully-qualified topic names in source records

([#24260](https://github.com/apache/pulsar/pull/24260)) \[improve]\[build] Upgrade Gradle Develocity Maven Extension dependencies

([#24258](https://github.com/apache/pulsar/pull/24258)) \[fix]\[test] Fix TestNG BetweenTestClassesListenerAdapter listener

([#24257](https://github.com/apache/pulsar/pull/24257)) \[fix]\[broker] Unregister non-static metrics collectors registered in Prometheus default registry

([#24251](https://github.com/apache/pulsar/pull/24251)) \[cleanup] Remove unused static fields in BrokerService

([#24252](https://github.com/apache/pulsar/pull/24252)) \[cleanup] remove unused config messagePublishBufferCheckIntervalInMillis

([#24249](https://github.com/apache/pulsar/pull/24249)) \[fix] chore: remove unused preciseTopicPublishRateLimiterEnable

([#24178](https://github.com/apache/pulsar/pull/24178)) \[fix]\[broker]fix memory leak, messages lost, incorrect replication state if using multiple schema versions(auto\_produce)

### MoP

([#1743](https://github.com/streamnative/mop/pull/1743)) Fix listener error

([#1734](https://github.com/streamnative/mop/pull/1734)) fix topic authentication issue

### KoP

Change the default value of the kopAllowedNamespaces

### StreamNative Pulsar Plugins

dbe501f9e Fix the issues when releasing 4.0.4.3

eb792c253 Return empty when getting global topic policies instead of returning exception

Upgrade commons-beanutils version to fix CVE-2025-48734

Add test to verify the sts module

Fix the backup tool can not use sts to authenticate

### pulsarctl

([#1809](https://github.com/streamnative/pulsarctl/pull/1809)) Use snstage org image

([#1790)](https://github.com/streamnative/pulsarctl/pull/1790))) ([#1793](https://github.com/streamnative/pulsarctl/pull/1793)) Revert "Add release workflow

([#1790](https://github.com/streamnative/pulsarctl/pull/1790)) Add release workflow

([#1784](https://github.com/streamnative/pulsarctl/pull/1784)) Update to Go 1.24 to avoid CVEs

([#1778](https://github.com/streamnative/pulsarctl/pull/1778)) Fix jwt cve

([#1762](https://github.com/streamnative/pulsarctl/pull/1762)) Fixed version

([#1734](https://github.com/streamnative/pulsarctl/pull/1734)) Fix json marshal error for Secrets when updating functions\&sinks\&sources

([#1758](https://github.com/streamnative/pulsarctl/pull/1758)) Upgrade x/net and x/crypto

([#1750](https://github.com/streamnative/pulsarctl/pull/1750)) Upgrade jwt to 4.5.1

([#1711](https://github.com/streamnative/pulsarctl/pull/1711)) fix code check

([#1704](https://github.com/streamnative/pulsarctl/pull/1704)) feat: Subscription get message by id json output

([#1699](https://github.com/streamnative/pulsarctl/pull/1699)) Update subscription get message by id typo lederId to ledgerId

### Cloud Pulsar Plugins

30cfe88 Fix build issue for releasing 4.0.4.3

### Function Mesh Worker Service

Support invalid name

Fix resource error during update and get connectors

Fix trigger function not support partitioned input topics error

Find specified ServiceAccount using oauth2's client role and use it when exist

validate function-mesh v0.24.1

### StreamNative Tiered storage

a1840fb3 Fix the compile issue

### StreamNative Unified RBAC

Use RoleBinding name to save bindings in metadata

\[branch-4.0] Fix license

Fix kafka permission format

Add KSN permissions

Upgrade project and sdk/java to 1.6.0

Add kafka condition

Reduce rbac filter timeout to 3 seconds by default and make it configurable

### StreamNative Ursa storage

Fail back to normal config when the external config miss.

Optimize entry reader read batch

Allow to configure the http client used by Azure

Fix build script typo

Support pulsar entry write into / read from the parquet file process

\[doc] WAL Cloud Storage Developer Guide

Update Metrics.md with accurate and concise descriptions

## Security Fixes


# V4.0.5.1
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.5.1



# StreamNative Weekly Release Notes v4.0.5.1

## Download

### Distributions

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.5.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.5.1/images/sha256-d6be7354c140ac02d477f8ec1dcad8e431385454bd4a7a52f4a10fcd045f3f52)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.5.1/images/sha256-df4f0cfdbd5947d8ffcea4212ddcc315d638c068405c90cc153e78165e5bb6ba)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.5.1/images/sha256-df4f0cfdbd5947d8ffcea4212ddcc315d638c068405c90cc153e78165e5bb6ba)

## General Changes

### Apache Pulsar

([#23611)](https://github.com/apache/pulsar/pull/23611))) ([#24429](https://github.com/apache/pulsar/pull/24429)) \[fix]\[broker]\[branch-4.0] Revert "\[improve]\[broker] Reduce memory occupation of the delayed message queue

([#24401](https://github.com/apache/pulsar/pull/24401)) \[fix]\[txn] Fix deadlock when loading transaction buffer snapshot

### MoP

([#1756](https://github.com/streamnative/mop/pull/1756)) Fix subscription authorization PREFIX mode

### KoP

Reduce unnecessary time-consuming topic replay for producer state recovery

### Function Mesh Worker Service

Set default VPA by default when HPA is not enabled

### StreamNative Tiered storage

tune the test timeout

Upgrade the integration test images to 4.0.4.3

## Security Fixes


# V4.0.5.2
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.5.2



# StreamNative Weekly Release Notes v4.0.5.2

## Download

### Distributions

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.5.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.5.2/images/sha256-cf457b6731508c53771eb3cf00412467635b7ff7266237155e80f7e470ac65d0)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.5.2/images/sha256-70ff9754251d521a5585b6fc7934fce23c0d47e2cfea22e7c87423f365fe337e)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.5.2/images/sha256-70ff9754251d521a5585b6fc7934fce23c0d47e2cfea22e7c87423f365fe337e)

## General Changes

### Apache Pulsar

([#24432](https://github.com/apache/pulsar/pull/24432)) \[fix]\[ml]Still got BK ledger, even though it has been deleted after offloaded

([#24351](https://github.com/apache/pulsar/pull/24351)) \[improve]\[broker] Deny removing local cluster from topic level replicated cluster policy

([#24419](https://github.com/apache/pulsar/pull/24419)) \[fix]\[broker] Once the cluster is configured incorrectly, the broker maintains the incorrect cluster configuration even if you removed it

([#24404](https://github.com/apache/pulsar/pull/24404)) \[fix]\[client] Prevent NPE when seeking with null topic in TopicMessageId

([#24405](https://github.com/apache/pulsar/pull/24405)) \[fix]\[ml]Received more than once callback when calling cursor.delete

([#24406](https://github.com/apache/pulsar/pull/24406)) \[fix]\[ml] Cursor ignores the position that has an empty ack-set if disabled deletionAtBatchIndexLevelEnabled

([#24407](https://github.com/apache/pulsar/pull/24407)) \[fix]\[broker] Fix the wrong cache name

([#24402](https://github.com/apache/pulsar/pull/24402)) \[fix]\[client] Fix some potential resource leak

871daf9515 Bump version to next snapshot version

### KoP

Update to Oxia 0.6.0 and use new group-id

### pulsarctl

Fix go cve CVE-2025-22874

### StreamNative Unified RBAC

Use RoleBinding name to save bindings in metadata

## Security Fixes


# V4.0.5.3
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.5.3



# StreamNative Weekly Release Notes v4.0.5.3

## Download

### Distributions

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.5.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.5.3/images/sha256-af1ad58a4fc559dfc7c323dd611d79ec10b833e4369b9654c7ba09fe4c98ceef)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.5.3/images/sha256-42fa8587beaa35cc29bd8b4e7f001d566f8fd2e5788379d70a71356cfb70c548)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.5.3/images/sha256-42fa8587beaa35cc29bd8b4e7f001d566f8fd2e5788379d70a71356cfb70c548)

## General Changes

### Apache Pulsar

([#24443](https://github.com/apache/pulsar/pull/24443)) \[fix]\[txn] Fix negative unacknowledged messages in transactions by ensuring that the batch size is added into CommandAck

([#24421](https://github.com/apache/pulsar/pull/24421)) \[fix]\[build] Add missing name to submodules

([#24459](https://github.com/apache/pulsar/pull/24459)) \[improve]\[broker] change to warn log level for ack validation error

([#24441](https://github.com/apache/pulsar/pull/24441)) \[fix]\[ml] Enhance OpFindNewest to support skip non-recoverable data

([#24454](https://github.com/apache/pulsar/pull/24454)) \[improve]\[broker]\[branch-4.0] Update to Oxia 0.6.0 and use new group-id

([#24448](https://github.com/apache/pulsar/pull/24448)) \[refactor]\[broker] Expose the managedLedger field for the sub class

### KoP

Add lookup cache for transaction marker channel manager

### StreamNative Pulsar Plugins

Fix the build issue

Upgrade zk version to 3.9.3 to avoid CVE-2024-51504

Bump com.fasterxml.jackson.core:jackson-core from 2.13.0 to 2.15.0 in sn-pulsar-tool

### pulsarctl

([#1801](https://github.com/streamnative/pulsarctl/pull/1801)) Enhance GetStatsCmd to include additional options for backlog statistics

### StreamNative Tiered storage

\[Branch-4.0] Cherry pick #1051 #1061

### StreamNative Ursa storage

aed7cdec Use maven 3.9.9 settings

24cd9b7e Change command

036a9127 Change command

a1f57661 change interactiveMode mode

e6117b73 change interactiveMode mode

09d4c7c7 change interactiveMode mode

0680adc3 Add local maven settings to skip twitter downloading

b023c907 Add local maven settings to skip twitter downloading

Fix build script

Refactor pulsar external table

Support load pulsar client token from file

Use the existing resources to init lakehouse reader

Renable the pulsar e2e tests

Get ledger metadata from Oxia

31275451 fix cherry-pick issue

Change default entrySerDeType to PULSAR\_BATCHED\_RAW\_PARQUET

support register managedledger meatadata in oxia

Pulsar support reading from parquet

store messageId into parquet file

Store ledger metadata to oxia

Use jar instead of nar for offloader

Add serialization type in the metadata

Revert to use normal file as the index file

Support save pulsar entry without parsing batch

Support skip system topic and black topics

Fix the bookkeeperApi can not get the index by secondary key

Using MapFile to speed up the seek performance

Support deleting the compacted data

Update the offload flag according to the ursa storage state

Make pulsar compaction worker enable iceberg external table writer

Reuse the pulsar storage configuration for the pulsar offloader

Update metadata store ledgers info after checking offloaded flag.

Fix the prepare task name compatibility issue

\[cleanup] remove unused code in the managedLedgerWithTs

Support generate the Ursa offset when committing the task

Update to Oxia 0.6.0 and use new group-id

Make bookkeeperStorageApi implement the StorageApi

Pulsar offloader leader support

Support write without parse content with schema

Support compress the index files

Make the tasks in the CommitTaskProvider sorted by the stream id and start offset

Pulsar offloader support offload to iceberg format

Support read/write index file with hadoop lib to cloud storage

Fix pulsar lakehouse reader memory leak issue

Integration test for the pulsar protocol compaction process

Refactored to allow support for low-latency storage class

Adapt the comapction process with pulsar related code

Fix the NPE when serializing the bytes schema version

Use static instances for compression codecs

Pin version for commons-lang3

Support bookkeeper entry reader to let compaction service read from bookkeeper

Support read and write bookkeeper entries

Introduce the api for bookkeeper generate offsets

Support trimming topic with the mark-deleted-offset

## Security Fixes


# V4.0.5.4
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.5.4



# StreamNative Weekly Release Notes v4.0.5.4

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.5.4](https://github.com/streamnative/pulsar/releases/tag/v4.0.5.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.5.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.5.4/images/sha256-9a2b87abd78018a51841e3584fff4bbc89e8321b1c95528d88edc4c69c52e38e)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.5.4/images/sha256-1e1759616d03040db80891d925451299f37ed140a56998378da0712c8bfb119d)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.5.4/images/sha256-1e1759616d03040db80891d925451299f37ed140a56998378da0712c8bfb119d)

## General Changes

### Apache Pulsar

([#24552](https://github.com/apache/pulsar/pull/24552)) \[improve]\[test] Remove EntryCacheCreator from ManagedLedgerFactoryImpl

([#24544](https://github.com/apache/pulsar/pull/24544)) \[improve] Upgrade pulsar-client-python to 3.8.0 in Docker image

([#24516](https://github.com/apache/pulsar/pull/24516)) \[fix]\[broker] Fix exclusive producer creation when last shared producer closes

([#24506](https://github.com/apache/pulsar/pull/24506)) \[fix]\[broker] Fix duplicate increment of ADD\_OP\_COUNT\_UPDATER in OpAddEntry

([#24543](https://github.com/apache/pulsar/pull/24543)) \[fix]\[broker] Fix matching of topicsPattern for topic names which contain non-ascii characters

([#24537](https://github.com/apache/pulsar/pull/24537)) \[fix]\[misc] Fix topics pattern consumer backwards compatibility

([#24539](https://github.com/apache/pulsar/pull/24539)) \[fix]\[client] Close orphan producer or consumer when the creation is interrupted

([#24521](https://github.com/apache/pulsar/pull/24521)) \[improve]\[client] Add `startTimestamp` and `endTimestamp` for consuming message in client cli

([#24542](https://github.com/apache/pulsar/pull/24542)) \[fix]\[broker]Fix thread safety issues in BucketDelayedDeliveryTracker with StampedLock optimistic reads

([#24533](https://github.com/apache/pulsar/pull/24533)) \[fix]\[ws] Fix WebSocket authorization issue due to originalPrincipal must be provided

([#24517](https://github.com/apache/pulsar/pull/24517)) \[fix]\[client] Fix ClientCnx handleSendError NPE

([#24515](https://github.com/apache/pulsar/pull/24515)) \[fix]\[ml] Fix asyncReadEntries might never complete if empty entries are read from BK

([#24534](https://github.com/apache/pulsar/pull/24534)) \[fix]\[io] Fix Kinesis checkpoint mechanism to prevent data duplication

([#24530](https://github.com/apache/pulsar/pull/24530)) \[improve]\[misc] Upgrade RE2/J to 1.8

([#24518](https://github.com/apache/pulsar/pull/24518)) \[fix]\[broker] Fix wrong backlog age metrics when the mark delete position point to a deleted ledger

([#24525](https://github.com/apache/pulsar/pull/24525)) \[improve]\[misc] Optimize topic list hashing so that potentially large String allocation is avoided

([#24528](https://github.com/apache/pulsar/pull/24528)) \[fix]\[client] Fix issue in auto releasing of idle connection with topics pattern consumer

([#24529](https://github.com/apache/pulsar/pull/24529)) \[fix]\[proxy] Fix default value of connectionMaxIdleSeconds in Pulsar Proxy

([#24519](https://github.com/apache/pulsar/pull/24519)) \[improve]\[test] Add test for concurrent processing of pending read Entries

([#24514](https://github.com/apache/pulsar/pull/24514)) \[improve]\[build] Bump org.apache.commons:commons-lang3 from 3.17.0 to 3.18.0

([#24492](https://github.com/apache/pulsar/pull/24492)) \[improve]\[ci] Fixes #23079: Checkstyle checks applied to all test

([#24468](https://github.com/apache/pulsar/pull/24468)) \[improve]\[broker] Upgrade bookkeeper to 4.17.2/commons-configuration to 2.x/grpc to 1.72.0 and enable ZooKeeper client to establish connection in read-only mode

([#24473](https://github.com/apache/pulsar/pull/24473)) \[improve]\[build] replace org.apache.commons.lang to org.apache.commons.lang3

([#24465](https://github.com/apache/pulsar/pull/24465)) \[fix]\[proxy] Fix proxy OOM by replacing TopicName with a simple conversion method

([#24434](https://github.com/apache/pulsar/pull/24434)) \[improve]\[broker] Improve the log when namespace bundle is not available

([#24472](https://github.com/apache/pulsar/pull/24472)) \[fix] Prevent IllegalStateException: Field 'message' is not set

([#24476](https://github.com/apache/pulsar/pull/24476)) \[fix]\[client] NPE in MultiTopicsConsumerImpl.negativeAcknowledge

([#24512](https://github.com/apache/pulsar/pull/24512)) \[fix]\[broker] Fix NPE when getting delayed delivery policy

([#24478](https://github.com/apache/pulsar/pull/24478)) \[fix]\[broker]Data lost due to conflict loaded up a topic for two brokers, when enabled ServiceUnitStateMetadataStoreTableViewImpl

([#24501](https://github.com/apache/pulsar/pull/24501)) \[fix]\[io] Fix data loss issue in Kinesis source connector

([#24495](https://github.com/apache/pulsar/pull/24495)) \[fix]\[io] Make record properties configurable for kinesis source

([#24453](https://github.com/apache/pulsar/pull/24453)) \[fix]\[broker] replication does not work due to the mixed and repetitive sending of user messages and replication markers

([#24424](https://github.com/apache/pulsar/pull/24424)) \[fix]\[broker] Fix the non-persistenttopic's replicator always get error "Producer send queue is full" if set a small value of the config replicationProducerQueueSize

([#24189](https://github.com/apache/pulsar/pull/24189)) \[fix]\[broker]excessive replication speed leads to error: Producer send queue is full

### KoP

Fix consumer close might be stuck when SyncGroup is in progress

### StreamNative Pulsar Plugins

Upgrade pulsar placement policy bk dependency to 4.17.2

### Cloud Pulsar Plugins

\[ApiKeys] Don't print full stacks for authentication failure

### Function Mesh Worker Service

Set minReplicas to parallelism when HPA is enabled

### StreamNative Tiered storage

3ed593d8 Fix integration test for branch-4.0

### StreamNative Unified RBAC

Use `GET_BUNDLE` operation to check the "get" permission for namespace

### StreamNative Ursa storage

Fix CI

Upgrade delta kernel to 4.0.0

Make the RawReader object pooled to avoid creating each time

Delete the compact task if the topic doesn't exist

\[refactor] Only open one parquet file for the lakehouse reader

Fix the metrics tests

Using the GlobalOpenTelemetry to register the reader metrics

Fix reader read failed in parallel

Delta external table introduce temporary credential

Introduce CustomKernelParquetWriter to support put write mode to improve memory usage

Fix the task compatibility issue.

Fix readIndexes bug

Fix the typo for the classname of PreparedCompactStreamTask

Introduce the task type to control the compaction handling

Add offload cursor to block data expire

Send the failed parse messages into a failure topic

Load configuration from the pulsar-client.conf file

Close the catalog resrouce after using

Delete the compaction task if it compacting the expired data

Add ML Cloud Storage Developer Guide

Reenable the pulsarE2ETest

Speed up the CI process from 1 hour to 20mins by separating to the different groups

Adapt for unity iceberg rest api

Fix the offloaded ledgermetadata is not synced with the offload state

chore: Add Claude Code Agents

Allow to disable sync UrsaMLMetadata

Support delete ledger from pulsar offload handler

Add metrics for pulsar read/writer lakehouse path

Redirect maven twitter repo to central

## Security Fixes

### Apache Pulsar

([#24547](https://github.com/apache/pulsar/pull/24547)) \[fix]\[sec] Upgrade pulsar-function-go dependencies to address CVE-2025-22868


# V4.0.5.5
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.5.5



# StreamNative Weekly Release Notes v4.0.5.5

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.5.5](https://github.com/streamnative/pulsar/releases/tag/v4.0.5.5)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.5.5/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.5.5/images/sha256-d155ff953e7e951f86b73441754a6a480b813394879d930ce790eb67e3b830cf)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.5.5/images/sha256-fa9db9a3e62c89f5fad2711a36c70dff55990bd67b4e96ae5239874728c124ce)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.5.5/images/sha256-fa9db9a3e62c89f5fad2711a36c70dff55990bd67b4e96ae5239874728c124ce)

## General Changes

### Apache Pulsar

([#24597](https://github.com/apache/pulsar/pull/24597)) \[improve]\[client] RawReader support pause and resume

([#24450](https://github.com/apache/pulsar/pull/24450)) \[fix]\[broker] Fix REST API to produce messages to single-partitioned topics

([#24570](https://github.com/apache/pulsar/pull/24570)) \[fix]\[broker] Fix NPE being logged if load manager class name is blank

([#24593](https://github.com/apache/pulsar/pull/24593)) \[fix]\[broker] Fix incorrect AuthData passed to AuthorizationService in proxy scenarios

([#24489](https://github.com/apache/pulsar/pull/24489)) \[improve]\[io] Add support for the complete KinesisProducerConfiguration in KinesisSinkConfig

([#24535](https://github.com/apache/pulsar/pull/24535)) \[improve]\[test] Add test for dead letter topic with max unacked messages blocking

([#24532](https://github.com/apache/pulsar/pull/24532)) \[fix]\[misc] Upgrade dependencies to fix critical security vulnerabilities

([#24542](https://github.com/apache/pulsar/pull/24542)) \[fix]\[broker]Fix thread safety issues in BucketDelayedDeliveryTracker with StampedLock optimistic reads

([#24520](https://github.com/apache/pulsar/pull/24520)) \[improve]\[broker] Extract duplication in AbstractTopic#incrementTopicEpochIfNeeded

([#24582](https://github.com/apache/pulsar/pull/24582)) \[improve]\[client] Support load RSA PKCS#8 private key

([#24591](https://github.com/apache/pulsar/pull/24591)) \[fix]\[broker] Fix namespace deletion TLS URL selection for geo-replication

([#24586](https://github.com/apache/pulsar/pull/24586)) \[improve]\[test] Refactor the way way pulsar-io-debezium-oracle nar file is patched when building the test image

([#24590](https://github.com/apache/pulsar/pull/24590)) \[fix]\[broker] Fix flaky testReplicatorsInflightTaskListIsEmptyAfterReplicationFinished

([#24542)](https://github.com/apache/pulsar/pull/24542))) Revert "\[fix]\[broker]Fix thread safety issues in BucketDelayedDeliveryTracker with StampedLock optimistic reads

([#24554](https://github.com/apache/pulsar/pull/24554)) ([#24571](https://github.com/apache/pulsar/pull/24571)) \[fix]\[client]\[branch-4.0] Partitioned topics are unexpectedly created by client after deletion

([#24576](https://github.com/apache/pulsar/pull/24576)) \[fix]\[test] fix flaky GrowableArrayBlockingQueueTest.testPollBlockingThreadsTermination

([#24569](https://github.com/apache/pulsar/pull/24569)) \[fix]\[broker] Fix ManagedCursor state management race conditions and lifecycle issues

([#24550](https://github.com/apache/pulsar/pull/24550)) \[improve]\[client] Terminate consumer.receive() when consumer is closed

([#24560](https://github.com/apache/pulsar/pull/24560)) \[fix]\[broker] Fix maxTopicsPerNamespace might report a false failure

([#24505](https://github.com/apache/pulsar/pull/24505)) \[fix]\[test]fix flaky test BrokerServiceAutoTopicCreationTest.testDynamicConfigurationTopicAutoCreationPartitioned

([#24551](https://github.com/apache/pulsar/pull/24551)) \[fix]\[broker] Fix Broker OOM due to too many waiting cursors and reuse a recycled OpReadEntry incorrectly

([#24511](https://github.com/apache/pulsar/pull/24511)) \[fix]\[broker] Fix deduplication replay might never complete for exceptions

([#24522](https://github.com/apache/pulsar/pull/24522)) \[fix]\[ml] Fix the possibility of message loss or disorder when ML PayloadProcessor processing fails

### AoP

Add qpid dependencies

### MoP

Fix MQTT message error handling and improve connection responses

### KoP

Do not set setReplicationClusters on createTopicIfNotExist

### StreamNative Pulsar Plugins

fix export duplicated JVM metrics on AuditLogMetrics

### Cloud Pulsar Plugins

Only append jwk when kty is rsa

### Function Mesh Worker Service

467ec4ba Update version when release

Implement agent function

### StreamNative Tiered storage

Fix build failure

### StreamNative Unified RBAC

\[feat] Extend functions\&connectors permissions

### StreamNative Ursa storage

Only publish BlobNotFound exception task to DLQ

Exclude OutOfMemoryError for DLQ

Support publish commit failed tasks to DLQ

Fix oxia lock leak when not acquired

Optimize Compaction Service heap memory usage

fix\[lock]: fixes memory leak on oxia distributed lock

Introduce catalog factory for iceberg

Add external table protobuf support for Ursa and Pulsar protocol

Make the pulsar compaction worker not record column stats.

Using bookkeeperStorageApi when configured pulsar client

Add compaction leader metric doc

Delete committed tasks and update oxia index

Fix the error handling

Remove unnecessary synchronized lock

fix: handle UUID logical type with string base type in AvroToIcebergConverter

Fix ConcurrentModificationException in CompactionTaskProvider.getTask

Cleanup stream when deleting unloaded topics

Add multi catalog user document

Add compaction service throguhput rate limiter for reading from BookKeeper

Support multiple catalog in namespace and topic level

Refactor update iceberg table properties

Separate managed and external writer for ursa

Add compaction leader metric

Add failure reason for the iceberg external writer

Optimize reset cursor

## Security Fixes

### Apache Pulsar

([#24562](https://github.com/apache/pulsar/pull/24562)) \[fix]\[sec] Remove dependency on out-dated commons-configuration 1.x

([#24564](https://github.com/apache/pulsar/pull/24564)) \[fix]\[sec] Upgrade Kafka connector and clients version to 3.9.1 to address CVE-2025-27818


# V4.0.6.1
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.6.1



# StreamNative Weekly Release Notes v4.0.6.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.6.1](https://github.com/streamnative/pulsar/releases/tag/v4.0.6.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.6.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.6.1/images/sha256-133e81b70f3123d06d602e085533e8bb1b086368942cd9acfee81937dade5361)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.6.1/images/sha256-e67bb42146ead9ed87167b551d22f99231f29a3e7ee353049393d8d53b795598)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.6.1/images/sha256-e67bb42146ead9ed87167b551d22f99231f29a3e7ee353049393d8d53b795598)

## General Changes

### Apache Pulsar

([#24601](https://github.com/apache/pulsar/pull/24601)) \[improve]\[doc] Improve the JavaDocs of sendAsync to avoid improper use

([#24599](https://github.com/apache/pulsar/pull/24599)) \[fix]\[client] Retry for unknown exceptions when creating a producer or consumer

### KoP

Auth SN github maven repo before claude review

### Function Mesh Worker Service

Support set agent tools config

Make MeshWorker able to run standalone and load additional servlets

Support load ConnectorCatalog using label

Update error msg in status

### StreamNative Unified RBAC

fix(misc): add some logs and missing output

### StreamNative Ursa storage

Refresh the catalog instance when using open catalog

Make update and delete compactTask async

Uniform all places configuration name for the data source type

Update the offload flag according to the each ledger state

Rename engineType to dataSourceForCompaction

Fix publish time

Fix committed task delete leak

Add engine type configuration

Fix deadlock by making getStreamId async in OffloadReadHandler

Fix task compability issue

Fix resource leak: close IndexFileReader in ParquetFileReader

## Security Fixes


# V4.0.6.2
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.6.2



## StreamNative Weekly Release Notes v4.0.6.2

#### General Changes

### Apache Pulsar

([#24602](https://github.com/apache/pulsar/pull/24602)) \[improve]\[broker]Part-2 Add Admin API to delete topic policies

([#24390](https://github.com/apache/pulsar/pull/24390)) \[improve]\[admin] PIP-422 part 1: Support global topic-level replicated clusters policy

([#24642](https://github.com/apache/pulsar/pull/24642)) \[fix]\[ws] Allow websocket principals to specify originalPrincipal without proxy role

([#24633](https://github.com/apache/pulsar/pull/24633)) \[fix]\[test]fix flaky ZeroQueueSizeTest.testZeroQueueGetExceptionWhenReceiveBatchMessage

([#24632](https://github.com/apache/pulsar/pull/24632)) \[fix]\[test] Fix ConcurrentModificationException in Ipv4Proxy

([#24617)](https://github.com/apache/pulsar/pull/24617))) Revert "\[improve]\[broker] Upgrade avro version to 1.12.0

([#24604](https://github.com/apache/pulsar/pull/24604)) \[improve]\[io] Add dependency file name information to error message when .nar file validation fails with ZipException

([#24617](https://github.com/apache/pulsar/pull/24617)) \[improve]\[broker] Upgrade avro version to 1.12.0

([#24606](https://github.com/apache/pulsar/pull/24606)) \[improve]\[broker]Remove block calling that named cursor.asyncGetNth when expiring messages

([#24607](https://github.com/apache/pulsar/pull/24607)) \[improve]\[broker]Improve the anti-concurrency mechanism expirationCheckInProgress

([#24615](https://github.com/apache/pulsar/pull/24615)) \[fix]\[ws] Fix WebSocket authentication with authenticateOriginalAuthData enabled

([#24613](https://github.com/apache/pulsar/pull/24613)) \[fix]\[ws] Fix WebSocket proxy originalPrincipal for HTTP admin API calls

([#24630](https://github.com/apache/pulsar/pull/24630)) \[fix]\[test]fix flaky ZeroQueueSizeTest.testZeroQueueGetExceptionWhenReceiveBatchMessage

([#24626](https://github.com/apache/pulsar/pull/24626)) \[fix]\[proxy] Fix TooLongFrameException with Pulsar Proxy

([#24621](https://github.com/apache/pulsar/pull/24621)) \[fix]\[broker] Fix duplicate watcher registration after SessionReestablished

([#24610](https://github.com/apache/pulsar/pull/24610)) \[fix]\[client]Prevent ZeroQueueConsumer from receiving batch messages when using MessagePayloadProcessor

### KoP

Fix Kafka Connect's topic replay loop might be stuck when all messages have been compacted out

Fix possible deadlock of system topic access due to blocking call when holding the lock

Fix incorrect ListOffsets result on a compacted topic

442d6967f \[branch-3.0] Bump version to 4.0.6.2

Add partition name to error logs in PartitionLog and UrsaPartitionLog classes

Fix retention.ms may overflow when converting to the Pulsar retention policy

### Function Mesh Worker Service

Create a new sub module mesh-worker-common

67fe201a Cleanup disk

### StreamNative Tiered storage

Introduce flag to control delta add file stats

Delta schema evolution support delete field

Unity catalog support update table schema.

### StreamNative Unified RBAC

fix(acl): avoid parsing token from data source

### StreamNative Ursa storage

43600e8f revert b3f2c62

([#1192)](https://github.com/streamnative/ursa-storage/pull/1192))) Revert "Adapt new changes for TopicCompactionService interface

Adapt new changes for TopicCompactionService interface

USe Hessian2 as the new task serialization.

Fix delete package task bug

Fix topic medata not found issue.

Fix: Use shared static thread pool for PulsarLakehouseReader idle timeout

Upgrade iceberg to 1.9.2

Unblock the compact process when encounter task deserialize exception

([#1192)](https://github.com/streamnative/ursa-storage/pull/1192))) Revert "Adapt new changes for TopicCompactionService interface

Adapt new changes for TopicCompactionService interface


# V4.0.6.3
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.6.3



# StreamNative Weekly Release Notes v4.0.6.3

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.6.3](https://github.com/streamnative/pulsar/releases/tag/v4.0.6.3)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.6.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.6.3/images/sha256-cdef556946ebfbbb6d1f75dc36aedda87a54c952e7af2287b68cfd5799eb3539)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.6.3/images/sha256-46485753cd8bbd429b9fbb745939afe920f6b32418d881d6ff084b1dd50b1108)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.6.3/images/sha256-46485753cd8bbd429b9fbb745939afe920f6b32418d881d6ff084b1dd50b1108)

## General Changes

### Apache Pulsar

([#24622](https://github.com/apache/pulsar/pull/24622)) \[improve]\[broker]Find the target position at most once, during expiring messages for a topic, even though there are many subscriptions

([#24651](https://github.com/apache/pulsar/pull/24651)) \[fix]\[broker]Failed to create partitions after the partitions were deleted because topic GC

([#24665](https://github.com/apache/pulsar/pull/24665)) \[fix]\[meta] Use `getChildrenFromStore` to read children data to avoid lost data

([#23977](https://github.com/apache/pulsar/pull/23977)) \[fix]\[broker] Invalid regex in PulsarLedgerManager causes zk data notification to be ignored

([#24427](https://github.com/apache/pulsar/pull/24427)) \[fix]\[broker] PIP-428: Fix corrupted topic policies issues with sequential topic policy updates

([#24663](https://github.com/apache/pulsar/pull/24663)) \[fix]\[client] Skip schema validation when sending messages to DLQ to avoid infinite loop when schema validation fails on an incoming message

([#24669](https://github.com/apache/pulsar/pull/24669)) \[improve]\[io] Support specifying Kinesis KPL native binary path with 1.0 version specific path

([#24668](https://github.com/apache/pulsar/pull/24668)) \[improve]\[build] Use org.apache.nifi:nifi-nar-maven-plugin:2.1.0 with skipDocGeneration=true

([#24666](https://github.com/apache/pulsar/pull/24666)) \[improve]\[build] Increase maven resolver's sync context timeout

([#24661](https://github.com/apache/pulsar/pull/24661)) \[improve]\[io] Upgrade AWS SDK v1 & v2, Kinesis KPL and KPC versions

([#24662](https://github.com/apache/pulsar/pull/24662)) \[fix]\[client] fix ArrayIndexOutOfBoundsException in SameAuthParamsLookupAutoClusterFailover

([#24659](https://github.com/apache/pulsar/pull/24659)) \[fix]\[misc] Upgrade fastutil to 8.5.16

([#24639](https://github.com/apache/pulsar/pull/24639)) \[fix]\[broker] Fix race condition in MetadataStoreCacheLoader causing inconsistent availableBroker list caching

([#24649](https://github.com/apache/pulsar/pull/24649)) \[fix]\[offload] Exclude unnecessary dependencies from tiered storage provider / offloader nar files

([#24643](https://github.com/apache/pulsar/pull/24643)) \[fix]\[broker] Add double-check for non-durable cursor creation

([#24655](https://github.com/apache/pulsar/pull/24655)) \[improve]\[ml] Optimize ledger opening by skipping fully acknowledged ledgers

### KoP

Remove dependencies with Confluent Community License

Ignore the read\_committed field in Ursa

026f463fb Bump version to 4.0.6.3 for new TopicPoliciesService interface

Resolve RBAC compatibility issue

Fix topic read authorization not applied for OffsetDelete requests

### StreamNative Pulsar Plugins

Add audit logging support for non-partitioned topic creation

### pulsarctl

Upgrade go to 1.24.6 to fix CVE-2025-47907

### Function Mesh Worker Service

Support set extra env for kafka connect

Support streamable http for AgentFunction and make trigger timeout value configurable

### StreamNative Ursa storage

Fix build issue with the 4.0.6.3

Use number to convert the digital type at AvroToGenericRowConvert

Fix branch-4.0 ci

Remove failed to deserilize log

Add more metrics

6a8ad6a1 fix ci

Run with different commit runner if task properties changed

Expose parquet file reader cache config

Fix NaN serialization issue in UrsaParquetFileWriter

Fix nested enum serialization failed

fix: Handle NPE when nested record default value has mismatched field names

Refactor the commit process to allow recreate commit runner

Pulsar worker support offload to delta

Remove tmate in CI

## Security Fixes


# V4.0.6.4
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.6.4



# StreamNative Weekly Release Notes v4.0.6.4

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.6.4](https://github.com/streamnative/pulsar/releases/tag/v4.0.6.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.6.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.6.4/images/sha256-921ca98b8ba08f80fe106cf2b3ff35e0435b357296d3deffc56be947051282c5)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.6.4/images/sha256-4e2a567c015138ba49da42b104b84f9736a292fd95804d979cd8750fad5b72d8)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.6.4/images/sha256-4e2a567c015138ba49da42b104b84f9736a292fd95804d979cd8750fad5b72d8)

## General Changes

### Apache Pulsar

([#24689](https://github.com/apache/pulsar/pull/24689)) \[feat]\[misc] upgrade oxia version to 0.6.2

([#24680](https://github.com/apache/pulsar/pull/24680)) \[fix]\[broker]\[branch-4.0]Can not access topic policies if topic partitions have not been created

([#24679](https://github.com/apache/pulsar/pull/24679)) \[fix]\[broker]Fix flaky test PartitionCreationTest.testCreateMissedPartitions

## Security Fixes


# V4.0.6.5
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.6.5



# StreamNative Weekly Release Notes v4.0.6.5

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.6.5](https://github.com/streamnative/pulsar/releases/tag/v4.0.6.5)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.6.5/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.6.5/images/sha256-8e78c7e131a32e4676c8ccd27b208f4f0329ad69b99c5d1c014c34c05551fd16)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.6.5/images/sha256-a0222c1d3c3db0d9a72d07421f786cd4153f813818b1fe7de821fc24cc00c92a)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.6.5/images/sha256-a0222c1d3c3db0d9a72d07421f786cd4153f813818b1fe7de821fc24cc00c92a)

## General Changes

### Apache Pulsar

([#24647](https://github.com/apache/pulsar/pull/24647)) \[improve]\[client] Add OpenTelemetry metrics for client memory buffer usage

([#24706](https://github.com/apache/pulsar/pull/24706)) \[fix]\[broker] Fix NPE and annotate nullable return values for ManagedCursorContainer

([#24696](https://github.com/apache/pulsar/pull/24696)) \[fix]\[broker]Fix dirty reading of namespace level offload thresholds

([#24594](https://github.com/apache/pulsar/pull/24594)) \[improve]\[build] Disable javadoc build failure

([#24634](https://github.com/apache/pulsar/pull/24634)) \[fix]\[broker]Dispatcher did unnecessary sort for recentlyJoinedConsumers and printed noisy error logs

([#24648](https://github.com/apache/pulsar/pull/24648)) \[fix]\[broker]User topic failed to delete after removed cluster because of failed delete data from transaction buffer topic

([#24683](https://github.com/apache/pulsar/pull/24683)) \[fix]\[broker]Fix the wrong logic of the test PartitionCreationTest.testCreateMissedPartitions

([#24719](https://github.com/apache/pulsar/pull/24719)) \[fix]\[broker] Fix memory leak when metrics are updated in a thread other than FastThreadLocalThread

([#23336](https://github.com/apache/pulsar/pull/23336)) \[fix]\[client] Fix ArrayIndexOutOfBoundsException when using SameAuthParamsLookupAutoClusterFailover

### KoP

Support broker side schema validation

Ursa: fix incorrect warn log when appending new messages

Correct token extraction for Kafka internal client

Remove useless authorization warning log

\[flaky-test] Fix ListConsumerGroupTest

### pulsarctl

Update stable version

### StreamNative Ursa storage

Add detailed label for the compaction commit and error metrics

Fix record type with instant field write failed issue.

Close the writer when there is exceptions

Pause the reader when release it

Split offload cursor update interval

Fix metrics name

Fix the external table commit state failed to check issue

Add more metrics for the error, commit time

Introduce exception code to improve the exception handling

Support disable the publish task by topic

upgrade oxia version to 0.6.2

feat: upgrade oxia version to 0.6.1

IcebergTable support schema evolution

Add test for #1253.

## Security Fixes

### Apache Pulsar

([#24717](https://github.com/apache/pulsar/pull/24717)) \[fix]\[sec] Upgrade Netty to 4.1.127.Final to address CVEs


# V4.0.6.6
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.6.6



# StreamNative Weekly Release Notes v4.0.6.6

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.6.6](https://github.com/streamnative/pulsar/releases/tag/v4.0.6.6)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.6.6/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.6.6/images/sha256-db7a2ca6858da32dd4fdaf47c26a71bd5e91941d3f7e74c659b3a74046ddd422)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.6.6/images/sha256-94e7ebbbdfd882bfb37b103775a66abd20f8f303dd3318c3f634789241d065cb)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.6.6/images/sha256-94e7ebbbdfd882bfb37b103775a66abd20f8f303dd3318c3f634789241d065cb)

## General Changes

### Apache Pulsar

([#24749](https://github.com/apache/pulsar/pull/24749)) \[fix] Exclude commons-lang dep from bookkeeper

([#24595](https://github.com/apache/pulsar/pull/24595)) \[fix]\[ci] Fix code coverage metrics in Pulsar CI

([#24743](https://github.com/apache/pulsar/pull/24743)) \[fix]\[client] Fix receiver queue auto-scale without memory limit

([#24742](https://github.com/apache/pulsar/pull/24742)) \[improve]\[build] Upgrade Apache Parent POM to version 35

([#24722](https://github.com/apache/pulsar/pull/24722)) \[fix]\[ml] Negative backlog & acked positions does not exist & message lost when concurrently occupying topic owner

([#24730](https://github.com/apache/pulsar/pull/24730)) \[fix]\[broker] Ensure KeyShared sticky mode consumer respects assigned ranges

([#24736](https://github.com/apache/pulsar/pull/24736)) \[fix]\[broker] Key\_Shared subscription doesn't always deliver messages from the replay queue after a consumer disconnects and leaves a backlog

([#24731](https://github.com/apache/pulsar/pull/24731)) \[fix]\[broker] Fix cannot shutdown broker gracefully by admin api

([#24735](https://github.com/apache/pulsar/pull/24735)) Revert "\[fix]\[broker] Key\_Shared subscription doesn't always deliver messages from the replay queue after a consumer disconnects and leaves a backlog"

([#24732](https://github.com/apache/pulsar/pull/24732)) \[fix]\[broker] Key\_Shared subscription doesn't always deliver messages from the replay queue after a consumer disconnects and leaves a backlog

([#24654](https://github.com/apache/pulsar/pull/24654)) \[fix]\[io] Improve Kafka Connect source offset flushing logic

([#24725](https://github.com/apache/pulsar/pull/24725)) \[fix]\[client] Avoid recycling the same ConcurrentBitSetRecyclable among different threads

([#24721](https://github.com/apache/pulsar/pull/24721)) \[feat]\[fn] Fallback to using `STATE_STORAGE_SERVICE_URL` in `PulsarMetadataStateStoreProviderImpl.init`

([#24580](https://github.com/apache/pulsar/pull/24580)) \[fix]\[broker]Fix never recovered metadata store bad version issue if received a large response from ZK

### StreamNative Pulsar Plugins

Fix LicenseAdditionalServletTest

Use snstage/pulsar image to integration test

Removed pinned version for nimbus-jose-jwt-9.37.2

Pin Netty version for bookie\_rackinfo

Fix BK exclusions to avoid commons-configuration and commons-langs deps

beeb69a7c fix image

Fix sn bom plugin typo

Adapt to the latest topic policies interface from PIP-428

### pulsarctl

Preserve tenant fields on partial update

upgrade client go version to 0.16.0

### Function Mesh Worker Service

feat: support multiple mcp servers

Support trigger agent function with properties

Update function-mesh version to v0.25.0 in pom.xml

Remove ConnectRestException from mesh-worker-common module

Support input-type-class and output-type-class arguments for Functions

### StreamNative Ursa storage

Use the azure latest image for testing

Remove parquet compression property from iceberg table

Fix topic properties can not be fetched on the serverless cluster

Fix cache reference staleness race condition in ObjectWalStorageImpl

Control sdt behavior by the properties

Fix offload cursor doesn't show in the consumer stats

Fix unity catalog can't convert decimal type issue.

Fix global open telemetry confict issue

## Security Fixes


# V4.0.6.7
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.6.7



# StreamNative Weekly Release Notes v4.0.6.7

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.6.7](https://github.com/streamnative/pulsar/releases/tag/v4.0.6.7)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.6.7/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.6.7/images/sha256-9a81b196b69abc1a89d44de62f9441961ff549a807cf782b818e7109aab9665d)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.6.7/images/sha256-9621cfce5d2a6e117406fc72d9da00eb3a7939218a06d43ea1a357701332f76e)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.6.7/images/sha256-9621cfce5d2a6e117406fc72d9da00eb3a7939218a06d43ea1a357701332f76e)

## General Changes

### Apache Pulsar

([#24779](https://github.com/apache/pulsar/pull/24779)) Bump org.apache.zookeeper:zookeeper from 3.9.3 to 3.9.4

([#24596](https://github.com/apache/pulsar/pull/24596)) \[improve]\[broker]Call scheduleAtFixedRateNonConcurrently for scheduled tasks, instead of scheduleAtFixedRate

([#24769](https://github.com/apache/pulsar/pull/24769)) \[fix]\[test] Flaky-test: BrokerServiceTest.testShutDownWithMaxConcurrentUnload

([#24764](https://github.com/apache/pulsar/pull/24764)) \[improve]\[build] Upgrade Mockito, AssertJ and ByteBuddy to fully support JDK25

([#24761](https://github.com/apache/pulsar/pull/24761)) \[fix]\[client] Exclude io.prometheus:simpleclient\_caffeine from client-side dependencies

([#24763](https://github.com/apache/pulsar/pull/24763)) \[improve]\[build] Upgrade Lombok to 1.18.42 to fully support JDK25

([#23634](https://github.com/apache/pulsar/pull/23634)) \[improve]\[broker] If there is a deadlock in the service, the probe should return a failure because the service may be unavailable

([#24738](https://github.com/apache/pulsar/pull/24738)) \[fix]\[broker] First entry will be skipped if opening NonDurableCursor while trimmed ledger is adding first entry.

([#24753](https://github.com/apache/pulsar/pull/24753)) \[fix]\[ml]Fix EOFException after enabled topics offloading

([#24772](https://github.com/apache/pulsar/pull/24772)) \[fix]\[misc] Fix compareTo contract violation for NamespaceBundleStats, TimeAverageMessageData and ResourceUnitRanking

([#24768](https://github.com/apache/pulsar/pull/24768)) \[improve]\[build] Upgrade SpotBugs to a version that supports JDK25

([#24767](https://github.com/apache/pulsar/pull/24767)) \[fix]\[ci] Fix CI for Java 25 including upgrade of Gradle Develocity Maven extension

([#24741](https://github.com/apache/pulsar/pull/24741)) \[fix]\[broker] Prevent unexpected recycle failure in dispatcher's read callback

([#24756](https://github.com/apache/pulsar/pull/24756)) \[fix]\[broker] Fix testServiceConfigurationRetentionPolicy unit test

([#24752](https://github.com/apache/pulsar/pull/24752)) \[fix]\[client] rollback TopicListWatcher retry behavior

([#24733](https://github.com/apache/pulsar/pull/24733)) \[improve]\[broker] Allow deletion of empty persistent topics regardless of retention policy

([#24698](https://github.com/apache/pulsar/pull/24698)) \[fix]\[client]TopicListWatcher not closed when calling PatternMultiTopicsConsumerImpl.closeAsync() method

### KoP

Add test cases for record schema validation with subject name strategy

Add more info to logs when the connection is closed

Change log level to warn for authentication failure

\[schema-registry] add option to disable compatibility mode configuration

Handle unexpected exception in decode for safe producer state recovery

Avoid blocking when the previous consumer closed without sending SyncGroup requests

Use new CompactedTopicUtils.asyncReadCompactedEntries API from apache/pulsar#24725

### StreamNative Pulsar Plugins

\[feat]\[topic-compaction-service] Implement clean expired  message during compaction

Fix gcs-connector cve

### Function Mesh Worker Service

Reject request when agent name is too long or sessionId is not valid

c1035c46 Allow to use long name for agent function

### StreamNative Tiered storage

Fix ci on the branch-4.0

### StreamNative Unified RBAC

feat: treat NotFound exception as success for deleting

feat: upgrade sdk-go version to v0.14.0

Add conditions for Catalog, CC and CE

feat: support extract variable claim from token

### StreamNative Ursa storage

fix: mask sensitive properties in CatalogKey toString to avoid credential leaks in logs

## Security Fixes


# V4.0.6.8
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.6.8



# StreamNative Weekly Release Notes v4.0.6.8

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.6.8](https://github.com/streamnative/pulsar/releases/tag/v4.0.6.8)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.6.8/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.6.8/images/sha256-87d05c59ec01bd3d9b101e38714ef5c0959ecd7355c1545097b8fb4cebfb0e7a)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.6.8/images/sha256-2324cc997112048ae8fdd5c93c0f0b6580a1f744826fef623639785c265b53b9)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.6.8/images/sha256-2324cc997112048ae8fdd5c93c0f0b6580a1f744826fef623639785c265b53b9)

## General Changes

### Apache Pulsar

([#24838](https://github.com/apache/pulsar/pull/24838)) \[fix]\[broker] Ensure LoadSheddingTask is scheduled after metadata service is available again

([#24841](https://github.com/apache/pulsar/pull/24841)) \[improve]\[ci] Upgrade GitHub Actions workflows to use ubuntu-24.04

([#24830](https://github.com/apache/pulsar/pull/24830)) \[fix]\[client] Fix getPendingQueueSize for PartitionedTopicProducerStatsRecorderImpl: avoid NPE and implement aggregation

([#24832](https://github.com/apache/pulsar/pull/24832)) \[fix] Fix mixed lookup/partition metadata requests causing reliability issues and incorrect responses

([#24829](https://github.com/apache/pulsar/pull/24829)) \[fix]\[broker] Allow intermittent error from topic policies service when loading topics

([#24785](https://github.com/apache/pulsar/pull/24785)) \[fix]\[broker] Fix incorrect topic loading latency metric and timeout might not be respected

([#24822](https://github.com/apache/pulsar/pull/24822)) \[fix]\[client] Make auto partitions update work for old brokers without PIP-344

([#24801](https://github.com/apache/pulsar/pull/24801)) \[improve]\[broker]Improve NamespaceService log that is printed when cluster was removed

([#24770](https://github.com/apache/pulsar/pull/24770)) \[fix]\[broker] Flaky-test: ExtensibleLoadManagerImplTest.testDisableBroker

([#24780](https://github.com/apache/pulsar/pull/24780)) \[improve]\[broker] Replace isServiceUnitActiveAsync with checkTopicNsOwnership

([#24824](https://github.com/apache/pulsar/pull/24824)) \[improve]\[ml] Upgrade Oxia client to 0.7.0

([#24812](https://github.com/apache/pulsar/pull/24812)) \[fix]\[build] Remove invalid profile in settings.xml that caused gpg signing to fail

([#24811](https://github.com/apache/pulsar/pull/24811)) \[fix]\[build] Fix maven deploy with maven-source-plugin 3.3.1

([#24813](https://github.com/apache/pulsar/pull/24813)) \[fix] Update gRPC to 1.75.0

([#24025](https://github.com/apache/pulsar/pull/24025)) \[improve] \[broker] Separate offload read and write thread pool

### AoP

32383ce Use commons-lang3

Fix configuration potential NPE in test

### MoP

89c72f0a Upgrade commons-lang to commons-lang3

### KoP

Make avro-maven-plugin version consistent with avro

823d5724d Exclude commons-lang dep

87f2c5247 Bump version to 4.0.6.8

Use new APIs from Oxia 0.7.0

\[Ursa] Remove oldest producers when the serialized producer state is too large

Support parse googles built in proto files on protobuf schema

Fix idempotent producer for classic engine

Improve topic loading time by skipping Pulsar message deduplication recovery

### StreamNative Pulsar Plugins

Upgrade commons-lang to commons-lang3

Upgrade Oxia to 0.7.0

Upgrade zk version to 3.9.4 to fix CVE

### Cloud Pulsar Plugins

Fix CacheMetricsCollector's package modified by upstream

eeb0457 Use commons-lang3

Make sure the custom dynamic configuration interceptor can handle the multiple auth providers

Only append jwk when kty is rsa

\[ApiKeys] Don't print full stacks for authentication failure

Change to use commons-lang3

### Function Mesh Worker Service

Use sn-operator to deploy pulsar cluster in CI

fix agent cannot update some fields error

Fix ci failure

fix build and license header

### StreamNative Tiered storage

disable protobuf schema check.

Upgrade aws sdk to 2.32.28 to keep sync with Pulsar

339997ca Use lang3 package StringUtils

### StreamNative Ursa storage

8188c7d0 use new test image

Fix delta not support timestamp\_ntz

Use strict match rule to fetch token from UnityCatalogSasTokenProviderTest

Disable flaky AsyncCleanerTest#simpleTest

Fix parquet reader handle union type bug

fix: support nested field partitioning in Iceberg tables

Remove METADATA\_DELETE\_AFTER\_COMMIT\_ENABLED property

Skip remove preserved properties

Fix catalog close bug

Add more log for parse entry

Refactor the offload format by adding a util to convert the entry to KafkaMessage

c5750648 use 4.0.6.8

Upgrade Oxia client to 0.7.0

Bump org.apache.commons:commons-lang3 from 3.17.0 to 3.18.0

Add more admin commands

Removed dependency on commons-lang 2.6

Fix catalog instance not shared

Move the iceberg table creation params to a common place

Fix the integration test mount jar issue.

## Security Fixes


# V4.0.6.9
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.6.9



# StreamNative Weekly Release Notes v4.0.6.9

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.6.9](https://github.com/streamnative/pulsar/releases/tag/v4.0.6.9)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.6.9/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.6.9/images/sha256-c44301d0d2b01552cc6fc5976531234c83844775e35370f723315f117acb2fea)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.6.9/images/sha256-916953a45e7069ffb9c272f55ecdebcfa1528098e0bf45bfbe8187756cb40bb2)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.6.9/images/sha256-916953a45e7069ffb9c272f55ecdebcfa1528098e0bf45bfbe8187756cb40bb2)

## General Changes

### Apache Pulsar

([#24863](https://github.com/apache/pulsar/pull/24863)) \[fix]Fixed getChildren('/') on Oxia based provider

([#24852](https://github.com/apache/pulsar/pull/24852)) \[fix]\[ml] Fix `getNumberOfEntries` may point to deleted ledger

### KoP

c991e802f Merge branch 'branch-4.0' into branch-4.0.6.9

Fix flaky-test: SimpleLoadBalanceTest.testBrokerRestart

a58312ce1 Remove commons-lang usage

Handle invalid topic format error for metadata request

feat(kafka-admin): Support describing topic configs with `kop.kafka.` prefix

Use the Pulsar format for the Kafka consumer offsets topic

fix(kafka-impl): correct topic partition extraction from Pulsar topic names

Reduce spamming logs from schema registry and topic lookup

Add producer ID expiration mechanism for classic engine

Fix released produce request buffer could be accessed

Write transaction log and offset log with partition log

Fix direct memory oom with ack=0

\[Ursa] Improve producer state snapshot taking to avoid metadata thread

### Function Mesh Worker Service

Use new way to build sn-operator image

### StreamNative Ursa storage

Respect markDeletedOffsets when reading entries from PersistStorageApi

Fix pulsar expired ledger not deleted bug

## Security Fixes


# V4.0.7.1
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.7.1



# StreamNative Weekly Release Notes v4.0.7.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.7.1](https://github.com/streamnative/pulsar/releases/tag/v4.0.7.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.7.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.7.1/images/sha256-3765a3f6f47e686515b2c992615732d1626b4735bf48065e4aa67e1ff7895607)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.7.1/images/sha256-0f4e6bbb979e04381286ba075375061fed75896928513bb5d77b2ac10e07aea7)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.7.1/images/sha256-0f4e6bbb979e04381286ba075375061fed75896928513bb5d77b2ac10e07aea7)

## General Changes

### Apache Pulsar

([#24948](https://github.com/apache/pulsar/pull/24948)) \[improve]\[test] Disable flaky PatternConsumerBackPressureTest until the problem is fixed

([#24947](https://github.com/apache/pulsar/pull/24947)) \[fix]\[broker]\[branch-4.0] Fix failed testFinishTakeSnapshotWhenTopicLoading due to topic future cache conflicts

([#23551](https://github.com/apache/pulsar/pull/23551)) \[fix]\[txn] fix concurrent error cause txn stuck in TransactionBufferHandlerImpl#endTxn

([#24939](https://github.com/apache/pulsar/pull/24939)) \[fix]\[broker] Avoid recursive update in ConcurrentHashMap during policy cache cleanup

([#21981](https://github.com/apache/pulsar/pull/21981)) \[fix]\[monitor] Fix the incorrect metrics name

([#24787](https://github.com/apache/pulsar/pull/24787)) \[improve]\[broker] Add tests for using absolute FQDN for advertisedAddress and remove extra dot from brokerId

([#24762](https://github.com/apache/pulsar/pull/24762)) \[fix]\[admin] Set local policies overwrites "number of bundles" passed during namespace creation

([#24941](https://github.com/apache/pulsar/pull/24941)) \[fix]\[broker] Fix bug in PersistentMessageExpiryMonitor which blocked further expirations

([#24929](https://github.com/apache/pulsar/pull/24929)) \[fix]\[test] Stabilize testMsgDropStat by reliably triggering non-persistent publisher drop

([#24934](https://github.com/apache/pulsar/pull/24934)) \[fix]\[broker] Fix stack overflow caused by race condition when closing a connection

([#24932](https://github.com/apache/pulsar/pull/24932)) \[fix]\[broker] ExtensibleLoadManager: handle SessionReestablished and Reconnected events to re-register broker metadata

([#24933](https://github.com/apache/pulsar/pull/24933)) \[fix]\[broker] Use `poll` instead `remove` to avoid `NoSuchElementException`

([#24898](https://github.com/apache/pulsar/pull/24898)) \[fix]\[broker] fix getMaxReadPosition in TransactionBufferDisable should return latest

([#24943](https://github.com/apache/pulsar/pull/24943)) \[improve]\[broker] Don't log an error when updatePartitionedTopic is called on a non-partitioned topic

([#24942](https://github.com/apache/pulsar/pull/24942)) \[improve]\[broker] Optimize lookup result warn log

([#24794](https://github.com/apache/pulsar/pull/24794)) \[fix]\[client] Fix thread leak in reloadLookUp method which is used by ServiceUrlProvider

([#24859](https://github.com/apache/pulsar/pull/24859)) \[fix]\[broker] Run ResourceGroup tasks only when tenants/namespaces registered

([#24915](https://github.com/apache/pulsar/pull/24915)) \[fix]\[broker] BacklogMessageAge is not reset when cursor mdPosition is on an open ledger

([#24860](https://github.com/apache/pulsar/pull/24860)) \[fix]\[broker] Fix wrong behaviour when using namespace.allowed\_clusters, such as namespace deletion and namespace policies updating

([#24917](https://github.com/apache/pulsar/pull/24917)) \[improve]\[ci] Move replication tests to new group Broker Group 5 in Pulsar CI

### KoP

Fix incorrect version ID displayed in schema retrieval log

Add log to trace existing schema retrieval

### pulsarctl

upgrade golang to 1.24.9

## Security Fixes

### Apache Pulsar

([#24937](https://github.com/apache/pulsar/pull/24937)) \[fix]\[sec] Override nimbus-jose-jwt to remediate CVE-2023-52428 and CVE-2025-53864

([#24936](https://github.com/apache/pulsar/pull/24936)) \[fix]\[sec] Override commons-beanutils and commons-configuration2 to remediate CVEs

([#24935](https://github.com/apache/pulsar/pull/24935)) \[fix]\[sec] Override kafka-clients in kinesis-kpl-shaded to remediate CVE-2024-31141 and CVE-2025-27817

([#24923](https://github.com/apache/pulsar/pull/24923)) \[fix]\[sec] Upgrade BouncyCastle FIPS to 2.0.10 to remediate CVE-2025-8916

([#24903](https://github.com/apache/pulsar/pull/24903)) \[fix]\[sec] Upgrade Spring to 6.2.12 to remediate CVE-2025-22233 and CVE-2025-41249


# V4.0.7.2
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.7.2



# StreamNative Weekly Release Notes v4.0.7.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.7.2](https://github.com/streamnative/pulsar/releases/tag/v4.0.7.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.7.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.7.2/images/sha256-47a4b3ea2619d895fce176224eba8c8528620856e9a99e1a72443462ddb34c63)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.7.2/images/sha256-300354c0a0b0fcb66bdd872b7df152f5c43818d9f04066b7a998b79e47f99191)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.7.2/images/sha256-300354c0a0b0fcb66bdd872b7df152f5c43818d9f04066b7a998b79e47f99191)

## General Changes

### Apache Pulsar

([#24983](https://github.com/apache/pulsar/pull/24983)) \[improve] Upgrade Apache Commons library versions

([#24995](https://github.com/apache/pulsar/pull/24995)) \[improve]\[test] Use Oxia project docker container for integration tests

([#24986](https://github.com/apache/pulsar/pull/24986)) \[fix] Handle TLS close\_notify to avoid SslClosedEngineException: SSLEngine closed already

([#24982](https://github.com/apache/pulsar/pull/24982)) \[improve]\[build] Upgrade Testcontainers to 1.21.3

([#24975](https://github.com/apache/pulsar/pull/24975)) \[improve]\[broker]Improve error response of failed to delete topic if it has replicators connected

([#24938](https://github.com/apache/pulsar/pull/24938)) \[fix]\[broker]Wrong backlog: expected 0 but got 1

([#24985](https://github.com/apache/pulsar/pull/24985)) \[improve] Upgrade Log4j2 to 2.25.2 and slf4j to 2.0.17

([#24871](https://github.com/apache/pulsar/pull/24871)) \[fix]\[test] Fixed Non-Guaranteed Order in PoliciesDataTest.propertyAdmin

([#24981](https://github.com/apache/pulsar/pull/24981)) \[fix]\[build] Remove Confluent and Restlet maven repositories from top level pom.xml

([#24976](https://github.com/apache/pulsar/pull/24976)) \[feat]\[meta] upgrade oxia version to 0.7.2

([#24446](https://github.com/apache/pulsar/pull/24446)) \[fix]\[cli] Print result of GetMessageIdByIndex command

([#24222](https://github.com/apache/pulsar/pull/24222)) \[feat]\[admin] PIP-415: Support getting message ID by index

([#24971](https://github.com/apache/pulsar/pull/24971)) \[fix]\[broker]Leaving orphan schemas and topic-level policies after partitioned topic is deleted by GC

([#24805](https://github.com/apache/pulsar/pull/24805)) \[fix]\[test] Made ProtobufNativeSchemaTest.testSchema order-independent

([#24962](https://github.com/apache/pulsar/pull/24962)) \[improve]\[client] Deduplicate getTopicsUnderNamespace in BinaryProtoLookupService

([#24972](https://github.com/apache/pulsar/pull/24972)) \[fix]\[test] Add Delta Tolerance in Double-Precision Assertions to Fix Rounding Flakiness

([#24872](https://github.com/apache/pulsar/pull/24872)) \[fix]\[test] Fixed ResponseBody Check in Test Helper

([#24969](https://github.com/apache/pulsar/pull/24969)) \[fix]\[test] Fixed Nondeterministic Ordering in SchemaInfoTest

([#24965](https://github.com/apache/pulsar/pull/24965)) \[fix]\[client] Fix deduplication for getPartitionedTopicMetadata to include method parameters

([#24945](https://github.com/apache/pulsar/pull/24945)) \[fix]\[broker]Transactional messages can never be sent successfully if concurrently taking transaction buffer snapshot

([#24955](https://github.com/apache/pulsar/pull/24955)) \[fix]\[test] Fix flaky KeySharedSubscriptionBrokerCacheTest.testReplayQueueReadsGettingCached

([#24957](https://github.com/apache/pulsar/pull/24957)) \[fix]\[test] Fix invalid test NonPersistentTopicTest.testProducerRateLimit

([#24952](https://github.com/apache/pulsar/pull/24952)) \[improve]\[fn] Use PulsarByteBufAllocator.DEFAULT instead of ByteBufAllocator.DEFAULT

([#24958](https://github.com/apache/pulsar/pull/24958)) \[cleanup]\[broker] Remove unused configuration maxMessageSizeCheckIntervalInSeconds

([#24954](https://github.com/apache/pulsar/pull/24954)) \[fix]\[broker] AvgShedder comparison error

([#24951](https://github.com/apache/pulsar/pull/24951)) \[fix]\[test] Fix flaky NonPersistentTopicTest.testProducerRateLimit

([#24802](https://github.com/apache/pulsar/pull/24802)) \[fix]\[broker] Trigger topic creation event only once for non-existent topic

### KoP

Bump branch-4.0 to 4.0.7.2

Prevent out-of-order messages caused by asynchronous authorization

Fix snapshot might not be taken when using system topic for producer state

Fix flaky testCommitOffsetsForMultiPartitions

Use `KopTopicTransactionBufferProvider` by default when transaction coordinator is enabled

Fix producer state snapshot

Add kop transaction buffer provider to disable transaction buffer recover for Kafka system topics

feat(kafka): Replace random UUID with Kafka Uuid for topic identification

Align Kafka version to 3.9.1 and replace removed kafka.admin.ConsumerGroupCommand in tests with Kafka Admin client

\[Ursa] Allow client to retry send when the producer state recovery fails

Fix Ursa storage in NamespaceBundleOwnershipListener unload behavior

Add producer ID expiration mechanism for Ursa engine

Remove "Found owner" logs with low lookup latency (\< 10ms)

Schedule unload group metadata if not owner broker

Don't return non-retriable error in metadata response when lookup fails

Fix OffsetCommit request might not take effect unless reloading from metadata store

Support client side retry for temporary metadata or write failures

Support returning partition count in create topics response

Bump spotbugs version to 4.9.8

### pulsarctl

Fix CVE CVE-2025-63811

### Function Mesh Worker Service

Support unified-rbac for all components

add kafka managed auth data annotation

bump funciton-mesh v0.26.0

bump function-mesh to 0.26.0

feat: make function api pass sink and source config

Add retry for ci

### StreamNative Tiered storage

Move the unsupported handler to info level

## Security Fixes

### Apache Pulsar

([#24987](https://github.com/apache/pulsar/pull/24987)) \[fix]\[sec] Bump github.com/dvsekhvalnov/jose2go from 1.6.0 to 1.7.0 in /pulsar-function-go

([#24953](https://github.com/apache/pulsar/pull/24953)) \[fix]\[sec] Update Hbase version to 2.6.3-hadoop3 and exclude Avro from hbase-client to remediate CVEs

([#24949](https://github.com/apache/pulsar/pull/24949)) \[fix]\[sec] Added Exclusions for tomcat-embed-core and derby and override mina-core to remediate CVEs

([#24950](https://github.com/apache/pulsar/pull/24950)) \[fix]\[sec] Upgrade hadoop3 version from 3.4.0 to 3.4.1


# V4.0.8.1
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.8.1



# StreamNative Weekly Release Notes v4.0.8.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.8.1](https://github.com/streamnative/pulsar/releases/tag/v4.0.8.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.8.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.8.1/images/sha256-168257c23d3dcd275a39eaab2e76f6d125d8b97186a170390a309a15950f6e98)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.8.1/images/sha256-362b0ff51114804d346791529589980146afaaeaec4f3062e76ed86a0ab8be91)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.8.1/images/sha256-362b0ff51114804d346791529589980146afaaeaec4f3062e76ed86a0ab8be91)

## General Changes

### Apache Pulsar

([#24994)](https://github.com/apache/pulsar/pull/24994))) Revert "\[improve]\[monitor] Upgrade OpenTelemetry to 1.56.0, Otel instrumentation to 2.21.0 and Otel semconv to 1.37.0

([#24833](https://github.com/apache/pulsar/pull/24833)) \[feat] PIP-442: Add memory limits for CommandGetTopicsOfNamespace

([#25016](https://github.com/apache/pulsar/pull/25016)) \[fix]\[broker]Fix memory leak when using a customized ManagedLedger implementation

([#25022](https://github.com/apache/pulsar/pull/25022)) \[fix] Upgrade gson to 2.13.2

([#25014](https://github.com/apache/pulsar/pull/25014)) \[fix]\[client] Fix thread-safety of AutoProduceBytesSchema

([#25015](https://github.com/apache/pulsar/pull/25015)) \[fix]\[client] Fix AutoProduceBytesSchema.clone() method

([#25018](https://github.com/apache/pulsar/pull/25018)) \[improve]\[broker]Remove the warn log that frequently prints

([#25011](https://github.com/apache/pulsar/pull/25011)) \[improve] Eliminate unnecessary duplicate schema lookups for partitioned topics in client and geo-replication

([#25004](https://github.com/apache/pulsar/pull/25004)) \[fix]\[broker] Add schema version in rest produce api

([#25013](https://github.com/apache/pulsar/pull/25013)) \[improve]\[client] Test no exception could be thrown for invalid epoch in message

([#25012](https://github.com/apache/pulsar/pull/25012)) \[fix]\[broker] Fix issue with schemaValidationEnforced in geo-replication

([#25008](https://github.com/apache/pulsar/pull/25008)) \[fix]\[client] Fix double recycling of the message in isValidConsumerEpoch method

([#25007](https://github.com/apache/pulsar/pull/25007)) \[fix]\[client] PIP-84: Skip processing a message in the message listener if the consumer epoch is no longer valid

([#25006](https://github.com/apache/pulsar/pull/25006)) \[fix]\[client] Skip processing messages in the listener when the consumer has been closed

([#24994](https://github.com/apache/pulsar/pull/24994)) \[improve]\[monitor] Upgrade OpenTelemetry to 1.56.0, Otel instrumentation to 2.21.0 and Otel semconv to 1.37.0

([#24471](https://github.com/apache/pulsar/pull/24471)) \[improve]\[broker]\[pip-431] PIP-431: Add Creation and Last Publish Timestamps to Topic Stats

([#24997](https://github.com/apache/pulsar/pull/24997)) \[fix]\[broker] Fix creation of replicated subscriptions for partitioned topics

### KoP

([#1644)](https://github.com/streamnative/ksn/pull/1644))) Revert "Fix breaking changes of latest 4.2.0-SNAPSHOT

a742e9654 Bump version to 4.0.8.1

Fix breaking changes of latest 4.2.0-SNAPSHOT

### StreamNative Pulsar Plugins

Update jose2go for CVE-2025-63811

### Cloud Pulsar Plugins

support

Add FileBasedJwksResolver

### Function Mesh Worker Service

Fix ci

### StreamNative Unified RBAC

fixes: the invalid token exception not have been caught

## Security Fixes

### Apache Pulsar

([#25024](https://github.com/apache/pulsar/pull/25024)) \[fix]\[sec] Eliminate commons-collections dependency


# V4.0.8.2
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.8.2



# StreamNative Weekly Release Notes v4.0.8.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.8.2](https://github.com/streamnative/pulsar/releases/tag/v4.0.8.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.8.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.8.2/images/sha256-6c459766683cd939b12555c8f7374cef71a4721c4e940ffa6a646b05b85c7a68)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.8.2/images/sha256-14c3482de12e922e09fb59a25db023f7537801ec336941c58ec36e9057c97225)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.8.2/images/sha256-14c3482de12e922e09fb59a25db023f7537801ec336941c58ec36e9057c97225)

## General Changes

### Apache Pulsar

([#25034](https://github.com/apache/pulsar/pull/25034)) \[improve]\[misc]introduce log4j Console appender ConsoleJson

([#25032](https://github.com/apache/pulsar/pull/25032)) \[fix]\[test] Replace LZ4FastDecompressor with LZ4SafeDecompressor in test

([#25027](https://github.com/apache/pulsar/pull/25027)) \[improve]\[misc] Add log4j-layout-template-json to server distribution to enable e.g. ECS template support in log4j configurations for Pulsar server components.

([#25026](https://github.com/apache/pulsar/pull/25026)) \[improve]\[broker]Add test for getting partitioned topic metadata with PulsarAdmin client

([#25036](https://github.com/apache/pulsar/pull/25036)) \[improve]\[client] Add null checks for MessageAcknowledger methods to prevent NullPointerException

([#25039](https://github.com/apache/pulsar/pull/25039)) \[fix]\[broker] Fix potential NPE in InMemTransactionBuffer.appendBufferToTxn by returning a valid Position

([#16651](https://github.com/apache/pulsar/pull/16651)) \[improve]\[broker] Fix replicated subscriptions race condition with mark delete update and snapshot completion

([#24579](https://github.com/apache/pulsar/pull/24579)) \[fix]\[broker]Avoid read a entry that entry id is -1 when calling getLastMessagePublishTime

([#25037](https://github.com/apache/pulsar/pull/25037)) \[fix]\[broker]Incorrect backlog that is larger than expected

([#24825](https://github.com/apache/pulsar/pull/24825)) \[improve]\[broker] Cache last publish timestamp for idle topics to reduce storage reads

### KoP

Refactor schema provider to use Confluent avro schema provider

([#1648)](https://github.com/streamnative/ksn/pull/1648))) Revert "Upgrade Confluent Schema Registry version to 7.9.4

Handle no zone case for ursa storage

Fix schema-registry docker image build workflow

Build docker image for schema-registry

\[Ursa] Recover producer state according to the client id's zone

Upgrade Confluent Schema Registry version to 7.9.4

\[Ursa] Fix producer state recovery will never complete when messages are written concurrently

Return UNKNOWN\_SERVER\_ERROR when topic fails to delete due to metadata store error

Fix flaky test `KafkaRbacCompatibilityAuthorizationTest`

Use metadata store to store KSN producer state snapshot by default

Fix ProducerStateManager last mapped offset

Improve logging when OutOfOrderSequenceException happens

Bump branch-4.0 to 4.0.8.2

Add detailed logging for Schema Registry error responses

### StreamNative Pulsar Plugins

Fix setup-go action version

### pulsarctl

Fix setup-go action issue

Fix setup-go action version and upgrade go version to fix CVE

### Cloud Pulsar Plugins

Use a specific Pulsar version for branch-4.0

### Function Mesh Worker Service

Add ComponentLimits to custom config

support plain auth for KafkaConnect and individual functions worker deployment

## Security Fixes


# V4.0.8.3
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.8.3



# StreamNative Weekly Release Notes v4.0.8.3

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.8.3](https://github.com/streamnative/pulsar/releases/tag/v4.0.8.3)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.8.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.8.3/images/sha256-3f0f1a348202fe09b521b41bf64101861ede6bdd2d3ac328e471a716d9b7f027)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.8.3/images/sha256-38726712bc6dab7fd58f96645c361998179a0db6fef0e1ee35bb445995de32b6)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.8.3/images/sha256-38726712bc6dab7fd58f96645c361998179a0db6fef0e1ee35bb445995de32b6)

## General Changes

### Apache Pulsar

([#25085](https://github.com/apache/pulsar/pull/25085)) \[improve]\[io] Replace Qpid in tests with RabbitMQ in Testcontainers and upgrade RabbitMQ client version

([#25084](https://github.com/apache/pulsar/pull/25084)) \[fix]\[build] Activate jdk21 and jdk24 profiles on Java 25

([#25073](https://github.com/apache/pulsar/pull/25073)) \[fix]\[broker]Infinitely failed to delete topic if the first time failed and enabled transaction

([#25047](https://github.com/apache/pulsar/pull/25047)) \[fix]\[broker]Fix incorrect backlog if use multiple acknowledge types on the same subscription

([#25066](https://github.com/apache/pulsar/pull/25066)) \[fix]\[broker] PIP-442: Fix race condition in async semaphore permit updates that causes memory limits to become ineffective

([#24980](https://github.com/apache/pulsar/pull/24980)) \[fix]\[broker] fix prepareInitPoliciesCacheAsync in SystemTopicBasedTopicPoliciesService

([#24658](https://github.com/apache/pulsar/pull/24658)) \[improve]\[broker] Optimize Reader creation in TopicPoliciesService

([#25053](https://github.com/apache/pulsar/pull/25053)) \[improve]\[broker] Use atomic counter for ongoing transaction count

([#25069](https://github.com/apache/pulsar/pull/25069)) \[fix]\[client] Fix invalid parameter type passed to Map.get in TopicsImpl.getListAsync method

([#25044](https://github.com/apache/pulsar/pull/25044)) \[improve]\[broker] Improve replicated subscription snapshot cache so that subscriptions can be replicated when mark delete position update is not frequent

([#25067](https://github.com/apache/pulsar/pull/25067)) \[fix]\[broker] Force EnsemblePolicies to resolve network location after rackInfoMap is updated due to changes in /ledgers/available znode

([#25050](https://github.com/apache/pulsar/pull/25050)) \[fix]\[admin] Refactor bookie affinity group sync operations to async in rest api

([#25059](https://github.com/apache/pulsar/pull/25059)) \[fix]\[broker] Fix various error-prone detected errors mainly in logging and String.format parameters

([#25054](https://github.com/apache/pulsar/pull/25054)) \[improve]\[build] Upgrade errorprone to 2.45.0 version

([#25056](https://github.com/apache/pulsar/pull/25056)) \[fix]\[cli] Fix output of --print-metadata in cli consume

([#25051](https://github.com/apache/pulsar/pull/25051)) \[fix]\[cli] Fix some pulsar-admin topicPolicies commands exiting before async operations complete

### KoP

Improve logging for read entry errors

fix(schemaregistry): add schema type check before compatibility checking

fix(schema-registry): validate JSON schema format during registration

Remove the immature producer side throttling feature

\[branch-4.0] Upgrade Confluent Schema Registry version to 7.9.4

### Function Mesh Worker Service

Update authorization error msg

build(deps): bump function-mesh.version to v0.26.1

## Security Fixes

### Apache Pulsar

([#25078](https://github.com/apache/pulsar/pull/25078)) \[fix]\[sec] Upgrade Netty to 4.1.130.Final

([#25045](https://github.com/apache/pulsar/pull/25045)) \[fix]\[sec] Bump at.yawk.lz4:lz4-java from 1.9.0 to 1.10.1 in /pulsar-common


# V4.0.8.4
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.8.4



# StreamNative Weekly Release Notes v4.0.8.4

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.8.4](https://github.com/streamnative/pulsar/releases/tag/v4.0.8.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.8.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.8.4/images/sha256-838f1ea69f315f06ca15cf566163312c30289910861632699207456d6bf4cbd1)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.8.4/images/sha256-a7cb07e385a9019eea4014bcc6cf7ae55c0a03de8e10a2a3f033fde579039ba0)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.8.4/images/sha256-a7cb07e385a9019eea4014bcc6cf7ae55c0a03de8e10a2a3f033fde579039ba0)

## General Changes

### Apache Pulsar

([#25105](https://github.com/apache/pulsar/pull/25105)) \[fix]\[broker]pulsar\_ml\_reads\_inflight\_bytes and pulsar\_ml\_reads\_available\_inflight\_bytes are 0 at the same time

([#25087](https://github.com/apache/pulsar/pull/25087)) \[fix]\[broker] Fix cursor position persistence in ledger trimming

### MoP

Fix mqtt disconnect due to appId permission bug when namespace policies update

### KoP

([#1691)](https://github.com/streamnative/ksn/pull/1691))) Revert "Return UNKNOWN\_TOPIC\_OR\_PARTITION error for partitioned metadata loss

Add github deploy profile

\[Ursa] Fix the pulsar internal topic owner issue caused create topic stuck

Fix potential NPE issue when initializing schema storage reader

Fix inflight reads limiter permits leak when offsetsForTimes is called

Return UNKNOWN\_TOPIC\_OR\_PARTITION error for partitioned metadata loss

Fix retention and TTL policies on metadata namespace

Fix producer state manager snapshot buffer start issue when use global zk

Correct token extraction for Kafka internal schema registry client

### StreamNative Pulsar Plugins

Upgrade alpine from 3.19.1 to 3.22

### pulsarctl

Fix assertion on TopiCreateTimeStamp

Upgrade pulsar go client to latest and golang to 1.25

### Function Mesh Worker Service

Uncomment connector copy commands in Dockerfile

fix: kafka sink auth inject not working

## Security Fixes

### Apache Pulsar

([#25102](https://github.com/apache/pulsar/pull/25102)) \[fix]\[sec] Upgrade log4j to 2.25.3 to address CVE-2025-68161

([#25095](https://github.com/apache/pulsar/pull/25095)) \[fix]\[sec] Upgrade jose4j to 0.9.6 to address CVE-2024-29371


# V4.0.8.5
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.8.5



# StreamNative Weekly Release Notes v4.0.8.5

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.8.5](https://github.com/streamnative/pulsar/releases/tag/v4.0.8.5)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.8.5/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.8.5/images/sha256-ce869094492513669b65b0d6ec436b0d380f19577c714028e0737d5cf00a1d1b)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.8.5/images/sha256-70f21db9e8b20f97b21f38bd2a534836055435e4d4b6a4ab1d31331a5884e44a)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.8.5/images/sha256-70f21db9e8b20f97b21f38bd2a534836055435e4d4b6a4ab1d31331a5884e44a)

## General Changes

### Apache Pulsar

([#25101](https://github.com/apache/pulsar/pull/25101)) \[fix]\[test] Fix ManagedCursorTest and NonDurableCursorTest flaky tests

([#25106](https://github.com/apache/pulsar/pull/25106)) \[fix]\[client]Producer stuck or geo-replication stuck due to wrong value of message.numMessagesInBatch

### StreamNative Pulsar Plugins

52d8c24c2 ignore flaky test

8e2d4607f fix checkstyle issue

5c3a4edb0 Fix checkstyle issue and opentel sdk spi version

9fe1181e4 Revert "add opentelemetry version"

Support protobuf-native type for rest-v2

ff46e00d1 add opentelemetry version

Add new rest-consume api

### Function Mesh Worker Service

Support custom agent framework and hpa for Agent

feat(auth): add API Keys authentication support

### StreamNative Tiered storage

Fix some CVE

### StreamNative Unified RBAC

perf(authz): optimize JWT parsing and metadata extraction with caching

Adapt new PrivilegesManager API for sdk-go-cloud

Add sdk-go-oxia

## Security Fixes


# V4.0.8.6
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.8.6



# StreamNative Weekly Release Notes v4.0.8.6

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.8.6](https://github.com/streamnative/pulsar/releases/tag/v4.0.8.6)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.8.6/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.8.6/images/sha256-36d0f6fb52cf9d1fd5a48bd20b8f2c3b4c3432c866e7fce368a44fd2f52a0de5)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.8.6/images/sha256-a66ef60b008c287a39395200b93d915726b371c2de6a722277b2e1d9600f83cb)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.8.6/images/sha256-a66ef60b008c287a39395200b93d915726b371c2de6a722277b2e1d9600f83cb)

## General Changes

### Apache Pulsar

([#25136](https://github.com/apache/pulsar/pull/25136)) \[fix]\[broker] Fix regex matching of namespace name which might contain a regex char

([#25110](https://github.com/apache/pulsar/pull/25110)) \[fix]\[broker] Fix markDeletedPosition race condition in ManagedLedgerImpl.maybeUpdateCursorBeforeTrimmingConsumedLedger() method

([#25125](https://github.com/apache/pulsar/pull/25125)) \[fix]\[test] Wait for txn.abort() to complete to avoid AdminApiTransactionTest.testAnalyzeSubscriptionBacklogWithTransactionMarker() flaky test

([#25091](https://github.com/apache/pulsar/pull/25091)) \[improve]\[admin] Add counter for marker messages in PersistentTopics.analyzeSubscriptionBacklog() rest api

([#25114](https://github.com/apache/pulsar/pull/25114)) \[fix]\[broker]Topic deleting failed after removed local cluster from namespace policies

([#25124](https://github.com/apache/pulsar/pull/25124)) \[fix]\[admin] Fix asyncGetRequest to handle 204

([#25119](https://github.com/apache/pulsar/pull/25119)) \[fix]\[broker] Fix compaction horizon might be reset to an old position when phase two is interrupted

([#25104](https://github.com/apache/pulsar/pull/25104)) \[improve]\[broker] Fix thread safety issue in ManagedCursorImpl.removeProperty

([#25121](https://github.com/apache/pulsar/pull/25121)) \[fix]\[broker] Fix MultiRolesTokenAuthorizationProvider error when subscription prefix doesn't match.

([#25089](https://github.com/apache/pulsar/pull/25089)) \[fix]\[ml] Fix cursor backlog size to account for individual acks

([#25077](https://github.com/apache/pulsar/pull/25077)) \[fix]\[broker] Fix chunked message loss when no consumers are available

([#25130](https://github.com/apache/pulsar/pull/25130)) \[improve]\[broker] Change the log level from error to info when throwing NotAllowedException

([#25048](https://github.com/apache/pulsar/pull/25048)) \[improve]\[broker] Enhance logging for adding schema failures in ServerCnx

### KoP

Ignore the exception for duplicated release on an entry

Return failed future instead of null when cursor manager is closed

Avoid closing a Kafka admin with small request timeout to speed up DescribeConsumerGroupTest

ff67c0e9a Delete ClusterLevelSchemaValidationTest since this feature is only available in 4.1

Cache Maven dependencies to speed up CI

Set shadow namespace load manager to void interceptor test fail

Reduce total test time of all workflows

Print maven effect setting before build

test(transaction): increase timeout for transaction recovery test

\[Ursa] Prevent possible partitioned metadata loss that fails ksqlDB's SHOW TOPICS command

Fix schema evolution version resetting after deleting a subject version in oxia schema registry

Add cloud plugin common library when building schema registry docker image

Remove ShadowTopicManager

### StreamNative Pulsar Plugins

aa51b1313 fix(rest): fix producer leak

feat(compaction): Support `compact,delete` cleanup Policy for Kafka topics

## Security Fixes


# V4.0.8.7
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.8.7



# StreamNative Weekly Release Notes v4.0.8.7

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.8.7](https://github.com/streamnative/pulsar/releases/tag/v4.0.8.7)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.8.7/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.8.7/images/sha256-5bbe6ab75517245387989a7e4dc55c9d3c95578bf87f482ef7b81642ab8f5565)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.8.7/images/sha256-079f91ffd04d80874ed0a1a4253011759400600a90c220a4cd6a5dc2d35615ae)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.8.7/images/sha256-079f91ffd04d80874ed0a1a4253011759400600a90c220a4cd6a5dc2d35615ae)

## General Changes

### Apache Pulsar

([#25177](https://github.com/apache/pulsar/pull/25177)) \[fix]\[ml] Fix NoSuchElementException in EntryCountEstimator caused by a race condition

([#25166](https://github.com/apache/pulsar/pull/25166)) \[improve]\[broker] Upgrade bookkeeper to 4.17.3

([#25132](https://github.com/apache/pulsar/pull/25132)) \[improve]\[broker] Ensure metadata session state visibility and improve Unstable observability for ServiceUnitStateChannelImpl

([#25070](https://github.com/apache/pulsar/pull/25070)) \[improve]\[broker] PIP-442: Add memory limits for topic list watcher (part 2)

([#25157](https://github.com/apache/pulsar/pull/25157)) \[fix]\[fn] Fix graceful Pulsar Function shutdown so that consumers and producers are closed

([#25151](https://github.com/apache/pulsar/pull/25151)) \[fix]\[broker] Fence reset cursor by timestamp to avoid concurrent timestamp-based position lookups

([#25148](https://github.com/apache/pulsar/pull/25148)) \[fix]\[ml] Retry offload reads when OffloadReadHandleClosedException is encountered

([#25149](https://github.com/apache/pulsar/pull/25149)) \[fix]\[admin] Fix offload policy incompatible issue.

([#25142](https://github.com/apache/pulsar/pull/25142)) \[fix]\[proxy] Fix memory leaks in ParserProxyHandler

([#25140](https://github.com/apache/pulsar/pull/25140)) \[fix]\[fn] complete flushAsync before closeAsync in ProducerCache and wait for completion in closing the cache

([#25031](https://github.com/apache/pulsar/pull/25031)) \[fix]\[broker] Avoid split non-existent bundle

### pulsarctl

fix: patch Go stdlib CVEs in pulsarctl (update to go 1.25.5)

### Cloud Pulsar Plugins

Add commons-lang dependency to sn-broker-inteceptors to fix the compile isuse

### Function Mesh Worker Service

\[branch-4.0] fix: apikeys auth handler uses incorrect issuer

### StreamNative Unified RBAC

fix the illegal license format

e41d505 Bump version to 1.7.3

b6e5296 Bump version to 1.7.2

cff40fa Bump version to 1.7.1

742bb20 fixes license and spotless

df7d236 Bump version to 1.7.0

fixes: workflow issue

feat: support schedule release for rbac maven sdk

## Security Fixes

### Apache Pulsar

([#25152](https://github.com/apache/pulsar/pull/25152)) \[fix]\[sec] Upgrade vertx to address CVE-2026-1002


# V4.0.8.8
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.8.8



# StreamNative Weekly Release Notes v4.0.8.8

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.8.8](https://github.com/streamnative/pulsar/releases/tag/v4.0.8.8)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.8.8/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.8.8/images/sha256-ea64d21c601a22faf6b6600e6eea8f4e6e9c527f5563afe5377f37cdcefac015)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.8.8/images/sha256-abd34d78ffc7f6c25889023c303efdfdd9cd94759c157e440601df6c3dbc8747)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.8.8/images/sha256-abd34d78ffc7f6c25889023c303efdfdd9cd94759c157e440601df6c3dbc8747)

## General Changes

### Apache Pulsar

([#25231](https://github.com/apache/pulsar/pull/25231)) \[fix]\[broker] Fix transactionMetadataFuture completeExceptionally with null value

([#25229](https://github.com/apache/pulsar/pull/25229)) \[fix]\[client] Send all chunkMessageIds to broker for redelivery

([#25221](https://github.com/apache/pulsar/pull/25221)) \[improve]\[broker] Give the detail error msg when authenticate failed with AuthenticationException

([#25227](https://github.com/apache/pulsar/pull/25227)) \[fix]\[test] Fix Mockito stubbing race in TopicListServiceTest

([#25228](https://github.com/apache/pulsar/pull/25228)) \[fix]\[broker] Fix incomplete futures in topic property update/delete methods

([#25224](https://github.com/apache/pulsar/pull/25224)) \[improve]\[broker] Add idle timeout support for http

([#25052](https://github.com/apache/pulsar/pull/25052)) \[improve]\[client] Make authorization server metadata path configurable in AuthenticationOAuth2

([#24944](https://github.com/apache/pulsar/pull/24944)) \[feat]\[client] oauth2 trustcerts file and timeouts

([#25185](https://github.com/apache/pulsar/pull/25185)) \[improve]\[broker] Add strictAuthMethod to require explicit authentication method

([#25223](https://github.com/apache/pulsar/pull/25223)) \[fix]\[broker] Fix httpProxyTimeout config

([#25195](https://github.com/apache/pulsar/pull/25195)) \[feat]\[io] implement pip-297 for jdbc sinks

([#25188](https://github.com/apache/pulsar/pull/25188)) \[fix]\[broker] Prevent missed topic changes in topic watchers and schedule periodic refresh with patternAutoDiscoveryPeriod interval

([#25207](https://github.com/apache/pulsar/pull/25207)) \[fix]\[client] Fix producer synchronous retry handling in failPendingMessages method

([#25199](https://github.com/apache/pulsar/pull/25199)) \[fix]\[broker]Fix ledgerHandle failed to read by using new BK API

([#25165](https://github.com/apache/pulsar/pull/25165)) \[fix]\[broker] Fix ManagedCursorImpl.asyncDelete() method may lose previous async mark delete properties in race condition

([#25216](https://github.com/apache/pulsar/pull/25216)) \[fix]\[test]Fix flaky ExtensibleLoadManagerImplTest\_testGetMetrics

([#25187](https://github.com/apache/pulsar/pull/25187)) \[improve]\[meta] PIP-453: Improve the metadata store threading model

([#25211](https://github.com/apache/pulsar/pull/25211)) \[improve]\[proxy] Add regression tests for package upload with 'Expect: 100-continue'

([#24994](https://github.com/apache/pulsar/pull/24994)) \[improve]\[monitor] Upgrade OpenTelemetry to 1.56.0, Otel instrumentation to 2.21.0 and Otel semconv to 1.37.0

([#25208](https://github.com/apache/pulsar/pull/25208)) \[fix]\[client] Fix race condition between isDuplicate() and flushAsync() method in PersistentAcknowledgmentsGroupingTracker due to incorrect use Netty Recycler

([#25209](https://github.com/apache/pulsar/pull/25209)) \[fix] \[test] Upgrade docker-java to 3.7.0

([#25197](https://github.com/apache/pulsar/pull/25197)) \[fix]\[misc] Allow JWT tokens in OpenID auth without nbf claim

([#25172](https://github.com/apache/pulsar/pull/25172)) \[improve]\[client]Reduce unnecessary getPartitionedTopicMetadata requests when using retry and DLQ topics.

([#25173](https://github.com/apache/pulsar/pull/25173)) \[improve]\[pip] PIP-453: Improve the metadata store threading model

([#25178](https://github.com/apache/pulsar/pull/25178)) \[fix]\[client] ControlledClusterFailover avoid unnecessary reconnection.

([#25179](https://github.com/apache/pulsar/pull/25179)) \[fix]\[proxy] Close client connection immediately when credentials expire and forwardAuthorizationCredentials is disabled

([#25182](https://github.com/apache/pulsar/pull/25182)) \[improve]\[misc] Upgrade snappy version to 1.1.10.8

([#25186](https://github.com/apache/pulsar/pull/25186)) \[fix]\[test] Bump org.assertj:assertj-core from 3.27.5 to 3.27.7

### KoP

Some operations can't work with super-user role

Fix race condition in concurrent Schema Registry requests handling

\[branch-4.0] Upgrade pulsar version to 4.0.8.8

\[branch-4.1] Upgrade unified rbac dependency to 1.7.3

Fix potential concurrent modification issue

Return references when getting schema by subject and version

Fix flaky test IdempotentProducerTest

### StreamNative Pulsar Plugins

898f3b879 fix incompatible with pulsar

Upgrade detector build image to 1.25

07dfa85d0 fix: update pulsar and sn.bom versions to 4.0.8.8 in pom.xml

b1f864ff0 fix: update Maven command to include update flag for dependencies

db522f9ed build detector multi-platform

d9b7c56ec fix: remove opentelemetry-sdk-testing dependency from pom.xml

dd9968bc1 using streamnative-bom opentelemetry version

fix: patch CVE-2025-61726, CVE-2025-61728, CVE-2025-61730 in stdlib

Fix OIDCServlet to use local metadata store instead of configuration metadata store

fix: upgrade zookeeper to 3.9.4 to patch CVE-2025-58457

### pulsarctl

fix: upgrade Go to 1.25.7 to fix CVE-2025-68121

fix: upgrade Go from 1.25.5 to 1.25.6 to patch CVE-2025-61726, CVE-2025-61728, CVE-2025-61730

### Cloud Pulsar Plugins

a2882f8 Revert "Add OpenTelemetry SDK extension dependency to test pom.xml"

93ed760 Add OpenTelemetry SDK extension dependency to test pom.xml

### Function Mesh Worker Service

3b32d782 Fix CI

Reuse authorization service when possible

a448fef3 Enhance CI

## Security Fixes

### Apache Pulsar

([#25095](https://github.com/apache/pulsar/pull/25095)) \[fix]\[sec] Upgrade jose4j to 0.9.6 to address CVE-2024-29371

([#25206](https://github.com/apache/pulsar/pull/25206)) \[fix]\[sec] Upgrade OpenSearch to 2.19.4 to remediate CVE-2025-9624

([#25198](https://github.com/apache/pulsar/pull/25198)) \[fix]\[sec] Exclude org.lz4:lz4-java and standardize on at.yawk.lz4-java to remediate CVE-2025-12183 and CVE-2025-66566

([#25175](https://github.com/apache/pulsar/pull/25175)) \[fix]\[sec] Bump org.apache.solr:solr-core from 9.8.0 to 9.10.1 in /pulsar-io/solr


# V4.0.9.1
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.9.1



# StreamNative Weekly Release Notes v4.0.9.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.9.1](https://github.com/streamnative/pulsar/releases/tag/v4.0.9.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.9.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.9.1/images/sha256-8a842270207e3c2c4fba62a885591cc79a79acacafc73b88eca97aca320b8946)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.9.1/images/sha256-77da6601ccb9e71832efcd8c8309cdf8f6ef1739c8d2408599926acfc94f9150)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.9.1/images/sha256-77da6601ccb9e71832efcd8c8309cdf8f6ef1739c8d2408599926acfc94f9150)

## General Changes

### Apache Pulsar

([#25187)](https://github.com/apache/pulsar/pull/25187))) Revert "\[improve]\[meta] PIP-453: Improve the metadata store threading model

### KoP

Upgrade testcontainers and docker-java to address min api version issue

Fix cursor leak from KafkaTopicConsumerManager

### StreamNative Pulsar Plugins

058132f0f fix: upgrade testcontainers to 1.21.4 and docker-java to 3.7.0 for Docker 29 compatibility

be06489c9 fix: move Docker setup to beginning of workflow

ae0861545 test: add minimal docker testcontainers workflow

ccd82e363 fix: correct Docker version format to v28.0.4

af189b308 fix: downgrade Docker to 28.0.4 for Testcontainers compatibility

8bab63881 fix: use correct environment variable DOCKER\_API\_VERSION for Docker 29 compatibility

1c4b8b8ff fix: set TESTCONTAINERS\_DOCKER\_API\_VERSION=1.44 for Docker 29 compatibility

526772135 fix: upgrade testcontainers to 1.20.6 for Docker 29 compatibility

698895d2a fix: add Docker setup step for Testcontainers in GitHub Actions

79615cdf9 fix incompatible with pulsar

### Function Mesh Worker Service

148e4784 Fix image non-exist error

666d69ca Fix CI

Use FunctionWorker crd to deploy registry service in CI

Do not allow to update connection and packageConnection

Add integration tests and OpenAPI docs for registry service

Implement registry endpoint

### StreamNative Tiered storage

43252bca Fix Docker environment detection issue by upgrading testcontainers and docker-java

## Security Fixes


# V4.0.9.2
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.9.2



# StreamNative Weekly Release Notes v4.0.9.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.9.2](https://github.com/streamnative/pulsar/releases/tag/v4.0.9.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.9.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.9.2/images/sha256-5dd2ae09ed83c48be9c05d80f9e8c8d7459f1a5aa679b0e56a4be74b125a8272)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.9.2/images/sha256-eaa32d9667890a8f5494735da0b19cdf9b57bd446aa1341d48312e26eceaaf88)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.9.2/images/sha256-eaa32d9667890a8f5494735da0b19cdf9b57bd446aa1341d48312e26eceaaf88)

## General Changes

### Apache Pulsar

([#25262](https://github.com/apache/pulsar/pull/25262)) \[fix]\[broker] Guard AsyncTokenBucket against long overflow

([#25200](https://github.com/apache/pulsar/pull/25200)) \[improve]\[broker] Change log level from warn to debug when cursor mark-deleted position ledger doesn't exist

([#25254](https://github.com/apache/pulsar/pull/25254)) \[fix]\[client] Reduce logging in OAuth auth to fix parsing of Pulsar cli command output

([#25253](https://github.com/apache/pulsar/pull/25253)) \[improve] Upgrade RoaringBitmap to 1.6.9 version

([#25251](https://github.com/apache/pulsar/pull/25251)) \[improve]\[fn] Upgrade Pulsar Python client version to 3.10.0

([#25246](https://github.com/apache/pulsar/pull/25246)) \[fix]\[meta] Metadata cache refresh might not take effect

([#25247](https://github.com/apache/pulsar/pull/25247)) \[fix]\[test] Fix ResourceQuotaCalculatorImplTest#testNeedToReportLocalUsage

([#25241](https://github.com/apache/pulsar/pull/25241)) \[fix]\[test] fix testBatchMetadataStoreMetrics.

([#25232](https://github.com/apache/pulsar/pull/25232)) \[improve] Upgrade Netty to 4.1.131.Final

([#25187)](https://github.com/apache/pulsar/pull/25187))) Reapply "\[improve]\[meta] PIP-453: Improve the metadata store threading model

### KoP

\[Ursa] Avoid replaying the whole topic for producer with no zone id after upgrade

Remove debug log for broker selection in UrsaLoadBalancer

Add consumer group and its consumers to Pulsar topic stats

### StreamNative Pulsar Plugins

8455da80b Fix Docker environment detection issue by upgrading testcontainers and docker-java

### pulsarctl

Add namespace-level inactive topic policies commands

### Function Mesh Worker Service

7db3471e Fix license missing error in retry

Use one Dockerfile for all CI

Use custom PulsarResources for Registry service

## Security Fixes

### Apache Pulsar

([#25256](https://github.com/apache/pulsar/pull/25256)) \[fix]\[sec] Upgrade aircompressor to 2.0.3 to resolve CVE-2025-67721

([#25250](https://github.com/apache/pulsar/pull/25250)) \[fix]\[sec] Upgrade Python protobuf version to 6.33.5 to address CVE-2026-0994


# V4.0.9.3
Source: https://docs.streamnative.io/release-notes/pulsar/v4.0/v4.0.9.3



# StreamNative Weekly Release Notes v4.0.9.3

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v4.0.9.3](https://github.com/streamnative/pulsar/releases/tag/v4.0.9.3)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/4.0.9.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/4.0.9.3/images/sha256-d008019b693fb450470acf2cc1280d8dd50ef3d52a73d91233f3d64a4e325fcf)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/4.0.9.3/images/sha256-9f65f864d7d905c19c79bba7ca044000b63bc523613d50a9b13bf78ab4fbab39)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/4.0.9.3/images/sha256-9f65f864d7d905c19c79bba7ca044000b63bc523613d50a9b13bf78ab4fbab39)

## General Changes

### Apache Pulsar

([#25269](https://github.com/apache/pulsar/pull/25269)) \[improve]\[broker] Optimize AsyncTokenBucket overflow solution further to reduce fallback to BigInteger

### KoP

Fix the jackson-dataformat-yaml dependency not found

Fix mvn deploy failure for oauth-client module

### StreamNative Pulsar Plugins

81a73ea81 fix

7878af36e Fix oidc test

53c5214f6 Fix oidc test

Fix enum type cause Avro deserialize error

### Function Mesh Worker Service

Update dynamic auth

## Security Fixes

### Apache Pulsar

([#25264](https://github.com/apache/pulsar/pull/25264)) \[fix]\[sec] Upgrade Jackson version to 2.18.6


# QuickStart - Kafka Client
Source: https://docs.streamnative.io/cloud/get-started/quickstart-kafka



<Note title="Note">
  * This quick start assumes that you already have a StreamNative Cloud account with a valid form of payment.
  * If you cannot enable the Kafka protocol on an existing cluster, [submit a ticket](https://support.streamnative.io/hc/en-us/requests/new) with this request to the support team.
</Note>

This QuickStart guides you through how to do the following:

* Set up a StreamNative cluster in a new organization and a new instance and enable the Kafka protocol on the Pulsar cluster.
* Configure a Kafka client to produce and consume messages.

For a general quick start on creating a cluster and using a Pulsar Java client to produce and consume messages to the Pulsar cluster, see [QuickStart - StreamNative Cloud](/cloud/get-started/quickstart-console).

## Step 1: Log in to StreamNative Cloud Console

To log in to the Streamnative Cloud Console, navigate to the [StreamNative Cloud Console login page](https://console.streamnative.cloud/?defaultMethod=signup), enter the email address and password, and then click **Log in** to log in to the StreamNative Cloud Console.

## Step 2: Create an organization

1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Organizations**.
2. On the **Organization** page, click **Create organization**.
3. Enter a name for the organization and then click **Create**.

You might have to wait briefly for the organization to be created. After your new organization is created, proceed to creating an instance.

## Step 3: Create an instance and a cluster

1. On the left navigation pane, click **Dashboard**.

2. On the **Instances** card, click **New**.

3. Click **Deploy Dedicated** to start the instance creation process.

4. On the **Instance Configuration** page, enter a name for your instance, select an infrastructure pool and the Availability Zone (AZ), and then click **Cluster Location**.

5. On the **Cluster Location** page, enter a name for your cluster, select the cluster location, and then click **Cluster Size**.

6. On the **Cluster Size** page, configure the cluster, and then click **Payment**.

   * On the **Basic** tab, select custom sizing options.

   * On the **Advanced** tab, in the **Features area**, enable the **Kafka Protocol** option.

7. If needed, on the **Payment** page, in the **Create Payment Method** box, enter a valid credit card number, and then click **Create Payment Method**.

8. Click **Finish**.

The cluster page displays, showing the cluster creation process. The cluster is ready for use after all components have been successfully deployed.

## Step 4: Create a service account

1. On the left navigation pane, click **Service Accounts**.

2. Click **Create Service Account**.

3. Enter a name for the service account, and then click **Confirm**.

After creating a service account, you need to grant the service account produce and consume permissions to a namespace on your Pulsar cluster.

## Step 5: Grant service account permissions

1. On the left navigation pane, in the **Admin** section, click **Tenants/Namespaces**.

2. Select the **Public** tenant, then select the **Default** namespace under the tenant.

3. Select the **POLICY** tab.

4. In the **Authorization** area, click **ADD ROLE** and select the service account you just created in the previous section.

5. In the **Authorization** area, on the drop-down menu below the service name you just added, select the **consume** and **produce** roles. The roles are added to your service account.

Now, you can use the service account to connect to your cluster with the Kafka CLI tool or Kafka client.

## Step 6: Connect to the cluster

For more information about how to connect to a Pulsar cluster using Kafka clients, see [use Kafka clients to connect to your Pulsar cluster](/cloud/build/kafka-clients/kafka-on-cloud#kafka-clients).


# Lakehouse Table Overview
Source: https://docs.streamnative.io/cloud/lakehouse/lakehouse-table-overview



Lakehouse Table is a zero-ETL integration that automatically converts streaming data from Apache Pulsar topics into open table formats -- Apache Iceberg and Delta Lake -- stored directly on object storage (AWS S3, GCS, Azure Blob Storage). This enables unified streaming and analytics access to the same data without building or maintaining separate data pipelines.

## Architecture

```
                          ┌──────────────────┐
                          │   Pulsar Broker  │
                          └────────┬─────────┘
                                   │
                   ┌───────────────┴────────────────┐
                   │  WAL Storage                   │
                   │  ┌──────────────────────────┐  │
                   │  │ Latency-Optimized:       │  │
                   │  │   Apache BookKeeper      │  │
                   │  ├──────────────────────────┤  │
                   │  │ Cost-Optimized:          │  │
                   │  │   Object Storage         │  │
                   │  │   (S3 / GCS / Azure)     │  │
                   │  └──────────────────────────┘  │
                   └───────────────┬────────────────┘
                                   │ Reads from both
                                   ▼
                    ┌─────────────────────────────┐
                    │    Compaction Service       │
                    │  (WAL → Parquet conversion) │
                    └────────────┬────────────────┘
                                 │ Commit
                                 ▼
                    ┌─────────────────────────────┐
                    │    Lakehouse Table          │
                    │  (Iceberg / Delta Lake)     │
                    └─────────────────────────────┘
                                 │
                    ┌────────────┴────────────┐
                    ▼                         ▼
             External Catalog          Query Engines
          (Unity Catalog, S3Table,   (Spark, Trino,
           BigLake, Snowflake)        DuckDB, Athena)
```

### WAL Storage Options

Lakehouse Table supports two WAL storage tiers:

* **Latency-optimized (Apache BookKeeper):** Low-latency writes for performance-sensitive workloads
* **Cost-optimized (Object Storage):** Direct writes to AWS S3, GCS, or Azure Blob Storage for cost efficiency

The **Compaction Service reads from both** BookKeeper and Object Storage, converts the data to Parquet format, and commits snapshots to the lakehouse catalog.

### Coordination

**Oxia** serves as the metadata store for coordination, leader election, schema storage, and offset index management.

The Compaction Service operates with a leader-worker architecture: the leader publishes compaction tasks and commits results to the lakehouse catalog, while workers perform the WAL-to-Parquet conversion.

## Table Modes

### External Table (SDT -- Stream Delivered Table)

An External Table delivers data from Pulsar topics into an external lakehouse catalog (such as Databricks Unity Catalog, Snowflake, AWS S3Table, or Google BigLake). A **separate copy** of the topic data is written to the Lakehouse table — the Pulsar topic and the Lakehouse table hold independent copies of the same records. In this mode:

* Data is written to Iceberg or Delta Lake tables managed by the external catalog as a separate copy from the Pulsar topic
* Analytical access via standard table APIs (Spark, Trino, DuckDB, Athena, etc.)
* Supports **upsert**, **partition key**, and **schema evolution**
* The external catalog governs data lifecycle (retention, deletion) for the Lakehouse copy independently of the Pulsar topic
* Streaming reads with offset semantics are **not** supported on the delivered data

**Use External Tables when:** you want to deliver streaming data into curated lakehouse tables for analytics, integrate with existing data platforms, or need upsert/deduplication capabilities.

### Internal Table (SBT -- Stream Backed Table)

> **Coming Soon** -- Internal Table support is under active development.

An Internal Table is managed entirely by Ursa Storage. The Pulsar topic and the Lakehouse table **share the same single copy of data** — there is no separate write to the Lakehouse table. The same physical data supports both streaming reads (with offset tracking and replay) and analytical queries -- true stream-table duality with zero data duplication.

## Supported Cluster Profiles and Protocols

StreamNative Private Cloud offers two cluster types (Pulsar and Kafka), each with two performance profiles (latency-optimized and cost-optimized). Lakehouse delivery support depends on the combination of cluster type, profile, and producer protocol.

| Cluster Type | Profile           | Producer Protocol | Lakehouse Delivery |
| ------------ | ----------------- | ----------------- | ------------------ |
| Pulsar       | Latency-optimized | Pulsar            | Supported          |
| Pulsar       | Latency-optimized | Kafka             | Coming Soon        |
| Pulsar       | Cost-optimized    | Kafka             | Supported          |
| Pulsar       | Cost-optimized    | Pulsar            | Not yet supported  |
| Kafka        | Cost-optimized    | Kafka             | Supported          |
| Kafka        | Latency-optimized | Kafka             | Coming Soon        |

Notes:

* A **Pulsar latency-optimized** cluster uses Apache BookKeeper as the WAL tier. Topic data produced via the Pulsar protocol can be delivered to Lakehouse today; Kafka-protocol delivery is on the roadmap.
* A **Pulsar cost-optimized** cluster uses object storage as the WAL tier. Topic data produced via the Kafka protocol is delivered to Lakehouse; the Pulsar protocol is not yet supported on this profile.
* A **Kafka cost-optimized** cluster delivers Kafka topic data to Lakehouse today.
* A **Kafka latency-optimized** cluster will support Lakehouse delivery in a future release.

## Supported Formats

| Format         | Status    |
| -------------- | --------- |
| Apache Iceberg | Supported |
| Delta Lake     | Supported |

## Supported Cloud Storage

| Provider             | WAL Storage | Lakehouse Table |
| -------------------- | ----------- | --------------- |
| AWS S3               | Supported   | Supported       |
| Google Cloud Storage | Supported   | Supported       |
| Azure Blob Storage   | Supported   | Supported       |

## Supported Catalogs

Catalog support varies by cloud provider:

| Catalog                                    | Table Format | AWS       | GCP       | Azure     |
| ------------------------------------------ | ------------ | --------- | --------- | --------- |
| Databricks Unity Catalog (Managed Iceberg) | Iceberg      | Supported | Supported | Supported |
| Databricks Unity Catalog (Delta Lake)      | Delta Lake   | Supported | Supported | Supported |
| Snowflake Horizon Catalog                  | Iceberg      | Supported | Supported | Supported |
| Snowflake Open Catalog (Polaris)           | Iceberg      | Supported | Supported | Supported |
| AWS S3Table                                | Iceberg      | Supported | --        | --        |
| Google BigLake                             | Iceberg      | --        | Supported | --        |

## Topic to lakehouse identifier mapping

Each Pulsar topic maps to exactly **one** Lakehouse table. The mapping is 1:1 regardless of how many partitions the topic has — data from all partitions of a partitioned topic is consolidated into a single Lakehouse table.

When data is delivered from a Pulsar topic to a lakehouse table, the topic's tenant, namespace, and topic name are mapped to a catalog **namespace** and a **table name**. The mapping rules differ by catalog type because Pulsar allows characters (`/`, `.`, `-`, `:`) that are not valid in many catalog identifiers.

The compaction service applies the following rules.

### Iceberg with hierarchical catalogs (Snowflake Open Catalog, Snowflake Horizon, Iceberg REST, Iceberg Hadoop)

The original Pulsar identifiers are used unchanged:

* Catalog namespace: `<tenant>.<namespace>` (two-level)
* Table name: the topic local name (the part after the namespace)

For example, the topic `persistent://my-tenant/my-namespace/orders` is mapped to namespace `my-tenant.my-namespace` and table `orders`.

### Iceberg with flat-namespace catalogs (AWS S3Tables, Google BigLake, Hive)

These catalogs only accept a single-level namespace, so the tenant and namespace are flattened into one identifier with a cluster-name prefix. Each component is escaped to remove invalid characters:

| Source character | Replacement               |
| ---------------- | ------------------------- |
| `/`              | `___` (three underscores) |
| `.`              | `_` (one underscore)      |
| `-`              | `__` (two underscores)    |
| `:`              | `____` (four underscores) |

* Catalog namespace: `<cluster>_<formatted-tenant>_<formatted-namespace>` (default cluster prefix is `pulsar`)
* Table name (S3Tables): the topic local name with the same character escapes applied
* Table name (BigLake, Hive): the topic local name as-is

For example, with the default cluster prefix `pulsar`, topic `persistent://public-v1/default.v2/test-table-v1`:

* On **AWS S3Tables**: namespace `pulsar_public__v1_default_v2`, table `test__table__v1`
* On **Google BigLake**: namespace `pulsar_public__v1_default_v2`, table `test-table-v1`

### Databricks Unity Catalog (Iceberg or Delta Lake)

Unity Catalog uses a three-level identifier (`catalog.schema.table`). The compaction service writes all topics into a single schema and encodes the full Pulsar topic path into the **table name**, so each catalog table maps 1:1 to a Pulsar topic. The full topic path `<tenant>/<namespace>/<topic>` is flattened with these escapes:

| Source character | Replacement               |
| ---------------- | ------------------------- |
| `/`              | `__` (two underscores)    |
| `.`              | `____` (four underscores) |
| `-`              | `___` (three underscores) |

For example, topic `persistent://public/default/test-topic` is mapped to table name `public__default__test___topic`. Topic `persistent://public/default/v1.events` is mapped to `public__default__v1____events`.

## Limitations

### Schema limitations

The following topic schema constructs are not supported when delivering data to a Lakehouse Table:

* **Recursive schemas:** Schemas that reference themselves -- directly or indirectly -- are not supported. For example, a record with a field whose type is the same record (such as a tree node that holds a list of child nodes of the same type). Iceberg and Delta Lake require a fixed, finite column structure, which cannot represent a self-referential schema. Topics that use a recursive schema cannot be delivered to a Lakehouse Table.

## Next Steps

* [Deploy Lakehouse Table](/cloud/lakehouse/deploy-lakehouse-table) -- Set up the infrastructure with `private-cloud.yaml`
* [Prepare Lakehouse Catalogs](/cloud/lakehouse/prepare-lakehouse-catalogs) -- Set up your external catalog service
* [Dynamic Configuration Guide](/cloud/lakehouse/dynamic-configuration) -- Cluster-name prefix, override priority, and the full set of dynamic configuration keys
* [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog) -- Connect catalogs to the compaction service
* [Enable Lakehouse Integration](/cloud/lakehouse/enable-lakehouse-integration) -- Enable at cluster, namespace, or topic level
* Features:
  * [Schema Evolution](/cloud/lakehouse/features/schema-evolution)
  * [Variant Type](/cloud/lakehouse/features/variant-type)
  * [Partition Key](/cloud/lakehouse/features/iceberg-partition-key)
  * [Upsert](/cloud/lakehouse/features/iceberg-upsert)
  * [Persist Key](/cloud/lakehouse/features/persist-key)
  * [Persist Extra Metadata](/cloud/lakehouse/features/persist-extra-metadata)
* [Observability](/cloud/lakehouse/lakehouse-observability) -- Metrics, alerts, and Grafana dashboard


# Overview - Lakehouse Tables
Source: https://docs.streamnative.io/cloud/lakehouse/lakehouse-tables-overview



**StreamNative Lakehouse Tables**

StreamNative Lakehouse Tables provide a unified way to expose streaming topics as open table format objects—such as Apache Iceberg and Delta Lake—directly within StreamNative Cloud. With Lakehouse Tables, data produced to Pulsar or Kafka-compatible topics can be automatically stored in object storage in a transactional, analytics-ready table format. This enables seamless integration between real-time data streams and downstream analytics, AI/ML, and governance systems.

**Overview**

StreamNative Lakehouse Tables bridge the gap between streaming and batch systems by converting message data from topics into table-backed datasets. Each Lakehouse Table maintains:

* Metadata (schema, manifest lists, snapshots)
* Data files (Parquet/columnar format)
* Transaction logs (for table evolution)

These components reside in user-controlled object storage, ensuring low cost, high durability, and interoperability with a broad ecosystem of tools.

**Key Capabilities**

**1. Open Table Format Support**

StreamNative Lakehouse Tables support industry-standard formats:

* Apache Iceberg
* Delta Lake (Delta 2.0 and above)

This ensures compatibility with engines such as Databricks, Snowflake, Spark, Trino, Flink, BigQuery, StarTree, and more.

**2. Native Topic-to-Table Mapping**

Each table is backed by one or more StreamNative topics. Data is automatically:

* Ingested from the streaming topic
* Serialized into Parquet files
* Committed into the table as immutable snapshots
* Made available for SQL queries and analytical engines

This provides a streaming-first lakehouse architecture without external ingestion pipelines.

**3. Object Storage as the Source of Truth**

All table artifacts are stored in object storage such as:

* Amazon S3
* Google Cloud Storage
* Azure Blob Storage

This enables low-cost storage, independent scaling, and easy interoperability.

**4. Schema-Aware and Schema-Safe**

Lakehouse Tables use StreamNative’s schema registry to ensure:

* Schema inference from topics
* Backward/forward compatible evolution
* Safe writes with schema enforcement
* Automatic mapping to Iceberg/Delta schemas
* Users retain full control over table evolution policies.

**5. Transactional Guarantees**

Using open table format guarantees, Lakehouse Tables support:

* ACID transactions
* Snapshot isolation
* Time travel (via historical snapshots)
* Incremental reads

This brings reliability and consistency to streaming data workflows.

6. Full Interoperability with Data and AI Platforms

Once a table is materialized in Iceberg or Delta Lake format, it is fully queryable by:

* Databricks
* Snowflake
* BigQuery Managed Tables
* Apache Spark, Flink, and Trino
* StarTree and Pinot
* DuckDB
* pandas & PyArrow

No connectors or intermediate ETL is required.


# Connect to your cluster using the Kafka Stream
Source: https://docs.streamnative.io/cloud/process/kafka-streams-and-ksql/cloud-connect-kafka-stream



<Note title="Note">
  This QuickStart assumes that you have created a StreamNative cluster with the Kafka protocol enabled, created a service account, and granted the service account `produce` and `consume` permissions to a namespace for the target topic.

  If you are using a [Ursa-Engine](/cloud/overview/data-streaming-engine) powered cluster, please note that KStreams and KSQLDB support in Ursa Engine has certain limitations. It doesn't support functionalities that require transactions and topic compaction.
</Note>

This document describes how to connect to your StreamNative cluster using the [Kafka Stream](https://kafka.apache.org/documentation/streams/) with OAuth2 authentication.

## Before you begin

* Get the OAuth2 credential file.

  1. On the left navigation pane, click **Service Accounts**.
  2. In the row of the service account you want to use, in the **Key File** column, click the **Download** icon to download the OAuth2 credential file to your local directory.

* Get the service URL of your StreamNative cluster.

  1. On the left navigation pane, in the **Admin** area, click **Pulsar Clusters**.
  2. Select the **Details** tab, and in the **Access Points** area, click **Copy** at the end of the row of the **Kafka Service URL (TCP)**.

- Create two topics for the kafka Stream application, named `<kafka-stream-application-name>-counts-store-repartition` and `<kafka-stream-application-name>-counts-store-changelog`.

## Steps

1. Add the Kafka Stream and OAuth Maven dependencies.

   ```xml theme={null}
   <dependency>
       <groupId>org.apache.kafka</groupId>
       <artifactId>kafka-clients</artifactId>
       <version>3.4.0</version>
   </dependency>
   <dependency>
       <groupId>io.streamnative.pulsar.handlers</groupId>
       <artifactId>oauth-client</artifactId>
       <version>3.1.0.1</version>
   </dependency>

   <!-- Optional: add an SLF4J dependency if you want to see the log output -->
   <dependency>
       <groupId>org.slf4j</groupId>
       <artifactId>slf4j-log4j12</artifactId>
       <version>1.7.30</version>
   </dependency>
   ```

2. (Optional) Add the [RocksDB](https://rocksdb.org/) dependency if you encounter the `java.lang.UnsatisfiedLinkError` error when starting the Kafka Stream application.

   ```xml theme={null}
   <dependency>
       <groupId>org.rocksdb</groupId>
       <artifactId>rocksdbjni</artifactId>
       <version>7.0.3</version>
   </dependency>
   ```

3. Build a Kafka Stream application.

   This example builds a Kafka Stream application named `wordcount-application`.

   ```java theme={null}
   import io.streamnative.pulsar.handlers.kop.security.oauth.OauthLoginCallbackHandler;
   import java.util.concurrent.CountDownLatch;
   import org.apache.kafka.common.serialization.Serdes;
   import org.apache.kafka.common.utils.Bytes;
   import org.apache.kafka.streams.KafkaStreams;
   import org.apache.kafka.streams.StreamsBuilder;
   import org.apache.kafka.streams.StreamsConfig;
   import org.apache.kafka.streams.kstream.KStream;
   import org.apache.kafka.streams.kstream.KTable;
   import org.apache.kafka.streams.kstream.Materialized;
   import org.apache.kafka.streams.state.KeyValueStore;
   import org.apache.kafka.clients.CommonClientConfigs;

   import java.util.Arrays;
   import java.util.Properties;
   import org.apache.log4j.BasicConfigurator;
   import org.apache.log4j.Level;
   import org.apache.log4j.Logger;

   public class SNWordCountApplication {

       public static void main(final String[] args) {

           BasicConfigurator.configure();
           Logger.getRootLogger().setLevel(Level.INFO);

           // Step 1: replace with your configurations
           String serverUrl = "SERVER-URL";
           String keyPath = "YOUR-KEY-FILE-PATH";
           String audience = "YOUR-AUDIENCE-STRING";

           // Step 2: create Kafka Stream properties
           Properties props = new Properties();
           // stream application name
           props.put(StreamsConfig.APPLICATION_ID_CONFIG, "wordcount-application");
           props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, serverUrl);
           props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
           props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());

           // OAuth config
           props.setProperty("sasl.login.callback.handler.class", OauthLoginCallbackHandler.class.getName());
           props.setProperty("security.protocol", "SASL_SSL");
           props.setProperty("sasl.mechanism", "OAUTHBEARER");
           final String jaasTemplate = "org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required"
                   + " oauth.issuer.url=\"%s\""
                   + " oauth.credentials.url=\"%s\""
                   + " oauth.audience=\"%s\";";
           props.setProperty("sasl.jaas.config", String.format(jaasTemplate,
                   "https://auth.streamnative.cloud/",
                   "file://" + keyPath,
                   audience
           ));

           // Step 3: build the Kafka Stream process
           String inputTopic = "TextLinesTopic";

           StreamsBuilder builder = new StreamsBuilder();
           KStream<String, String> textLines = builder.stream(inputTopic);
           KTable<String, Long> wordCounts = textLines
                   .flatMapValues(textLine -> {
                       System.out.println("stream application receive: " + textLine);
                       return Arrays.asList(textLine.toLowerCase().split("\\W+"));
                   })
                   .groupBy((key, word) -> word)
                   .count(Materialized.<String, Long, KeyValueStore<Bytes, byte[]>>as("counts-store"));
           wordCounts.toStream()
                   .foreach((word, count) -> System.out.println("word: " + word + " -> " + count));

           KafkaStreams streams = new KafkaStreams(builder.build(), props);

           final CountDownLatch latch = new CountDownLatch(1);
           Runtime.getRuntime().addShutdownHook(new Thread("stream") {
               @Override
               public void run() {
                   streams.close();
                   latch.countDown();
               }
           });

           try {
               // Step 4: start the Kafka Stream
               streams.start();
               latch.await();
           } catch (InterruptedException e) {
               e.printStackTrace();
           }
           System.exit(0);
       }
   }
   ```

   * `serverUrl`: the Kafka service URL of your StreamNative cluster.
   * `keyPath`: the path to your downloaded OAuth2 credential file.
   * `audience`: the `audience` parameter is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.

4. Run the Kafka Stream application to check the connectivity.

   1. Open another terminal and use the Kafka CLI tool to send some messages to the `TextLinesTopic` topic.

      ```bash theme={null}
      ./bin/kafka-console-producer.sh \
          --bootstrap-server `SERVER-URL` \
          --producer.config ./kafka.properties \
          --topic TextLinesTopic
      ```

   2. Type some texts.

      ```bash theme={null}
      hello world
      hello world hello world
      ```

   Then, you should see the following input:

   ```bash theme={null}
   stream application receive: hello world
   stream application receive: hello world hello world
   word: hello -> 3
   word: world -> 3
   ```


# Connect to your cluster using KSQL
Source: https://docs.streamnative.io/cloud/process/kafka-streams-and-ksql/cloud-connect-ksql



<Note title="Note">
  This QuickStart assumes that you have created a StreamNative cluster with the Kafka protocol enabled, created a service account, and granted the service account `produce` and `consume` permissions to a namespace for the target topic.

  If you are using a [Ursa-Engine](/cloud/overview/data-streaming-engine) powered cluster, please note that KStreams and KSQLDB support in Ursa Engine has certain limitations. It doesn't support functionalities that require transactions and topic compaction.
</Note>

This document describes how to connect to your StreamNative cluster using [KSQL](https://www.confluent.io/blog/ksql-streaming-sql-for-apache-kafka/) with [SASL/PLAIN](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#kafka-clients) authentication.

## Before you begin

<Note title="Note">
  * Before using an API key, verify that the service account is authorized to access the resources, such as tenants, namespaces, and topics.
  * The password for different utilities as `kcat` will be equal to `token:<API KEY>`.
</Note>

You can follow the instructions to [create an API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#using-api-keys-to-connect-to-your-cluster) for the service account you choose to use.

* [Download the KSQL server and KSQL CLI tool](https://docs.confluent.io/platform/current/installation/available_packages.html#confluent-ksqldb).

## Steps

1. Open the `etc/ksqldb/ksql-server.properties` file and configure the KSQL server with the following properties:

   ```conf theme={null}
   #------ Kafka -------
   # The set of Kafka brokers to bootstrap Kafka cluster information from:
   bootstrap.servers=<SERVER-URL>
   security.protocol=SASL_SSL
   sasl.mechanism=PLAIN
   sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule \
     required username="public/default" \
     password="token:<YOUR-API-KEY>";
   ```

   * `bootstrap.servers`: the Kafka service URL of your StreamNative cluster.
   * `password`: an API key of your service account.

2. Start the KSQL server.

   ```shell theme={null}
   bin/ksql-server-start etc/ksqldb/ksql-server.properties
   ```

   After the KSQL server is started, you should see the following output:

   ```text theme={null}
   [2023-04-06 14:51:02,811] INFO ksqlDB API server listening on http://0.0.0.0:8088 (io.confluent.ksql.rest.server.KsqlRestApplication:382)

                     ===========================================
                     =       _              _ ____  ____       =
                     =      | | _____  __ _| |  _ \| __ )      =
                     =      | |/ / __|/ _` | | | | |  _ \      =
                     =      |   <\__ \ (_| | | |_| | |_) |     =
                     =      |_|\_\___/\__, |_|____/|____/      =
                     =                   |_|                   =
                     =        The Database purpose-built       =
                     =        for stream processing apps       =
                     ===========================================

   Copyright 2017-2022 Confluent Inc.

   Server 7.3.2 listening on http://0.0.0.0:8088

   To access the KSQL CLI, run:
   ksql http://0.0.0.0:8088

   [2023-04-06 14:51:02,814] INFO Server up and running (io.confluent.ksql.rest.server.KsqlServerMain:153)
   [2023-04-06 14:51:04,117] INFO Successfully submitted metrics to Confluent via secure endpoint (io.confluent.support.metrics.submitters.ConfluentSubmitter:146)
   ```

3. Start the KSQL CLI tool.

   ```shell theme={null}
   LOG_DIR=./ksql_logs bin/ksql http://localhost:8088
   ```

   After the KSQL CLI tool is started, you should see the following output:

   ```text theme={null}
   CLI v7.3.2, Server v7.3.2 located at http://localhost:8088
   Server Status: RUNNING
   ```

4. Create a stream and tables using the KSQL CLI tool.

   a. Create a stream named `riderLocations`:

   ```sql theme={null}
   CREATE STREAM riderLocations (profileId VARCHAR, latitude DOUBLE, longitude DOUBLE)
     WITH (kafka_topic='locations', value_format='json', partitions=1);
   ```

   b. Create two tables (`currentLocation` and `ridersNearMountainView` ) to track the latest location of the riders using a materialized view.

   ```sql theme={null}
   CREATE TABLE currentLocation AS
     SELECT profileId,
           LATEST_BY_OFFSET(latitude) AS la,
           LATEST_BY_OFFSET(longitude) AS lo
     FROM riderlocations
     GROUP BY profileId
     EMIT CHANGES;
   ```

   ```sql theme={null}
   CREATE TABLE ridersNearMountainView AS
     SELECT ROUND(GEO_DISTANCE(la, lo, 37.4133, -122.1162), -1) AS distanceInMiles,
           COLLECT_LIST(profileId) AS riders,
           COUNT(*) AS count
     FROM currentLocation
     GROUP BY ROUND(GEO_DISTANCE(la, lo, 37.4133, -122.1162), -1);
   ```

5. Insert and query data.

   a. Open a terminal to run a push query over the stream.

   ```sql theme={null}
   -- Mountain View lat, long: 37.4133, -122.1162
   SELECT * FROM riderLocations
     WHERE GEO_DISTANCE(latitude, longitude, 37.4133, -122.1162) <= 5 EMIT CHANGES;
   ```

   b. Open another terminal to start another KSQL CLI tool and insert data into the stream.

   ```sql theme={null}
   INSERT INTO riderLocations (profileId, latitude, longitude) VALUES ('c2309eec', 37.7877, -122.4205);
   INSERT INTO riderLocations (profileId, latitude, longitude) VALUES ('18f4ea86', 37.3903, -122.0643);
   INSERT INTO riderLocations (profileId, latitude, longitude) VALUES ('4ab5cbad', 37.3952, -122.0813);
   INSERT INTO riderLocations (profileId, latitude, longitude) VALUES ('8b6eae59', 37.3944, -122.0813);
   INSERT INTO riderLocations (profileId, latitude, longitude) VALUES ('4a7c7b41', 37.4049, -122.0822);
   INSERT INTO riderLocations (profileId, latitude, longitude) VALUES ('4ddad000', 37.7857, -122.4011);
   ```

   You should see the following output in the first terminal:

   ```sql theme={null}
   >  WHERE GEO_DISTANCE(latitude, longitude, 37.4133, -122.1162) <= 5 EMIT CHANGES;
   +---------------------------------+---------------------------------+---------------------------------+
   |PROFILEID                        |LATITUDE                         |LONGITUDE                        |
   +---------------------------------+---------------------------------+---------------------------------+
   |4ab5cbad                         |37.3952                          |-122.0813                        |
   |8b6eae59                         |37.3944                          |-122.0813                        |
   |4a7c7b41                         |37.4049                          |-122.0822                        |
   ```


# Kafka Streams & KSQL
Source: https://docs.streamnative.io/cloud/process/kafka-streams-and-ksql/kafka-streams-and-ksql



StreamNative Cloud provides a fully managed data streaming services that is compatible with Kafka protocol. Despite we don't offer fully managed Kafka streams applications and KSQL on Cloud, you can still use Kafka Streams and KSQL to build your streaming applications on StreamNative Cloud.

Below are the guides to help you get started with Kafka Streams and KSQL on StreamNative Cloud.

* [Kafka Streams QuickStart](/cloud/process/kafka-streams-and-ksql/cloud-connect-kafka-stream)
* [KSQL QuickStart](/cloud/process/kafka-streams-and-ksql/cloud-connect-ksql)


# Manage Kafka Clusters
Source: https://docs.streamnative.io/kafka/kafka-cluster-guide

Create, configure, and manage Kafka clusters on StreamNative Cloud with the Ursa Engine.

StreamNative Kafka Service runs on the Ursa Engine, a lakehouse-native stream storage engine that delivers native Kafka API with lakehouse-native storage. You can use standard Kafka clients, tools, and ecosystems to produce and consume data without modifying application code.

This guide walks you through choosing a cluster profile, selecting a deployment option, creating a Kafka cluster, and configuring it for production workloads.

## Cluster profiles

StreamNative provides two cluster profiles for Kafka clusters. Choose a profile based on your workload's latency requirements and cost sensitivity.

<Tabs>
  <Tab title="Cost-Optimized">
    The Cost-Optimized profile uses the Ursa Engine with object storage (Amazon S3, Google Cloud Storage, or Azure Blob Storage) as the primary data persistence layer. This profile is ideal for workloads where throughput and cost efficiency matter more than ultra-low latency.

    **Best for:**

    * Event streaming and data pipelines
    * Log aggregation and analytics
    * Change data capture (CDC)
    * Long-term data retention

    **Performance characteristics:**

    * Sub-second end-to-end latency (typically above 200 ms)
    * Up to 95% lower storage cost compared to disk-based clusters
    * Unlimited, elastic storage capacity

    <Tip>
      The Cost-Optimized profile uses Oxia for metadata management and leverages cloud-native object storage, making it well-suited for workloads with large data volumes and longer retention periods.
    </Tip>
  </Tab>

  <Tab title="Latency-Optimized">
    The Latency-Optimized profile keeps the classic Kafka disk-based architecture, using KRaft for controller management and ISR (In-Sync Replicas) for data replication. This profile is designed for real-time, interactive, and mission-critical workloads that require consistently fast data access.

    **Best for:**

    * Real-time trading and financial systems
    * Gaming and interactive applications
    * Fraud detection and event processing
    * User activity tracking with strict latency requirements

    **Performance characteristics:**

    * Sub-10 ms end-to-end latency (typically 5-200 ms)
    * Predictable, low-latency performance under high throughput
    * Disk-based storage for fast reads and writes
  </Tab>
</Tabs>

For a detailed comparison of profile features by deployment type, see [Cluster Profiles](/cloud/clusters/cluster-profiles-overview).

## Deployment options

StreamNative offers Kafka clusters in Dedicated and BYOC deployment options on AWS and Google Cloud, and BYOC deployment options on Microsoft Azure.

<CardGroup>
  <Card title="Dedicated" icon="server">
    Fully managed clusters on StreamNative infrastructure with dedicated resources on AWS or Google Cloud. Supports multi-AZ high availability.
  </Card>

  <Card title="BYOC" icon="cloud">
    Deploy clusters in your own cloud account on AWS, Google Cloud, or Microsoft Azure while StreamNative manages operations. Provides private networking and data sovereignty.
  </Card>
</CardGroup>

<Note>
  Kafka Clusters are not available in Serverless deployment yet. Serverless support for Kafka Clusters is coming soon.
</Note>

For a full feature comparison across deployment options, see [Cluster Types and Regions](/cloud/clusters/cluster-types).

## Create a Kafka cluster

Follow these steps to create a Kafka cluster using the StreamNative Console.

### Prerequisites

* A StreamNative Cloud account. If you do not have one, [sign up](https://console.streamnative.cloud/).
* An organization in StreamNative Cloud. For details, see [Organizations](/cloud/security/access/resource-hierarchy/organizations).

### Steps

1. **Log in and create an organization** (if you have not already). Log in to the [StreamNative Console](https://console.streamnative.cloud/) and create or select your organization.

2. **Create an instance.** Navigate to **Instances** and click **New**. Select your deployment type (**Dedicated** or **BYOC**). Enter a name for your instance, select your preferred cloud provider and region, and then proceed.

3. **Choose a resource type.** On the **Resource Type** page, select **Kafka Cluster**. The page displays a comparison between Pulsar Cluster and Kafka Cluster with their supported features.

   <img alt="Resource Type Selection" />

4. **Configure the cluster.** Enter a cluster name, select your cloud environment, and choose a cluster profile (**Latency Optimized** or **Cost Optimized**). Select your preferred availability zone configuration (Multi AZ is recommended for production workloads).

   <img alt="Cluster Configuration" />

5. **Configure lakehouse table (optional).** On the **Lakehouse Table** page, optionally enable lakehouse table support for your cluster.

   <img alt="Lakehouse Table" />

6. **Set the cluster size.** Configure the cluster size using **Throughput Units**. Each Throughput Unit provides a defined capacity for ingress (data in), egress (data out), and data entries per second. Adjust the slider to match your expected workload.

   <img alt="Cluster Size" />

7. **Finish.** Review and confirm your configuration to create the cluster.

Wait for the cluster to finish provisioning. The cluster is ready when all components show a healthy status.

<Note>
  Each StreamNative instance can support multiple clusters. However, Pulsar Clusters and Kafka Clusters cannot currently co-exist in the same instance.
</Note>

For step-by-step instructions for each deployment type, see:

* [Manage Dedicated Clusters](/cloud/clusters/manage-clusters/manage-dedicated-clusters)
* [Manage BYOC Clusters](/cloud/clusters/manage-clusters/manage-byoc-clusters)

## Topic management

You can create and manage Kafka topics through the StreamNative Console, the Kafka CLI, or any Kafka AdminClient-compatible tool. You can use standard Kafka APIs to configure topics, partitions, and retention policies.

When configuring topics, consider the following settings:

* **Partitions**: Set the number of partitions based on your target parallelism and throughput. You can increase partitions after creation, but you cannot decrease them.
* **Retention**: Configure time-based or size-based retention policies to control how long messages are stored. On the Cost-Optimized profile, object storage provides cost-efficient long-term retention.
* **Replication**: StreamNative manages replication based on your cluster profile and availability zone configuration.

### Consumer group management

StreamNative supports standard Kafka consumer groups. You can monitor and manage consumer groups through the StreamNative Console or Kafka CLI tools.

Key operations include:

* Viewing active consumer groups and their members
* Monitoring consumer lag per partition
* Resetting consumer group offsets

For details on connecting Kafka consumers, see [Build Kafka Client Applications](/cloud/build/kafka-clients/kafka-on-cloud).

### Kafka Queues with share groups

StreamNative Kafka Service supports Kafka Queues across both Latency-Optimized and Cost-Optimized profiles. Kafka Queues use the share groups consumption model, which lets multiple consumers cooperatively read records from the same partition instead of enforcing a strict 1:1 partition-to-consumer mapping. This unlocks queue-style task distribution, individual message acknowledgment and retry, and elastic consumer scaling beyond the partition count—useful for AI agent task orchestration, notification fan-out, image and document processing, job scheduling, and bursty async workers. To enable queue semantics, configure your Kafka client consumer with a share group ID; see [Build Kafka Client Applications](/cloud/build/kafka-clients/kafka-on-cloud) for client setup details.

## Scaling

StreamNative Kafka clusters use **Throughput Units** for scaling. Each Throughput Unit provides a defined amount of ingress, egress, and data entry throughput. Adjust the number of Throughput Units to match your workload requirements.

<Tip>
  For Cost-Optimized clusters, storage scales automatically with object storage. For Latency-Optimized clusters, disk capacity scales with Throughput Units.
</Tip>

## Related topics

* [Kafka Cluster vs. KSN on Pulsar Clusters](/kafka/kafka-cluster-vs-ksn)
* [Kafka Client Applications on StreamNative Cloud](/cloud/build/kafka-clients/kafka-on-cloud)
* [Kafka Compatibility](/cloud/build/kafka-clients/compatibility/kafka-compatibility)
* [Kafka Schema Registry](/cloud/governance/kafka-schemas/kafka-schema-registry)


# Kafka Cluster vs. KSN on Pulsar Clusters
Source: https://docs.streamnative.io/kafka/kafka-cluster-vs-ksn

Compare native Kafka Clusters with Kafka compatibility on Pulsar Clusters (KSN). Understand when to use each option.

StreamNative Cloud offers two ways to run Kafka workloads: **Kafka Clusters** (native Apache Kafka) and **Kafka compatibility on Pulsar Clusters** (via KSN). This page helps you understand the differences and choose the right option.

## Overview

* **Kafka Clusters** run native Apache Kafka on the Ursa Engine. There is no protocol translation — your Kafka clients connect directly to a native Kafka broker. This is the recommended option for Kafka workloads.
* **KSN** (on Pulsar Clusters) provides Kafka API compatibility within Pulsar Clusters. KSN translates the Kafka protocol to Pulsar storage, allowing Kafka clients to produce and consume messages on Pulsar topics. This option is best when you need both Kafka and Pulsar clients on the same cluster.

## Feature comparison

| Feature                      | Kafka Cluster                                                      | KSN                                                                                             |
| ---------------------------- | ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------- |
| **Protocol**                 | Native Apache Kafka                                                | Kafka protocol translated to Pulsar                                                             |
| **Kafka API compatibility**  | 100% — native Kafka                                                | High — see [compatibility matrix](/cloud/build/kafka-clients/compatibility/kafka-compatibility) |
| **Transactions**             | Supported on Latency-Optimized; coming soon on Cost-Optimized      | Supported on Latency-Optimized; coming soon on Cost-Optimized                                   |
| **Topic compaction**         | Supported on Latency-Optimized; coming soon on Cost-Optimized      | Supported on Latency-Optimized; coming soon on Cost-Optimized                                   |
| **Kafka Streams**            | Supported                                                          | Supported with limitations on Cost Optimized                                                    |
| **KSqlDB**                   | Supported                                                          | Supported with limitations on Cost Optimized                                                    |
| **Pulsar client access**     | Not available                                                      | Supported — both Pulsar and Kafka clients on the same cluster                                   |
| **Multi-tenancy**            | Topic-level ACLs                                                   | Tenant and namespace isolation                                                                  |
| **Built-in geo-replication** | Via [Universal Linking](/cloud/universal-linking/unilink-overview) | Supported                                                                                       |
| **Subscription types**       | Consumer groups                                                    | Exclusive, shared, failover, key-shared                                                         |
| **Retention policy**         | Standard Kafka retention (time/size-based)                         | Pulsar retention (subscription-based by default)                                                |
| **Cluster profiles**         | Latency-Optimized and Cost-Optimized                               | Latency-Optimized and Cost-Optimized                                                            |
| **Status**                   | Public Preview                                                     | GA                                                                                              |

## When to use Kafka Clusters

Choose Kafka Clusters when:

* You want **full native Kafka behavior** with no protocol translation.
* Your workloads are **Kafka-only** and you do not need Pulsar or MQTT client access.
* You need **Kafka transactions and topic compaction** on the Ursa Engine.
* You are **migrating from Amazon MSK, Confluent, or self-managed Kafka** and want a drop-in replacement.
* You want **standard Kafka retention policies** without learning Pulsar retention concepts.

## When to use KSN on Pulsar Clusters

Choose KSN on Pulsar Clusters when:

* You need **multi-protocol access** — both Kafka and Pulsar clients reading from the same topics.
* You have **existing Pulsar Clusters** and want to add Kafka client access to them.
* You need **Pulsar-specific features** like built-in geo-replication, flexible subscriptions, or multi-tenancy with tenant/namespace isolation.
* You want **MQTT support** on the same cluster via MoP.

## Migration between options

If you are currently using KSN on Pulsar Clusters and want to move to native Kafka Clusters, you can use [Universal Linking](/cloud/universal-linking/unilink-overview) to replicate data between clusters during the transition.

<Note>
  Kafka Clusters and Pulsar Clusters cannot currently co-exist in the same StreamNative instance. Plan your instance topology accordingly.
</Note>

## Next steps

<CardGroup>
  <Card title="Get Started with Kafka Service" icon="rocket" href="/kafka/kafka-getting-started">
    Create a native Kafka Cluster and produce your first message.
  </Card>

  <Card title="Kafka on Pulsar Clusters" icon="arrows-rotate" href="/cloud/build/kafka-clients/kafka-on-cloud">
    Set up Kafka compatibility on your existing Pulsar Clusters using KSN.
  </Card>
</CardGroup>


# Get Started with Kafka Service
Source: https://docs.streamnative.io/kafka/kafka-getting-started

Start producing and consuming messages with StreamNative Kafka Service in under 5 minutes.

Get started with StreamNative Kafka Service. This guide walks you through creating a Kafka cluster and producing your first message.

## Prerequisites

<Note>
  Before you begin, make sure you have the following:

  * A [StreamNative Cloud account](https://console.streamnative.cloud/?defaultMethod=signup). If you do not have one, sign up for a free trial.
  * A supported web browser (Chrome, Firefox, Safari, or Edge).
  * [Apache Kafka CLI tools](https://kafka.apache.org/downloads) (v3.1.0 or later) installed on your local machine.
</Note>

## Step 1: Log in to StreamNative Cloud Console

Navigate to the [StreamNative Cloud Console](https://console.streamnative.cloud) and sign in with your credentials.

## Step 2: Create a Kafka cluster

Create an organization, an instance, and a Kafka cluster powered by the Ursa Engine.

1. In the upper-right corner, click your profile icon and select **Organizations**.

2. Click **Create Organization** and enter a name for your organization.

3. On the left navigation pane, click **Dashboard**.

4. On the **Instances** card, click **New**, then select your deployment type (**Dedicated** or **BYOC**).

5. Enter a name for your instance, select your preferred cloud provider and region, and proceed to the next step.

6. On the **Resource Type** page, select **Kafka Cluster**.

   <img alt="Resource Type Selection" />

7. Enter a name for your cluster, select your cloud environment, and choose a cluster profile (**Latency Optimized** or **Cost Optimized**). Select your availability zone configuration.

   <img alt="Cluster Configuration" />

8. Optionally configure lakehouse table settings, then proceed to **Cluster Size**.

   <img alt="Lakehouse Table" />

9. Configure the cluster size using the **Throughput Units** slider to match your expected workload, then click **Finish**.

   <img alt="Cluster Size" />

Wait for the cluster to finish provisioning. The cluster is ready when all components show a healthy status.

<Tip>
  For detailed instructions on configuring Kafka clusters, including advanced options and profile selection, see [Create a Kafka Cluster](/kafka/kafka-cluster-guide).
</Tip>

## Step 3: Create a service account and API key

Create a service account and generate an API key for authenticating your Kafka clients.

1. On the left navigation pane, click **Service Accounts**.
2. Click **Create Service Account**, enter a name, and click **Confirm**.
3. Select the service account you created, then click the **API Keys** tab.
4. Click **Create API Key**, copy the generated key, and store it securely.

<Note>
  Grant your service account `produce` and `consume` permissions on the topics you plan to use. Navigate to **Admin > Topics**, select your topic, and assign the appropriate permissions to your service account.
</Note>

## Step 4: Produce and consume messages

Use the Kafka CLI tools to produce and consume messages on your cluster.

First, create a configuration file named `kafka.properties` with your connection details:

```properties theme={null}
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="public/default" \
  password="token:YOUR_API_KEY";
```

Replace `YOUR_API_KEY` with the API key you generated in the previous step.

Your bootstrap server endpoint follows this format:

```
<cluster-name>-<instance-name>.<org-name>.streamnative.cloud:9093
```

<CodeGroup>
  ```bash Producer theme={null}
  kafka-console-producer.sh \
    --bootstrap-server <cluster-name>-<instance-name>.<org-name>.streamnative.cloud:9093 \
    --producer.config kafka.properties \
    --topic my-first-topic
  ```

  ```bash Consumer theme={null}
  kafka-console-consumer.sh \
    --bootstrap-server <cluster-name>-<instance-name>.<org-name>.streamnative.cloud:9093 \
    --consumer.config kafka.properties \
    --topic my-first-topic \
    --from-beginning \
    --group my-consumer-group
  ```
</CodeGroup>

1. Open a terminal and start the consumer. The consumer waits for messages on the `my-first-topic` topic.
2. Open a second terminal and start the producer.
3. Type a message in the producer terminal (for example, `Hello, Kafka!`) and press **Enter**.
4. Verify that the message appears in the consumer terminal.

<Tip>
  The `--from-beginning` flag tells the consumer to read from the earliest offset in the partition. The `--group` flag assigns the consumer to the `my-consumer-group` consumer group, which tracks the offsets for your consumer.
</Tip>

## Next steps

<CardGroup>
  <Card title="Kafka Client Guides" icon="code" href="/cloud/build/kafka-clients/kafka-on-cloud">
    Connect your applications using Kafka client libraries for Java, Python, Go, Node.js, and more.
  </Card>

  <Card title="Kafka Compatibility" icon="list-check" href="/cloud/build/kafka-clients/compatibility/kafka-compatibility">
    Check supported Kafka APIs, client versions, and feature compatibility.
  </Card>

  <Card title="Migration Guide" icon="arrow-right-arrow-left" href="/kafka/kafka-migration-guide">
    Migrate your existing Kafka workloads to StreamNative Kafka Service with zero code changes.
  </Card>
</CardGroup>


# StreamNative Kafka Service
Source: https://docs.streamnative.io/kafka/overview

Native Apache Kafka service powered by the Ursa Engine. Leaderless, lakehouse-native, and up to 95% lower cost.

StreamNative Kafka Service is a fully managed, **native Apache Kafka** service built on the [Lakestream architecture](/cloud/overview/lakestream-overview) and powered by the [Ursa Engine](/cloud/overview/data-streaming-engine). It is not a Kafka compatibility layer — it runs native Apache Kafka with a lakehouse-native storage engine that delivers leaderless, compute-storage separated architecture with up to 95% lower infrastructure costs.

Your existing Kafka clients, Kafka Connect connectors, and Kafka Streams applications work without any code changes. Just point them at your StreamNative Kafka endpoint.

## Key advantages

<CardGroup>
  <Card title="Native Apache Kafka" icon="check">
    Runs native Apache Kafka — not a compatibility layer. Use any standard Kafka client, tool, or ecosystem without modification.
  </Card>

  <Card title="Up to 95% Lower Cost" icon="arrow-down">
    Leaderless architecture with compute/storage separation eliminates cross-AZ replication fees. Data writes directly to object storage.
  </Card>

  <Card title="Lakehouse-Native" icon="database">
    Every topic is simultaneously an Iceberg or Delta Lake table. Zero-copy streaming into your lakehouse with no connectors or ETL.
  </Card>

  <Card title="Zero Operations" icon="gear">
    Fully managed. No brokers to provision, no ZooKeeper to manage, no partitions to rebalance. Auto-scaling handles the rest.
  </Card>
</CardGroup>

## Coming from an existing Kafka deployment?

<CardGroup>
  <Card title="From Amazon MSK" icon="aws" href="/kafka/kafka-migration-guide#from-amazon-msk">
    Lower cost with lakehouse-native storage. No cross-AZ replication fees. Same Kafka APIs.
  </Card>

  <Card title="From Confluent" icon="arrows-rotate" href="/kafka/kafka-migration-guide#from-confluent-cloudplatform">
    Open formats, no vendor lock-in. Data stored in Iceberg and Delta Lake on your object storage.
  </Card>

  <Card title="From Self-Managed" icon="server" href="/kafka/kafka-migration-guide#from-self-managed-apache-kafka">
    Zero operational overhead. Auto-scaling, managed infrastructure, and enterprise-grade security.
  </Card>
</CardGroup>

## Deployment options

StreamNative Kafka Service is available in multiple deployment configurations:

| Option         | Description                                         | Best for                              |
| -------------- | --------------------------------------------------- | ------------------------------------- |
| **Serverless** | Fully managed, pay-per-use                          | Development, testing, small workloads |
| **Dedicated**  | Dedicated infrastructure, predictable performance   | Production workloads                  |
| **BYOC**       | Runs in your cloud account, managed by StreamNative | Data sovereignty, compliance          |
| **BYOC Pro**   | Advanced BYOC with custom networking and DNS        | Enterprise, regulated industries      |

See [Cluster Types](/cloud/clusters/cluster-types) for detailed feature comparison.

## How it works

StreamNative Kafka Service runs on the **Ursa Engine**, a cloud-native storage engine at the heart of the [Lakestream architecture](/cloud/overview/lakestream-overview).

* **Stateless brokers**: Kafka brokers are stateless and leaderless. Any broker can handle produce or fetch requests for any partition.
* **Object storage**: Data writes directly to S3, GCS, or Azure Blob Storage. No local disks, no inter-broker replication.
* **Lakehouse-native**: Data is stored in open table formats (Iceberg, Delta Lake) on your object storage, queryable by any analytics engine.

Learn more about the [Lakestream architecture](/cloud/overview/lakestream-overview) and the [Ursa Engine](/cloud/overview/data-streaming-engine).

## Get started

<CardGroup>
  <Card title="Quick Start" icon="play" href="/kafka/kafka-getting-started">
    Create a Kafka cluster and produce your first message in under 5 minutes.
  </Card>

  <Card title="Migration Guide" icon="right-left" href="/kafka/kafka-migration-guide">
    Migrate from MSK, Confluent, or self-managed Kafka with zero code changes.
  </Card>

  <Card title="Kafka Compatibility" icon="list-check" href="/cloud/build/kafka-clients/compatibility/kafka-compatibility">
    Check supported Kafka APIs, client versions, and feature compatibility.
  </Card>

  <Card title="Choose Kafka or Pulsar" icon="code-compare" href="/cloud/overview/choose-kafka-or-pulsar">
    Not sure which protocol fits your workload? Compare Kafka and Pulsar side by side.
  </Card>
</CardGroup>


# StreamNative Weekly Release Notes v2.10.3.2
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/component-changelogs-v2.10.3.2



# StreamNative Weekly Release Notes v2.10.3.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.10.3.2](https://github.com/streamnative/pulsar/releases/tag/v2.10.3.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.10.3.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.10.3.2/images/sha256-1376926751ee493439788cd87836f90344a3ac2d54e8c2cb1037cb047069112c)

## General Changes

### Apache Pulsar

\[branch-2.10]\[fix] Fix the compile issue of Branch-2.10

\[improve]\[broker] Add ref count for sticky hash to optimize the performance of Key\_Shared subscription

Debezium sources: Support loading config from secrets

### KoP

Optimize performance of EncodeResult.updateProducerStats and DecodeResult.updateConsumerStats

Remove unnecessary Boolean boxing

### Function Mesh Worker Service

Check FunctionMesh when init


# StreamNative Weekly Release Notes v2.10.3.3
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/component-changelogs-v2.10.3.3



# StreamNative Weekly Release Notes v2.10.3.3

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.10.3.3](https://github.com/streamnative/pulsar/releases/tag/v2.10.3.3)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.10.3.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.10.3.3/images/sha256-1dfa159f4c373e9189e1d9860235b7153f5ad1c251f1772c6dbff548f307aefc)

## General Changes

### Apache Pulsar

\[fix]\[broker] fixed the build error for pattern matching variable in lower JVM versions

\[fix]\[client] Fix reader listener can't auto ack with pooled message.

\[improve]\[broker] Replaced checkBackloggedCursors with checkBackloggedCursor(single subscription check) upon subscription

\[fix]\[cli]\[branch-2.10] Fix mbeans to json

\[improve]\[broker] Added isActive in ManagedCursorImpl

\[fix] \[ml] Topics stats shows msgBacklog but there reality no backlog

\[fix]\[broker] Fix open cursor with null-initialPosition result with earliest position

\[fix]\[build] Upgrade dependency-check-maven plugin to fix broken OWASP check

\[improve]\[sec] Suppress false positive OWASP reports

\[fix]\[broker] AbstractBatchedMetadataStore - use AlreadyClosedException instead of IllegalStateException

\[improve]\[websocket]\[branch-2.10] Add ping support

\[fix]\[broker] Pass subName for subscription operations in ServerCnx

### Cloud Storage Connector

\[fix]\[sink] Reset currentBatchSize & currentBatchBytes on failing records

\[fix]\[sink] Fix sink failing upon schema retrieval exceptions

\[docs] Add troubleshooting section to readme

### StreamNative Pulsar Plugins

\[fix]\[rest] Fix produce message without schema

### Function Mesh Worker Service

update function mesh to version v0.10.0

support hpa strategies

Support vpa

*Snowflake Connector*

fix docs format

Update content and code format in snowflake-sink.md

Fix stage name


# StreamNative Weekly Release Notes v2.10.3.4
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/component-changelogs-v2.10.3.4



# StreamNative Weekly Release Notes v2.10.3.4

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.10.3.4](https://github.com/streamnative/pulsar/releases/tag/v2.10.3.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.10.3.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.10.3.4/images/sha256-4b0e0632ce5b912b267694f0a8ad5657ac1df1957fa2e3efd48551ce671eeed4)

## General Changes

### Apache Pulsar

\[fix]\[security] Fix secure problem CVE-2017-1000487

\[cleanup]\[broker] Validate originalPrincipal earlier in ServerCnx

\[fix]\[broker] Make ServerCnx#originalAuthData volatile

\[fix]\[fn] Fix k8s merge runtime opts bug

\[fix]\[client] Fix async completion in ConsumerImpl#processPossibleToDLQ

\[fix]\[ml] Reset individualDeletedMessagesSerializedSize after acked all messages.

\[fix]\[authorization] Fix the return value of canConsumeAsync

\[Improve]\[broker] Support clear old bookie data for BKCluster

\[fix]\[broker] Remove timestamp from broker metrics

\[fix]\[broker] Fix race condition while updating partition number

\[cherry-pick]\[branch-2.10] Allow superusers to abort transactions

\[fix]\[txn] fix txn coordinator recover handle committing and aborting txn race condition.

\[improve]\[txn] Handle changeToReadyState failure correctly in TC client

\[fix] \[ml] The atomicity of multiple fields of ml is broken

\[fix]\[ml] Fix potential NPE cause future never complete.

\[fix]\[broker] Fix PulsarRegistrationClient and ZkRegistrationClient not aware rack info problem.

([#12615)](https://github.com/apache/pulsar/pull/12615)) \[revert]\[misc] "modify check waitingForPingResponse with volatile

\[cherry-pick]\[branch-2.10] Close TransactionBuffer when create persistent topic timeout

([#19129)](https://github.com/apache/pulsar/pull/19129)) Revert "\[fix]\[broker] Topic could be in fenced state forever if deletion fails

\[fix]\[broker] Expect msgs after server initiated CloseProducer

\[improve]\[broker] Copy subscription properties during updating the topic partition number.

\[improve]\[broker] Added isActive in ManagedCursorImpl

\[improve]\[broker] Added isActive in ManagedCursorImpl

\[fix]\[txn] Catch and log runtime exceptions in async operations

\[fix]\[broker] Topic could be in fenced state forever if deletion fails

\[fix] \[ml] Fix the incorrect total size if use ML interceptor

\[fix]\[broker] Support deleting partitioned topics with the keyword `-partition-`

\[improve]\[client] Change the get lastMessageId to debug level

\[fix] \[broker] getLastMessageId returns a wrong batch index of last message if enabled read compacted

\[fix]\[misc] do not require encryption on system topics

\[fix]\[broker]fix multi invocation for ledger createComplete

\[fix]\[txn] Correct the prompt message

\[fix]\[broker] Pass subscriptionName to auth service

\[fix]\[broker]optimize the shutdown sequence of broker service when it close

Close TransactionBuffer when MessageDeduplication#checkStatus failed

\[fix]\[io] Update Elasticsearch sink idle cnx timeout to 30s

\[fix]\[proxy] Only go to connecting state once

\[fix]\[client] Set fields earlier for correct ClientCnx initialization

\[fix]\[client] Prevent DNS reverse lookup when physical address is an IP address

### KoP

\[bugfix] Fix NPE in PendingTopicFutures and fix KafkaMessageOrderTestBase

\[cleanup] Remove static LOOKUP\_CLIENT\_MAP

Extract KafkaTopicLookupManager from KafkaTopicManager to decouple it from Kafka request handling and producer/consumer caching logic

Optimize performance of EncodeResult.updateProducerStats and DecodeResult.updateConsumerStats

\[flaky-test] Fix MultiLedgerTest.testListOffsetForEmptyRolloverLedger flaky test

\[fix] Make ProducerIdManagerImpl thread safe

\[fix] Use thread-safe list in TransactionMarkerChannelManager

Transactions - reduce log level

\[improve] Remove partition log when bundle unload

\[bugfix] Fix memory leak in case of closed connections with pending requests

\[Doc] - Add entryFormat description and performance test class

### StreamNative Pulsar Plugins

\[rest] Support consumer API manual ack message.

\[pulsar-rest] Supports automatic cleaning of idle consumers.

### Function Mesh Worker Service

Infer type class name

Support hot-reloading built-in connectors when the config file changes

### Lakehouse Connector

\[fix]\[sec] Fix CVEs introduced by log4j 1.2.17


# StreamNative Weekly Release Notes v2.10.3.5
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/component-changelogs-v2.10.3.5



# StreamNative Weekly Release Notes v2.10.3.5

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.10.3.5](https://github.com/streamnative/pulsar/releases/tag/v2.10.3.5)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.10.3.5/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.10.3.5/images/sha256-9bff7c9f1217379874eb931f16708d21c75278b3608fd09d851494f37cda0dd2)

## General Changes

### Apache Pulsar

\[improve] upgrade the bookkeeper version to 4.14.7

([#19302)](https://github.com/apache/pulsar/pull/19302)) Revert "\[improve] \[admin] Make the default value of param --get-subscription-backlog-size of admin API topics stats true

\[improve] \[admin] Make the default value of param --get-subscription-backlog-size of admin API topics stats true

\[fix] \[ml] topic load fail by ledger lost

\[branch-2.10]\[build]Fix check License

([#19425)](https://github.com/apache/pulsar/pull/19425)) Revert "\[improve] Upgrade to zk 3.8.1

\[improve] Upgrade lombok to 1.8.26

\[improve] Upgrade to zk 3.8.1

\[improve]\[misc] Upgrade Netty to 4.1.87.Final

\[improve] Upgrade wildfly-eytron (used by debezium) to fix CVE-2022-3143

\[branch-2.10]\[test]Run and fix tests

\[improve]\[broker] Follow up #19230 to tighten the validation scope

\[cleanup]\[broker] Simplify extract entryMetadata code in filterEntriesForConsumer

\[fix]\[client] Set authentication when using loadConf in client and admin client

\[improve]\[broker] Add UncaughtExceptionHandler for every thread pool

\[feature]\[txn] Fix individual ack batch message with transaction abort redevlier duplicate messages

\[fix]\[client] Fix authentication not update after changing the serviceUrl

\[improve]\[broker] Use shrink map for trackerCache

\[fix] \[broker] Incorrect service name selection logic

\[Improve]\[broker]Reduce GetReplicatedSubscriptionStatus local REST call

\[fix]\[broker] PulsarRegistrationClient - implement getAllBookies and follow BookieServiceInfo updates

\[fix]\[admin] Fix `validatePersistencePolicies` that Namespace/Topic persistent policies cannot set to \< 0

\[fix]\[broker]\[branch-2.10] Fix geo-replication admin

\[fix]\[broker] Copy command fields and fix potential thread-safety in ServerCnx

\[fix]\[client] Broker address resolution wrong if connect through a multi-dns names proxy

\[fix]\[broker] Allow proxy to pass same role for authRole and originalRole

\[fix]\[broker] Make authentication refresh threadsafe

\[fix]\[test] ProxyWithAuthorizationTest remove SAN from test certs

\[branch-2.10]\[fix]\[proxy] Fix using wrong client version in pulsar proxy

09f00eea93 \[fix]\[broker] Correct MockAlwaysExpiredAuthenticationState test impl

1935f070cf \[fix]\[broker] Call originalAuthState.authenticate in ServerCnx

\[improve]\[broker] Add test to verify authRole cannot change

\[feat]\[broker] Cherry-pick tests from

\[improve]\[broker] ServerCnx: go to Failed state when auth fails

\[improve]\[broker] Require authRole is proxyRole to set originalPrincipal

\[fix]\[broker]\[branch-2.10] Replace sync method call in async call chain to prevent ZK event thread deadlock

\[fix] \[ml] messagesConsumedCounter of NonDurableCursor was initialized incorrectly

\[fix]\[broker] Fix loadbalance score caculation problem

\[fix]\[broker] ServerCnx broken after recent cherry-picks

### KoP

Add multi-tenant support for OAuth authentication

### Cloud Storage Connector

\[chore] Rename io-cloud-storage-sink.md to cloud-storage-sink.md

### pulsarctl

fix: upgrade cobra

Add Fish shell completion

fix: fix token exp

### StreamNative Pulsar Plugins

\[rest] Fix Big message is truncated silently

Fix memory leak in that message is not pooled.

Add test to cover concurrent call

\[improve]\[rest] Create a new rest plugin document.

\[improve] \[Audit Log] Split Pub\&Sub and client registration events into different types

### Function Mesh Worker Service

Update function-mesh to v0.11.0

### Lakehouse Connector

([#122)](https://github.com/streamnative/pulsar-io-lakehouse/pull/122)) Revert "upgrade jdk17.

([#254)](https://github.com/streamnative/pulsar-io-lakehouse/pull/254)) Revert "Make unit tests can run with JDK8

Make unit tests can run with JDK8

Fix CVEs introduced by hadoop-common

cb71d3b Adjust codeowner to ecosystem

upgrade jdk17.


# StreamNative Weekly Release Notes v2.10.3.6
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/component-changelogs-v2.10.3.6



# StreamNative Weekly Release Notes v2.10.3.6

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.10.3.6](https://github.com/streamnative/pulsar/releases/tag/v2.10.3.6)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.10.3.6/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.10.3.6/images/sha256-e13b0ff33a36cd010dc4aeeeb6f3721b2f64c2db2f7269745f70b0bc2b859055)

## General Changes

### Apache Pulsar

\[improve]\[broker] Authorize originalPrincipal when provided

\[cherry-pick]\[branch-2.10] KCA: picking fixes from master

\[fix]\[broker] Fix potential exception cause the policy service init fail.

\[fix]\[client]\[branch-2.10]Return local thread for the `newThread`

\[cherry-pick]\[branch-2.10] Fix deadlock causes session notification not to work

\[fix] \[broker] Topic close failure leaves subscription in a permanent fence state

\[fix] \[client] fix memory leak if enabled pooled messages

f5c7de2d00 Revert "release 2.10.4 test"

d8763d4e70 release 2.10.4 test

\[improve] Simplify enabling Broker, WS Proxy hostname verification

\[branch-2.10]\[broker] Support zookeeper read-only config.

### AoP

improve dependencies

### KoP

If all partition messages are sent, cancel the delayed tasks to avoid full gc or oom

\[docs] Update some outdated documents

Upgrade nexus staging maven plugin

### Cloud Storage Connector

\[feat]\[json] Allow NaN numbers when flag is set

\[fix]\[config] Fix NPE if partitionerType is empty

\[chore] upgrade transitive dependencies with severe vulnerabilities

### StreamNative Pulsar Plugins

\[improve]\[detector] Fix crash when pulsar is unavailable

### Function Mesh Worker Service

4cd4005 Replace PulsarResources with null for compatible

add helm chart for standalone mode

Pass ProcessingGuarantee to FunctionMesh

Bump function-mesh to v0.11.1


# StreamNative Weekly Release Notes v2.10.3.7
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/component-changelogs-v2.10.3.7



# StreamNative Weekly Release Notes v2.10.3.7

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.10.3.7](https://github.com/streamnative/pulsar/releases/tag/v2.10.3.7)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.10.3.7/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.10.3.7/images/sha256-517b986cc84b715cf320342ab13cc0830eb53e51b0f18d16633e94fc1d94161d)

## General Changes

### Apache Pulsar

\[fix] \[cli] Fix Broker crashed by too much memory usage of pulsar tools

\[refactor]\[broker] Use AuthenticationParameters for rest producer

\[refactor]\[fn] Use AuthorizationServer more in Function Worker API

\[improve]\[txn] Cleanup how superusers abort txns

\[fix]\[broker] Only validate superuser access if authz enabled

\[fix]\[broker] Ignore and remove the replicator cursor when the remote cluster is absent

\[branch-2.10] \[fix] \[auth] fix not forward compatible config saslJaasServerRoleTokenSignerSecretPath after cherry-pick #15121

\[Build] Make the test JVM exit if OOME occurs

\[branch-2.10]\[fix]\[broker] Fix index generator is not rollback after entries are failed added

\[fix]\[ci]\[branch-2.10] Fix the release tools

\[Authenticate] fix Invalid signature error when use Kerberos Authentication

\[fix]\[sec] Fix transitive critical CVEs in file-system tiered storage

\[fix] \[admin] fix incorrect state replication.connected on API partitioned-topic stat

\[fix] \[proxy] Used in proxyConf file when configuration is missing in the command line

\[fix] \[broker] Counter of pending send messages in Replicator incorrect if schema future not complete

\[fix] \[admin] Make response code to 400 instead of 500 when delete topic fails due to enabled geo-replication

\[improve]\[admin]\[branch-2.10] Unset namespace policy to improve deleting namespace

45f303c65d \[test] Fix ServerCnxTest failing after merge of #19830

### KoP

\[improve] Add message.max.bytes for describe broker config

Fix flaky KopEventManagerTest.testOneTopicGroupState

\[tests] Unflaky MessagePublishBufferThrottleTestBase

\[improvement] Do not cache topics in KafkaTopicManagerSharedState

Improve OAuth documentation

### Cloud Storage Connector

baa409c Upgrade depends sn pulsar version to 2.10.3.6

3c63ccb Fix unit test compatibility with getReaderSchema() method

cd6df22 Compatible with generics.

Fix JsonFormat convert failed when use array jsonBytes or jsonString.

\[improve]\[logging] Log uploads per partition instead of entire batch

### StreamNative Pulsar Plugins

Fix the backup failed because of the upload failed

### Cloud Pulsar Plugins

Handle null AuthenticationDataSource when checking super user roles

Fix concurrent modification exception

### Function Mesh Worker Service

adopt new api branch 2.10.3

Set functions.useDedicatedRunner to false explicitly

support to disable runtime

verify 0.12.0


# StreamNative Weekly Release Notes v2.10.4.1
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/component-changelogs-v2.10.4.1



# StreamNative Weekly Release Notes v2.10.4.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.10.4.1](https://github.com/streamnative/pulsar/releases/tag/v2.10.4.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.10.4.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.10.4.1/images/sha256-185fc44a46f4fe842424f174f4ced9b1bf0f34d036dafb4f47d0b42f892d6f0a)

## General Changes


# StreamNative Weekly Release Notes v2.10.4.2
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/component-changelogs-v2.10.4.2



# StreamNative Weekly Release Notes v2.10.4.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.10.4.2](https://github.com/streamnative/pulsar/releases/tag/v2.10.4.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.10.4.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.10.4.2/images/sha256-ac2fb2084f43776fcd3366a280421489cec2fef2ad3d473607c8b98e1729320a)

## General Changes

### Apache Pulsar

\[fix] \[broker] \[branch-2.10] Upgrade rocksDB version to 6.16.4 to keep sync with BookKeeper 4.14.7

\[improve] \[broker] Skip split boundle if only one broker

([#20190)](https://github.com/apache/pulsar/pull/20190))) Revert "\[improve] \[broker] Skip split boundle if only one broker

\[fix]\[monitor] topic with double quote breaks the prometheus format

Revert "\[fix] \[broker] \[branch-2.10] Upgrade rocksDB version to 6.29.4.1 to keep in sync with BookKeeper's RocksDB version

\[fix]\[broker] Fix `RoaringBitmap.contains` can't check value 65535

\[fix]\[broker] Fix the reason label of authentication metrics

\[improve] \[broker] Skip split boundle if only one broker

\[fix]\[txn] Fix transaction is not aborted when send or ACK failed

\[fix] \[broker] Fix infinite ack of Replicator after topic is closed

\[fix]\[monitor] Fix the partitioned publisher topic stat aggregation bug

\[fix] \[broker] Upgrade rocksDB version to 6.29.4.1 to keep in sync with BookKeeper's RocksDB version

bd7c9ff9fd \[cleanup]\[test] fix incorrect license-header of java file

\[fix] \[broker] Producer created by replicator is not displayed in topic stats

793de9aa9d \[cleanup]\[test] fix incorrect license-header of java file

\[fix]\[client] Release the orphan producers after the primary consumer is closed

\[fix] \[broker] delete topic failed if disabled system topic

([#19875)](https://github.com/apache/pulsar/pull/19875))) Revert "\[fix]\[broker] Fix NPE when update topic policy.

\[fix] \[ml] make the result of delete cursor is success if cursor is deleted

\[fix] \[broker] Fast fix infinite HTTP call getSubscriptions caused by wrong topicName

\[fix]\[broker] Fix issue where msgRateExpired may not refresh forever

\[fix]\[broker] Fix can't send ErrorCommand when message is null value

\[fix]\[broker] Fix NPE when update topic policy.

\[fix]\[client] Fix DeadLetterProducer creation callback blocking client io thread.

\[fix]\[broker] Fix Return value of getPartitionedStats doesn't contain subscription type

### KoP

\[fix]\[branch-2.10.4] Unify fetch offset topic name

Increase timeout for KafkaListenerNameTest

### Function Mesh Worker Service

branch2.10 fix list namespaced custom object call


# StreamNative Weekly Release Notes v2.10.4.3
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/component-changelogs-v2.10.4.3



# StreamNative Weekly Release Notes v2.10.4.3

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.10.4.3](https://github.com/streamnative/pulsar/releases/tag/v2.10.4.3)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.10.4.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.10.4.3/images/sha256-0a41f7d24622521843b1a56601944c33a649c07501eeae783e6054808d97dedd)

## General Changes

### Apache Pulsar

\[improve]\[monitor] Add JVM start time metric

\[fix]\[ml] Fix ledger left in OPEN state when enable `inactiveLedgerRollOverTimeMs`

\[fix]\[broker] Fix default bundle size used while setting bookie affinity

\[fix]\[broker] Fix the behavior of delayed message in Key\_Shared mode

\[improve]\[broker] Get lowest PositionImpl from NavigableSet

\[fix] \[broker] error TimeUnit to record publish latency

\[fix] \[broker] In Key\_Shared mode: remove unnecessary mechanisms of message skip to avoid unnecessary consumption stuck

\[fix]\[broker]Fix deadlock of metadata store

\[fix]\[build] update the zookeeper version to 3.6.4

### KoP

Fix list offsets for times failure when ledgers are removed by a rollover operation

Fix flaky-test: KafkaNonPartitionedTopicTest.testNonPartitionedTopic

Use brokerClientTlsEnabled to configure pulsar client

Fix pulsar entry formatter encode zero timestamp record caused exception

Upgrade org.json and kaml to non vulnerable versions

### StreamNative Pulsar Plugins

delete useless maven repo

Bump Pulsar to `2.10.4.3`

Support publish json content type message

Unify the dependencies version between Pulsar and plugins

### Function Mesh Worker Service

fix yq

fix e2e pulsar install

Reuse existing configs

Show windowConfig when get function

Cleanup functions/sinks/sources after delete


# StreamNative Weekly Release Notes v2.10.4.4
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/component-changelogs-v2.10.4.4



# StreamNative Weekly Release Notes v2.10.4.4

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.10.4.4](https://github.com/streamnative/pulsar/releases/tag/v2.10.4.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.10.4.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.10.4.4/images/sha256-28d5f2280ac696a90ddedb599f5c1c2f7ecb403f1f0b01ca4e60f2b808bc6a53)

## General Changes

### Apache Pulsar

Optimize conusmer pause

55609eb0db resolve conflict after updating

\[fix]\[broker] Fix the publish latency spike from the contention of MessageDeduplication

\[fix]\[client]Fix deadlock issue of consumer while using multiple IO threads

\[improve]\[test]\[branch-2.10] Backport disabling disk usage threshold for Elastic Testcontainers

\[improve]\[broker]\[branch-2.10] Backport Linux metrics changes from master branch

\[cleanup]\[broker] Validate authz earlier in delete subscription logic

\[fix]\[broker] release orphan replicator after topic closed

\[fix]\[broker] REST Client Producer fails with TLS only

\[fix]\[broker] Restore solution for certain topic unloading race conditions

\[fix]\[ml] There are two same-named managed ledgers in the one broker

\[fix]\[fn] Configure pulsar admin for TLS

\[fix]\[fn] Go functions must retrieve consumers by non-particioned topic ID

6506d6a0ac Backport test for #20326 to Java 8

\[fix]\[broker] Fix skip message API when hole messages exists

\[fix]\[io] Close the kafka source connector if there is uncaught exception

33a45e0b4c \[fix]\[build] Don't publish docker image with "latest" tag to docker repository

\[fix]\[sec] Upgrade Guava to 32.0.0 to address CVE-2023-2976

071af38592 Bump version to 2.10.5-SNAPSHOT

\[fix]\[build]\[branch-2.10] Fix ci-license check

03c7add59f Fix license header with missing \*

\[fix]\[fn] Go functions need to use static grpcPort in k8s runtime

\[fix]\[client] Cache empty schema version in ProducerImpl schemaCache.

\[fix]\[fn]Reset idle timer correctly

\[improve]\[misc] Upgrade Netty to 4.1.93.Final

\[improve]\[misc] Upgrade Netty to 4.1.89.Final

\[fix]\[broker] If ledger lost, cursor mark delete position can not forward

\[fix]\[sec] Upgrade sqlite-jdbc to resolve CVE-2023-32697

\[fix]\[ci] Update nar maven plugin version to fix excessive downloads

\[fix]\[broker] partitioned \_\_change\_events topic is policy topic

\[fix]\[fn] Make pulsar-admin support update py/go with package url

### KoP

\[bugfix]\[transactions] Release memory in TransactionMarkerChannelHandler

\[fix] Fix read unstable messages

### pulsarctl

Fixed remove auth plugin suffix

Removed error char

### StreamNative Pulsar Plugins

Fix NPE when token used (Vault authentication) doesn't exists

Fix charts repo

### Cloud Pulsar Plugins

Bump Pulsar to `2.10.4.3 attentive`

fixed check styles and metric method name

Added revocation check in AuthenticationProviderApiKeys.

Added RevocationList

fixed license headers for pulsar-broker-auth-apikeys project

Added RevocationClient

Renamed Authentication/AuthorizationProviderOAuth to Authentication/AuthorizationProviderApiKeys

a copy of pulsar-broker-oauth2 for the new plugin work for api-key project

Release pulsar broker api keys

### Function Mesh Worker Service

f2295dd4 Fix ci

Create VolumeMounts based on PVC

Support http protocol

Make log config works independent on CustomRuntimeOptions

Add support for node affinities and VolumeClaimTemplates

Support set log config name and key

Support liveness probe

fix auth e2e

set value schema when pulsar-client produce

Freeznet/use local registry for e2e

fix Not enough non-faulty bookies available

fix jdk build

Implement trigger

Fix restartFunctions and enhance ci

Add imagePullSecrets to CustomRuntimeOptions

release function-mesh 0.14.0

Use k8s namespace to fetch resources

### Aws EventBridge Connector

Fix incorrect doc file name.


# StreamNative Weekly Release Notes v2.10.4.5
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/component-changelogs-v2.10.4.5



# StreamNative Weekly Release Notes v2.10.4.5

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.10.4.5](https://github.com/streamnative/pulsar/releases/tag/v2.10.4.5)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.10.4.5/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.10.4.5/images/sha256-11638e9a688986f4fb0587bcca054c425d27be62ba6e9b8ec8ad12d85d8d2b4e)

## General Changes

### Apache Pulsar

\[fix]\[branch-2.10]Fix compilation issue introduced byfix Repeated messages of shared dispatcher

Issue 16802: fix Repeated messages of shared dispatcher

\[fix]\[branch-2.10]Fix compilation issue introduced by Save createIfMissing in TopicLoadingContext

\[fix] \[Perf] PerformanceProducer do not produce expected number of messages.

\[improve]\[broker] Save createIfMissing in TopicLoadingContext

\[fix]\[broker] Invalidate metadata children cache after key deleted

\[improve] \[broker] Avoid `PersistentSubscription.expireMessages` logic check backlog twice.

\[fix]\[meta] Adding the missed bookie id in the registration manager.

\[fix]\[sec] Upgrade snappy-java to address multiple CVEs

\[fix]\[broker]fix the publish latency spike issue with large number of producers

\[fix]\[branch-2.10] Fix duplicated deleting topics

\[fix]\[io] Close the kafka source connector got stuck

\[fix]\[fn] Exit JVM when main thread throws exception

### AoP

\[ci] Ignore the jms1\_1 test first

### KoP

\[branch-2.10.4] Upgrade kafka client version to 2.1.1


# StreamNative Weekly Release Notes v2.10.4.6
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/component-changelogs-v2.10.4.6



# StreamNative Weekly Release Notes v2.10.4.6

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.10.4.6](https://github.com/streamnative/pulsar/releases/tag/v2.10.4.6)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.10.4.6/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.10.4.6/images/sha256-8bd8edb73618a019d46bafab36d48db24cf922d23d82273a4feaa9dc0e6eecbf)

## General Changes

### Apache Pulsar

5c556334a0 Fix breaking change of the deprecated constructor of PersistentMessageExpiryMonitor

\[fix]\[broker]\[branch-2.10] Fix NPE when reset Replicator's cursor by position.

([#20597)](https://github.com/apache/pulsar/pull/20597))) Revert "\[fix]\[broker] Fix NPE when reset Replicator's cursor by position.

f8729c0053 \[fix]\[broker] Fix test and checkstyle

c7925b31ec Delete MockManagedCursor.java

a4b3ae5711 Update ConnectionHandlerTest.java

0e70bddbdb Update FaultInjectableZKRegistrationManager.java

7670e01d14 Update MetadataStoreTest.java - The prior PR was not checked against codestyle

\[fix] \[client] Messages lost when consumer reconnect

\[fix]\[client] Make the whole grabCnx() progress atomic

\[fix]\[meta] Bookie Info lost by notification race condition.

9a6a460563 \[fix]\[test] Fix the test introduced in #18804

\[fix]\[broker] Fix namespace deletion if \_\_change\_events topic has not been created yet

\[fix]\[schema] Only handle exception when there has

\[fix]\[broker] Topic policy can not be work well if replay policy message has any exception.

7b6d1c9116 \[fix]\[test] Fix the compilation issue introduced in #20597

\[fix]\[broker] Fix NPE when reset Replicator's cursor by position.

e6d4f09c44 \[fix]\[test] Fix test `testThreadSwitchOfZkMetadataStore`

\[fix]\[test] Replace test call to Auth0 with call to WireMock

\[fix]\[broker] Fix return the earliest position when query position by timestamp.

\[fix]\[offload] Filesystem offloader class not found hadoop-hdfs-client

\[improve]\[admin] Return BAD\_REQUEST on cluster data is null for createCluster

\[fix] \[meta]Switch to the metadata store thread after zk operation

\[fix]\[test] Fix flaky testCreateTopicWithZombieReplicatorCursor

\[fix]\[client] Fix race condition that leads to caching failed CompletableFutures in ConnectionPool

\[fix]\[fn] Make KubernetesRuntime translate characters in function tenant, namespace, and name during function removal to avoid label errors

\[fix]\[broker] Return if AbstractDispatcherSingleActiveConsumer closed

\[fix]\[fn] Fix JavaInstanceStarter inferring type class name error

\[fix] \[txn] fix consumer can receive aborted txn message when readType is replay

\[broker] clean inactive bundle from bundleData in loadData and bundlesCache

\[fix]\[ws] Remove unnecessary ping/pong implementation

\[fix]\[flaky-test]NamespaceServiceTest.flaky/testModularLoadManagerRemoveBundleAndLoad

fix: bundle-data metadata leak because of bundlestats was not clean

### KoP

\[branch-2.10.4]\[improve] Pass group ID to authorizer when using OAuth

\[improvement] Remove expensive useless String.format() in canConsumeAsync

### Cloud Pulsar Plugins

\[improve] Add cache for parse claims Jwt

Fix authenticate http request

\[improve] Move authz parse jwt body logic to authenticaiton state init stage

### Aws EventBridge Connector

c3367b7 Fix v2.10.4.x will fix sink name.

Improve prerequisites docs.


# StreamNative Weekly Release Notes v2.10.4.7
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/component-changelogs-v2.10.4.7



## StreamNative Weekly Release Notes v2.10.4.7

#### General Changes

### AoP

remove useless test code

### MoP

Change log level to debug

Add Cache for event writer.

b5b63e6 Fix connection timeout ms


# StreamNative Weekly Release Notes v2.10.4.8
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/component-changelogs-v2.10.4.8



## StreamNative Weekly Release Notes v2.10.4.8

#### General Changes

### MoP

([#1047)](https://github.com/streamnative/mop/pull/1047))) Revert "Add ping request for adapter channel

Fix close reader NPE.

Add ping request for adapter channel

Fix mock object

782c303 Improve test.

Fix publish latency unit

Improve log to avoid too much error

Fix bundle is being unload IllegalState exception

Fix NPE cased by inflating message

Fix lookup issue for MQTT-5

Fix `IllegalReferenceCountException` exception to break the callback

a752a2b Fix JDK version in the workflow

Bump project & pulsar version to 2.10.4.6

Fix dispatch docker build

Support fast build docker image

### Function Mesh Worker Service

remove java17 grammar for backward comp

add service account annotation if is created via sn cloud service account


# StreamNative Weekly Release Notes v2.10.5.1
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/component-changelogs-v2.10.5.1



## StreamNative Weekly Release Notes v2.10.5.1

#### General Changes

### MoP

365dd13 remove repository id

637fdd6 remove ossrh

### SN KoP

\[build] Fix oauth client release

ca93e164 \[branch-2.10] Fix CI workflow

Fix wrong offset increment for Sarama message set

\[improve]\[oauthclient] Support decode base64 format credentials URL

\[CI] Upload surefire artifacts when tests failed

\[branch-2.10] Fix broken branch-2.10 due to wiremock

\[oauthclient] Create a zero-dependencies jar: - remove Async HTTP client and use the standard JDK Http client - shade and relocate Jackson Databind, used for JSON

Update README.md

\[security] Communications between broker inherit BrokerClient configuration

\[improvement] Save resources on the BK threads by not accessing the metrics context

\[improvement] Do not use a static LOOKUP\_CACHE

\[bugfix] AppendRecordsContext cannot be Recyclable

\[transactions] log when a TX Coordinator is still loading

\[branch-2.10.4]\[ci] Speed up CI test and fix flaky test

([#1897](#)) Optimize getHeadersFromMetadata, replace Java streams with for loop

\[fix] not response for PRODUCE when acks=0

\[debug] add better log for CONCURRENT\_TRANSACTIONS error

\[perf]\[improvement] Improvements for PulsarEntryFormatter

\[fix]\[transaction] TransactionMarkerRequestCompletionHandler retries on UNKNOWN\_SERVER\_ERROR

\[bugfix] Fix decode pulsar format batch records timestamp

\[improve] Get size from ByteBuf earlier to prevent unnecessary retention

Prevent double-release on timeout

Document the tlsEnabled configuration for legacy KoP versions

\[improvement] hide scary InterruptedException in KopEventManager during broker shutdown

\[bugfix]\[transactions] Make TxnTransitMetadata.topicPartitions immutable

fix: remove topic.getManagedLedger().asyncDeleteCursor

\[bugfix]\[transactions] Prevent ConcurrentModificationException in getProducer()

### Cloud Pulsar Plugins

Support audience list

Do not catch parseClaimsJwt method exception

### Function Mesh Worker Service

Remove enableStateStore from CustomRuntimeOptions

allow by-pass the class loader from connector package

release function-mesh 0.15.0

Use state store to query/put state

Add missing config

Handle delete response when failed to delete k8s object

2bee464b Use pulsarctl runner image

Add missed permissions checks

Append version to the description field of ConnectorDefinition

Respect --update-auth-data parameter

prevent cleanup fail the action

expose k8s 404 and other rest errors with correct error code

bump k8s to 1.23.17

### Lakehouse Connector

3115a20 Bump from 2.10.4 to 2.10.5-SNAPSHOT

### Aws EventBridge Connector

Refactor docs struct and content.

Improve config docs.


# V2.10.5.10
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/v2.10.5.10



# StreamNative Weekly Release Notes v2.10.5.10

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.10.5.10](https://github.com/streamnative/pulsar/releases/tag/v2.10.5.10)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.10.5.10/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.10.5.10/images/sha256-e20e7bb436c022011b4be31355b4aa3d5e6c22077ce6fac0271f3f8a6bbcafff)

## General Changes

### Apache Pulsar

\[authentication] Update original auth data after auth data refresh

\[fix]\[ml] Fix unfinished callback when deleting managed ledger

\[fix]\[broker] Fix setReplicatedSubscriptionStatus incorrect behavior

\[fix]\[client] Fix print error log 'Auto getting partitions failed' when expend partition.

\[fix]\[broker] Avoid pass null role in MultiRolesTokenAuthorizationProvider

\[fix]\[broker]Fixed produce and consume when anonymousUserRole enabled

\[fix] \[broker] do not filter system topic while shedding.

\[fix]\[broker] Fix the deadlock when using BookieRackAffinityMapping with rackaware policy

([#20659)](https://github.com/apache/pulsar/pull/20659))) Revert "\[improve]\[broker]\[branch-2.10] Backport Linux metrics changes from master branch

\[fix] \[log] fix the vague response if topic not found

\[improve] \[broker] Let the producer request success at the first time if the previous one is inactive

([#21220)](https://github.com/apache/pulsar/pull/21220))) Revert "\[improve] \[broker] Let the producer request success at the first time if the previous one is inactive

\[improve] \[broker] Let the producer request success at the first time if the previous one is inactive

\[fix]\[broker] Correct schema deletion for parititioned topic

\[branch-2.10]\[fix]\[broker] Duplicate LedgerOffloader creation when na…

### AoP

Add rabbitmq amqp-client dependency for test

### KoP

\[branch-2.10.5] Fix service unit not ready caused UnknownServerException

\[branch-2.10.5] Fix list offset convert long to int caused overflow

### AMQP1\_0 Connector

d7a957a Fix docker file

### AWS SQS Connector

Refactor create a connector section docs.

Try fix auto labeling.

### StreamNative Pulsar Plugins

Fix the dependency conflict with pulsar broker

\[pulsarctl-plugin] Bump client-go to `0.20.15`

a5b91842 Fix cve(#1323)

### Function Mesh Worker Service

Fallback to reason field if the lastState's message is empty

make memory padding configurable

Fix getSinkList and getSourceList impl to avoid showing fields details.

### Google BigQuery Sink Connector

Refactor create a connector section docs.

Fix auto label bot not work.

### Aws EventBridge Connector

Fix typos in doc

Refactor create a connector section docs.

Fix auto label bot not work.

Fix some docs and deprecated eventBusResourceName config.


# V2.10.5.11
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/v2.10.5.11



# StreamNative Weekly Release Notes v2.10.5.11

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.10.5.11](https://github.com/streamnative/pulsar/releases/tag/v2.10.5.11)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.10.5.11/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.10.5.11/images/sha256-2617b39f5058be834a62d90c10840475667803fc722aee134a7f2d31315c03ee)

## General Changes

### Apache Pulsar

\[fix] \[broker] network package lost if enable haProxyProtocolEnabled

fix] \[ml] Fix orphan scheduled task for ledger create timeout check

\[fix] \[broker] Fix thousands orphan PersistentTopic caused OOM

2e86e0710c cve: exclude ch.qos.logback in canal.protocol \* resolve CVE-2023-6378

### AMQP1\_0 Connector

Refactor create a connector section docs.

### AWS SQS Connector

7044dce Fix format errors for note.

### StreamNative Pulsar Plugins

Fix the packages cloud storage failed to find gs schema

### Google BigQuery Sink Connector

0c4a5ed Fix format errors for note.


# V2.10.5.12
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/v2.10.5.12



# StreamNative Weekly Release Notes v2.10.5.12

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.10.5.12](https://github.com/streamnative/pulsar/releases/tag/v2.10.5.12)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.10.5.12/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.10.5.12/images/sha256-eca3fcb9782faf7cfb91c821a9997dc346af3b7548ca8ada0d0126fb935793af)

## General Changes

### Apache Pulsar

([#21057)](https://github.com/apache/pulsar/pull/21057))) Revert "\[fix]\[misc] Bump GRPC version to 1.55.3 to fix CVE

\[fix] \[ml] Fix retry mechanism of deleting ledgers to invalidate

\[fix] \[broker] Update topic policies as much as possible when some ex was thrown

\[fix]\[broker] Fix typo in the config key

\[improve]\[broker] Support not retaining null-key message during topic compaction

5e70810f98 Upgrade OWASP dependency check maven plugin version

\[fix]\[broker] Fix the issue of topics possibly being deleted.

7348cd1ace \[fix]\[sec] exclude logback from zookeeper(#14601)

\[fix]\[sec] Upgrade Netty to 4.1.100 to address CVE-2023-44487

\[improve]\[build] Upgrade Apache ZooKeeper to 3.9.1

\[fix]\[misc] Bump GRPC version to 1.55.3 to fix CVE

\[fix]\[sec] Bump avro version to 1.11.3 for CVE-2023-39410

\[fix]\[sec] Upgrade snappy-java to 1.1.10.5

### Cloud Storage Connector

Update nick-invision to nick-fields

### AMQP1\_0 Connector

a15d831 Fix typos in doc

### AWS SQS Connector

Update nick-invision to nick-fields

### AWS Lambda Connector

Update nick-invision to nick-fields

### StreamNative Pulsar Plugins

Fix the metadata tool CI

### Function Mesh Worker Service

Support load docsLink and iconLink for connector catalog.

update retry github action owner

### Google Pub / Sub Connector

Update nick-invision to nick-fields

### Google BigQuery Sink Connector

Update nick-invision to nick-fields

### Snowflake Connector

Update nick-invision to nick-fields

### Aws EventBridge Connector

Update nick-invision to nick-fields


# V2.10.5.13
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/v2.10.5.13



# StreamNative Weekly Release Notes v2.10.5.13

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.10.5.13](https://github.com/streamnative/pulsar/releases/tag/v2.10.5.13)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.10.5.13/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.10.5.13/images/sha256-db8bc19d517edbda68f0f15bc9ba2a06e78bb4d5d6b5ff058f4f57cfe79202d9)

## General Changes

### Apache Pulsar

05ac1f9dda \[fix]\[build] Fix compatibility issue introduced by #20750

### StreamNative Pulsar Plugins

([#1054)](https://github.com/streamnative/sn-pulsar-plugins/pull/1054))) Revert "\[detector] Separate E2E latency detector per broker

([#1094)](https://github.com/streamnative/sn-pulsar-plugins/pull/1094))) Revert "Extend receive timeout to avoid context timeout

([#1100)](https://github.com/streamnative/sn-pulsar-plugins/pull/1100))) Revert "Support pulsar detector dashboard

([#1292)](https://github.com/streamnative/sn-pulsar-plugins/pull/1292))) Revert "\[fix]\[detector] Cleanup inactive broker's e2e detector

\[fix]\[cve] Exclude logback from zookeeper

\[fix]\[detector] Cleanup inactive broker's e2e detector

\[branch-2.10]\[pulsar-detector] Upgrade go dependencies to fix CVEs

Upgrade go version

\[Snyk] Security upgrade golang from 1.15.6 to 1.18.6

Support pulsar detector dashboard

Extend receive timeout to avoid context timeout

\[detector] Separate E2E latency detector per broker


# V2.10.5.14
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/v2.10.5.14



## StreamNative Weekly Release Notes v2.10.5.14

#### General Changes

### Apache Pulsar

\[fix]\[sec] Upgrade commons-compress to 1.26.0

\[fix]\[broker] Support running docker container with gid != 0

\[branch-2.10]\[improve]\[broker] Do not retain the data in the system topic

\[fix]\[broker]\[branch-3.1] Avoid PublishRateLimiter use an already closed RateLimiter

\[fix]\[broker] Sanitize values before logging in apply-config-from-env.py script

d8a6d9898e \[fix]\[build] Delete unused imported introduced by #21947

\[fix] \[broker] Replication stopped due to unload topic failed

\[fix]\[client] Fix ConsumerBuilderImpl#subscribe silent stuck when using pulsar-client:3.0.x with jackson-annotations prior to 2.12.0

\[fix]\[broker] Fix memory leak during topic compaction

### KoP

\[transactions] Implement KIP-664 DescribeProducers

### Cloud Storage Connector

Update base image

Update base image

### AMQP1\_0 Connector

Update base image

### AWS SQS Connector

Update base image

### AWS Lambda Connector

update-base-image

### pulsarctl

\[branch-2.10.5]\[cve] Update golang.org/x/net

### StreamNative Pulsar Plugins

e7d800d4 \[fix]\[cve] Exclude org.apache.avro from org.apache.hadoop

### Function Mesh Worker Service

reduce integration test image size with slim base image

bump function-mesh to 0.19.0

clean up the disk

Ignore exception when connector customize catalogs is empty.

### Google BigQuery Sink Connector

Update docker base

### Snowflake Connector

Update base image


# V2.10.5.15
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/v2.10.5.15



## StreamNative Weekly Release Notes v2.10.5.15

#### General Changes

### Apache Pulsar

5812b306e6 Bump version to 2.10.0-SNAPSHOT

a76ddbe5af \[fix]\[build] Delete unused import

\[improve]\[broker] Consistently add fine-grain authorization to REST API

\[improve] \[broker] Do not print an Error log when responding to `HTTP-404` when calling `Admin API` and the topic does not exist.

\[improve]\[admin]internalGetMessageById shouldn't be allowed on partitioned topic

\[improve]\[broker] Avoid print redirect exception log when get list from bundle

\[fix]\[sec] Upgrade Jetty to 9.4.54.v20240208 to address CVE-2024-22201

\[improve]\[fn] Add configuration for connector & functions package url sources

7256bb7167 Adjust license header format

4b5dba7794 Fix #22163 cherry-picking problem

\[improve]\[broker] Add fine-grain authorization to retention admin API

\[fix]\[sec] Upgrade Jetty to 9.4.53 to address CVE-2023-44487

1b72d46206 Revert changes to functions\_worker.conf used in system tests

5624616dbe Fix warning "calling yaml.load() without Loader=... is deprecated"

\[improve]\[fn] Optimize Function Worker startup by lazy loading and direct zip/bytecode access

\[fix]\[sec] Upgrade commons-compress to 1.26.0

\[fix]\[broker] Support running docker container with gid != 0

### KoP

Update LICENSE

### AWS Lambda Connector

Enable unit tests for weekly release

### StreamNative Pulsar Plugins

Use an old version of the sn/charts

### Function Mesh Worker Service

21f4587f Fix ci

949a8a60 Deprecate classloader

### Google Pub / Sub Connector

ff32f8f Add puul\_request trigger condition


# V2.10.5.3
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/v2.10.5.3



# StreamNative Weekly Release Notes v2.10.5.3

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.10.5.3](https://github.com/streamnative/pulsar/releases/tag/v2.10.5.3)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.10.5.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.10.5.3/images/sha256-189089f1305c1bba0a06b93822df6a455d731850a065689fd7e817c06a690cdd)

## General Changes

### Apache Pulsar

\[branch-2.10] \[fix] \[broker] Fix isolated group not work problem.

\[branch-2.10] Fix flaky test

fd86adad02 set project version

\[fix]\[client] Avoid ack hole for chunk message

\[fix]\[client] Fix consumer can't consume resent chunked messages

\[fix]\[broker]Fix chunked messages will be filtered by duplicating

\[improve] \[broker] Improve cache handling for partitioned topic metadata when doing lookup

\[improve] Introduce the sync() API to ensure consistency on reads during critical metadata operation paths

### KoP

fbe39ff5 Fix GssapiAuthenticationTest test

Optimize authorization by caching authorization results

### AMQP1\_0 Connector

4e5887b change test images

### Function Mesh Worker Service

Fix possible NPE errors

Set retain\[Key]Ordering to false if it is null

Load connector definition from ConnectorCataLog CRD.

Support json format logs and yaml format log config file

Change integration test ci trigger mode to pull\_request.

Support using sidecar to send logs to pulsar

Use AuthConfig.GenericAuth field to replace auth secret


# V2.10.5.4
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/v2.10.5.4



# StreamNative Weekly Release Notes v2.10.5.4

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.10.5.4](https://github.com/streamnative/pulsar/releases/tag/v2.10.5.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.10.5.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.10.5.4/images/sha256-ba892bd65c697801714222a77d860d8c83608068312b3e108979d1c686243eac)

## General Changes

### Apache Pulsar

\[fix] \[broker] Make specified producer could override the previous one

1d2260de9b \[fix]\[test] Fix test caused by #21144

\[imporve] \[bookie] Upgrade BookKeeper dependency to 4.14.8 for branch 2.10

\[fix]\[ci] Fix license check issue

\[improve] \[broker] improve read entry error log for troubleshooting

\[fix] \[client] fix same producer/consumer use more than one connection per broker

\[fix]\[client] Fix repeat consume when using n-ack and batched messages

### AMQP1\_0 Connector

Remove wrong COPY command

\[CI] Adjust CI to test the corresponding Pulsar image version

Fix integration test.

### AWS SQS Connector

Improve sqs source doc.

Improve sqs sink docs.

### Function Mesh Worker Service

Change connectorSearchIntervalSeconds default value to 600s.


# V2.10.5.5
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/v2.10.5.5



# StreamNative Weekly Release Notes v2.10.5.5

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.10.5.5](https://github.com/streamnative/pulsar/releases/tag/v2.10.5.5)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.10.5.5/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.10.5.5/images/sha256-82bef1f4dc1c607f15dae02419dae4b26483135a5055a2edbee21031b414d636)

## General Changes

### Apache Pulsar

\[fix]\[ml] Fix thread safe issue with RangeCache.put and RangeCache.clear

\[fix]\[broker]\[branch-2.10] Fix inconsistent topic policy

\[fix]\[broker] Fixed reset for AggregatedNamespaceStats

\[improve] \[client] Merge lookup requests for the same topic

\[improve] \[proxy] Not close the socket if lookup failed caused by too many requests

\[fix]\[broker] Fix write duplicate entries into the compacted ledger after RawReader reconnects

\[fix]\[broker]Backport fix UniformLoadShedder selecet wrong overloadbroker and underloadbroker

### pulsarctl

Support status check for pulsarctl command

### Cloud Pulsar Plugins

Fix REST API interceptor check for creating partitioned topic with properties

### Function Mesh Worker Service

Read connector catalogs from namespaces.

### Google BigQuery Sink Connector

Improve sink docs.


# V2.10.5.6
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/v2.10.5.6



# StreamNative Weekly Release Notes v2.10.5.6

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.10.5.6](https://github.com/streamnative/pulsar/releases/tag/v2.10.5.6)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.10.5.6/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.10.5.6/images/sha256-64701b3bf1258a1820fb719566165f2adc43bba27082cfd9dee34f3448e99077)

## General Changes

### Apache Pulsar

\[fix] \[bk-client] Fix bk client MinNumRacksPerWriteQuorum and EnforceMinNumRacksPerWriteQuorum not work problem.

\[fix] \[ml] fix wrong msg backlog of non-durable cursor after trim ledgers

\[fix] \[ml] Reader can set read-pos to a deleted ledger

\[fix]\[test] Fix flaky test NarUnpackerTest

\[improve] \[broker] Not close the socket if lookup failed caused by bundle unloading or metadata ex

\[fix] \[client] fix reader.hasMessageAvailable return false when incoming queue is not empty

\[improve] \[broker] Print warn log if ssl handshake error & print ledger id when switch ledger

### AMQP1\_0 Connector

7466823 Cherr-picked from #721: Improve sink and source connector docs.

### Function Mesh Worker Service

Set usingInsecureAuth to false by default

Add configs: javaOpts/labels/logConfig to CustomRuntimeOptions

### Google BigQuery Sink Connector

b935880 Cherr-picked by #395 and #409: improve sources docs.


# V2.10.5.7
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/v2.10.5.7



# StreamNative Weekly Release Notes v2.10.5.7

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.10.5.7](https://github.com/streamnative/pulsar/releases/tag/v2.10.5.7)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.10.5.7/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.10.5.7/images/sha256-8209201bcdf0f28456b99d11b5d45ac0562142b7b0d184f3ec273aa85386249a)

## General Changes

### Apache Pulsar

\[fix]\[broker] Fix MultiRoles token provider NPE when using anonymous clients

\[fix]\[sec] Fix MultiRoles token provider when using anonymous clients

\[fix]\[broker]Check that the super user role is in the MultiRolesTokenAuthorizationProvider plugin

### KoP

Ignore the flaky MultiLedgerTest.testListOffsetForEmptyRolloverLedger

### Function Mesh Worker Service

a7c15441 Fix state store

Bump function-mesh to v0.18.0

6f014fb6 Bump function mesh to v0.17.0

disable golang runtime by default

allow submit very long name resources


# V2.10.5.8
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/v2.10.5.8



# StreamNative Weekly Release Notes v2.10.5.8

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.10.5.8](https://github.com/streamnative/pulsar/releases/tag/v2.10.5.8)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.10.5.8/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.10.5.8/images/sha256-f52894894c2861d97397976f2b5d879acc117ea5647220af6c667639ddd2515e)

## General Changes

### Apache Pulsar

\[fix]\[proxy] Move status endpoint out of auth coverage

\[fix] \[broker] Make the new exclusive consumer instead the inactive one faster


# V2.10.5.9
Source: https://docs.streamnative.io/release-notes/pulsar/v2.10/v2.10.5.9



# StreamNative Weekly Release Notes v2.10.5.9

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.10.5.9](https://github.com/streamnative/pulsar/releases/tag/v2.10.5.9)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.10.5.9/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.10.5.9/images/sha256-7d4f7705dc43c6b464bbd42e37a025dbe54100f5d39c8e756a257f24250e9657)

## General Changes

### Apache Pulsar

\[fix]\[broker] Do not write replicated snapshot marker when the topic which is not enable replication

\[fix]\[broker] Fix create topic with different auto creation strategies causes race condition

\[fix]\[broker] Fix namespace bundle stuck in unloading status

fix duplicate calculation for msgRateIn and msgThroughputIn in replication stats

\[fix]\[broker] namespace not found will cause request timeout

\[fix]\[txn] Ack all message ids when ack chunk messages with transaction.

### AWS Lambda Connector

Make AWS Lambda sink connector private

### Function Mesh Worker Service

55bc200f Use local registry for generic runner images

Support read customize connector catalogs.

Support generic runtime

Append version to the description field of ConnectorDefinition for load from connector catalog

### Lakehouse Connector

Update snappy dependency


# StreamNative Weekly Release Notes v2.11.0.1
Source: https://docs.streamnative.io/release-notes/pulsar/v2.11/component-changelogs-v2.11.0.1



# StreamNative Weekly Release Notes v2.11.0.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.11.0.1](https://github.com/streamnative/pulsar/releases/tag/v2.11.0.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.11.0.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.11.0.1/images/sha256-9a098ab605456c9d4967b75df0d3564be5f56e44da937c661f190d5b177244d7)

## General Changes


# StreamNative Weekly Release Notes v2.11.0.2
Source: https://docs.streamnative.io/release-notes/pulsar/v2.11/component-changelogs-v2.11.0.2



# StreamNative Weekly Release Notes v2.11.0.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.11.0.2](https://github.com/streamnative/pulsar/releases/tag/v2.11.0.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.11.0.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.11.0.2/images/sha256-a8cc0e2ef8903480c078e8b5cfaf34dce6b5f685b0100682c6e76fb6f76244dd)

## General Changes

### Apache Pulsar

([#19199)](https://github.com/apache/pulsar/pull/19199))) Fix race condition while updating partition number

\[fix]\[broker] Create replicated subscriptions for new partitions when needed

e49bf11f07 Fix compile issue.

\[improve]\[broker]Only create extended partitions when updating partition number

\[fix]\[broker]fix catching ConflictException when update topic partition

\[fix]\[broker]fail to update partition meta of topic due to ConflictException: subscription already exists for topic

\[fix]\[txn]fix receive duplicated messages due to pendingAcks in PendingAckHandle

\[feature]\[txn] Fix individual ack batch message with transaction abort redevlier duplicate messages

\[improve]\[broker] Use shrink map for trackerCache

\[fix] \[test] Wrong mock-fail of the test ManagedLedgerErrorsTest.recoverLongTimeAfterMultipleWriteErrors

\[fix]\[client] Fix authentication not update after changing the serviceUrl

\[fix]\[broker] Fix loadbalance score caculation problem

\[fix]\[broker] Fixed history load not releasing

\[improve]\[test] Add test `brokerReachThreshold` for ThresholdShedderTest

\[fix] \[client] fix memory leak if enabled pooled messages

\[fix]\[broker] Filter system topic when getting topic list by binary proto.

\[fix]\[cli] Fix Pulsar admin tool is ignoring tls-trust-cert path arg

\[fix]\[broker] Fix geo-replication admin

\[improve]\[broker]\[branch-2.11] Improve tls config on replication client and cluster cli

\[improve] Simplify enabling Broker, WS Proxy hostname verification

\[fix] \[broker] Make the service name resolver cache of PulsarWebResource expire after access

\[fix] \[broker] Incorrect service name selection logic

\[fix]\[ci] Fix broken CI

([#19302)](https://github.com/apache/pulsar/pull/19302))) Revert "\[improve] \[admin] Make the default value of param --get-subscription-backlog-size of admin API topics stats true

\[fix]\[test]\[branch-2.10] Remove testGetTopic method

\[improve] \[admin] Make the default value of param --get-subscription-backlog-size of admin API topics stats true

\[fix] \[ml] topic load fail by ledger lost

### SN KoP

Upgrade nexus staging maven plugin

### StreamNative Pulsar Plugins

\[improve]\[detector] Fix crash when pulsar is unavailable

\[rest] Fix Big message is truncated silently

Fix memory leak in that message is not pooled.

Add test to cover concurrent call

### Function Mesh Worker Service

Bump function-mesh to v0.11.1

### Lakehouse Connector

remove unit-test jdk8 from branch 2.11

Make unit tests can run with JDK8

Fix CVEs introduced by hadoop-common

\[fix]\[sec] Fix CVEs introduced by log4j 1.2.17

a034ed0 Adjust codeowner to ecosystem

\[Bugfix] Fix the problem when read as a delta table


# StreamNative Weekly Release Notes v2.11.0.3
Source: https://docs.streamnative.io/release-notes/pulsar/v2.11/component-changelogs-v2.11.0.3



# StreamNative Weekly Release Notes v2.11.0.3

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.11.0.3](https://github.com/streamnative/pulsar/releases/tag/v2.11.0.3)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.11.0.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.11.0.3/images/sha256-6f60b50fdc8db28a20399febfd10b4cebe37688acf68a4a4a6225609426a9d31)

## General Changes

### Apache Pulsar

\[improve]\[meta]Allow version to start positive and grow by more than one

\[fix]\[broker] Fix potential exception cause the policy service init fail.

\[fix] \[broker] Topic close failure leaves subscription in a permanent fence state

### Cloud Storage Connector

\[chore] Upgrade to JDK 17

\[feat]\[json] Allow NaN numbers when flag is set

\[fix]\[config] Fix NPE if partitionerType is empty

\[chore] upgrade transitive dependencies with severe vulnerabilities

\[chore] Rename io-cloud-storage-sink.md to cloud-storage-sink.md

\[fix]\[sink] Reset currentBatchSize & currentBatchBytes on failing records

\[fix]\[sink] Fix sink failing upon schema retrieval exceptions

\[docs] Add troubleshooting section to readme

d9f4c4e Adjust codeowner to ecosystem

\[fix]\[schema] Fix Bytes schema conversion to AutoConsume schema

\[misc] Add @streamnative/ecosystem to CODEOWNERS

Remove @streamnative/platform to avoid notification noise

\[#491 ]Fix flush stalling for large pendingQueueSize with small batchSize/maxBatchBytes

\[feat] Option to use message index instead of record sequence for partitioning

\[fix]\[ci] Change the release process

\[#20]\[docs] Update docs structure for pulsar-hub sync

\[#472]\[misc] Update PULL\_REQUEST\_TEMPLATE.md for auto-labeling

\[feat] Support for Azure Blob Storage

Bump jackson-databind from 2.13.2.1 to 2.13.4.1

\[misc] Bump checkstyle to 8.29

\[feat] Make compression algorithm configurable for parquet format

\[misc] Move log 'Skip flushing because the pending flush is empty' to debug level

\[docs] Add pathPrefix to docs

\[config] Make pendingQueueSize default to batchSize

\[feat] Support for KeyValue schema

\[#445]\[sink] Add trigger for flushing batch when maxBatchBytes is reached

\[fix]\[sink] Fix multiple topics in the same output blob

\[fix]\[config] Validate endpoint URI scheme


# StreamNative Weekly Release Notes v2.11.0.4
Source: https://docs.streamnative.io/release-notes/pulsar/v2.11/component-changelogs-v2.11.0.4



# StreamNative Weekly Release Notes v2.11.0.4

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.11.0.4](https://github.com/streamnative/pulsar/releases/tag/v2.11.0.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.11.0.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.11.0.4/images/sha256-8a86467f16f11cbca9efc51cc86a65efc22758bef466d3545d4342b400bcbe94)

## General Changes

### Apache Pulsar

\[improve]\[client] Exclude log4j-slf4j-impl from compile dep in pulsar-client-all

\[fix]\[admin] Delete tenant local policy only if exist

\[fix] \[admin] Make response code to 400 instead of 500 when delete topic fails due to enabled geo-replication

\[fix]\[io] KCA sink: handle null values with `KeyValue<Avro,Avro>` schema

\[fix]\[broker] Fix delete system topic clean topic policy

\[fix]\[meta] Follow up #19817, Fix race condition between ResourceLock update and invalidation

\[fix]\[client] moving get sequenceId into the sync code segment

\[improve] Upgrade bookkeeper to 4.15.4

13f4a0dc0a \[test] Fix ServerCnxTest failing after merge of #19830

ad06fac088 \[test] Fix MockMutableAuthenticationState implementation

\[improve]\[broker] Authorize originalPrincipal when provided

\[improve]\[broker] Follow up #19230 to tighten the validation scope

\[improve]\[txn]\[branch-2.11] Add getState in transaction for client API

\[fix] \[broker] Counter of pending send messages in Replicator incorrect if schema future not complete

\[fix]\[meta] Fixed race condition between ResourceLock update and invalidation

\[fix]\[client] Fix async completion in ConsumerImpl#processPossibleToDLQ

\[fix]\[meta] Fix close borrowed executor

\[fix]\[broker] Copy subscription properties during updating the topic partition number.

3a03292411 Fix license header issue.

\[fix]\[authentication] Store the original authentication data

\[fix]\[broker] Fix index generator is not rollback after entries are failed added.

\[fix] \[broker] delete topic failed if disabled system topic

KCA: picking fixes from master

\[fix]\[broker] Fix issue where msgRateExpired may not refresh forever

\[fix]\[meta] Fix deadlock causes session notification not to work

### KoP

Fix flaky KopEventManagerTest.testOneTopicGroupState

### Cloud Storage Connector

\[improve]\[logging] Log uploads per partition instead of entire batch

### Cloud Pulsar Plugins

Fix concurrent modification exception

### Function Mesh Worker Service

fix branch 2.11

e9504ae Use null for configMetadataStore for compatiblity

verify 0.12.0

support to disable runtime

add helm chart for standalone mode

Pass ProcessingGuarantee to FunctionMesh


# StreamNative Weekly Release Notes v2.11.0.5
Source: https://docs.streamnative.io/release-notes/pulsar/v2.11/component-changelogs-v2.11.0.5



# StreamNative Weekly Release Notes v2.11.0.5

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.11.0.5](https://github.com/streamnative/pulsar/releases/tag/v2.11.0.5)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.11.0.5/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.11.0.5/images/sha256-a7deff6eafc0e8cd31e3775f348e389d1368690b9ed572e77224e7a520a52392)

## General Changes

### Apache Pulsar

eeab999755 Fix license header issue.

\[fix]\[client] Fix DeadLetterProducer creation callback blocking client io thread.

\[fix] \[ml] make the result of delete cursor is success if cursor is deleted

\[fix]\[build] Client modules should be built with Java 8

\[fix]\[broker] Fix NPE when update topic policy.

\[fix]\[sec] Fix transitive critical CVEs in file-system tiered storage

\[improve]\[io] KCA: flag to force optional primitive schemas

\[improve]\[io] KCA: option to collapse partitioned topics

\[fix] \[admin] fix incorrect state replication.connected on API partitioned-topic stat

### KoP

Support passing the token directly as the password for Schema Registry

\[docs] Update some outdated documents

\[improve] Add message.max.bytes for describe broker config

Improve OAuth documentation

\[tests] Unflaky MessagePublishBufferThrottleTestBase

\[ksqldb] Add implementation for describeCluster

\[improvement] Do not cache topics in KafkaTopicManagerSharedState

If all partition messages are sent, cancel the delayed tasks to avoid full gc or oom

### Cloud Storage Connector

abdccf8 Upgrade depends sn pulsar version to 2.11.0.4.

fc0fab1 Fix unit test compatibility with getReaderSchema() method

Fix JsonFormat convert failed when use array jsonBytes or jsonString.

### Cloud Pulsar Plugins

Handle null AuthenticationDataSource when checking super user roles

### Function Mesh Worker Service

Set functions.useDedicatedRunner to false explicitly


# StreamNative Weekly Release Notes v2.11.1.0
Source: https://docs.streamnative.io/release-notes/pulsar/v2.11/component-changelogs-v2.11.1.0



# StreamNative Weekly Release Notes v2.11.1.0

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.11.1.0](https://github.com/streamnative/pulsar/releases/tag/v2.11.1.0)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.11.1.0/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.11.1.0/images/sha256-80a51e9a284adf57df865764407e69a02e9121e1c5806e3e2b36e792addd937b)

## General Changes

### Apache Pulsar

\[fix] \[ml] Fix uncompleted future when remove cursor

\[refactor]\[broker] Use AuthenticationParameters for rest producer

\[fix] \[cli] Fix Broker crashed by too much memory usage of pulsar tools

\[fix]\[proxy] Fix connection read timeout handling in Pulsar Proxy

\[improve]\[proxy] Implement graceful shutdown for Pulsar Proxy

\[fix]\[build] Suppress Guava CVE-2020-8908 in OWASP dependency check

\[fix]\[build] Client modules should be built with Java 8

\[fix]\[broker] Only validate superuser access if authz enabled

\[improve]\[txn] Cleanup how superusers abort txns

\[refactor]\[fn] Use AuthorizationServer more in Function Worker API

\[fix] \[admin] fix incorrect state replication.connected on API partitioned-topic stat

\[fix]\[broker] Return if AbstractDispatcherSingleActiveConsumer closed

\[fix]\[client] Fix DeadLetterProducer creation callback blocking client io thread.

\[fix] \[admin] Make response code to 400 instead of 500 when delete topic fails due to enabled geo-replication

\[fix]\[broker] Fix NPE when update topic policy.

\[fix]\[meta] Follow up #19817, Fix race condition between ResourceLock update and invalidation

\[fix]\[meta] Follow up #19817, Fix race condition between ResourceLock update and invalidation

\[improve]\[broker] Follow up #19230 to tighten the validation scope (#19234)

\[improve]\[broker] Authorize originalPrincipal when provided

\[fix] \[ml] make the result of delete cursor is success if cursor is deleted

\[fix]\[meta] Fixed race condition between ResourceLock update and invalidation

\[improve] Upgrade bookkeeper to 4.15.4

\[fix]\[broker] Copy subscription properties during updating the topic partition number.

\[fix]\[meta] Fix close borrowed executor

\[fix]\[broker] Fix issue where msgRateExpired may not refresh forever

\[fix]\[meta] Fix deadlock causes session notification not to work

\[fix]\[broker] Fix potential exception cause the policy service init fail.

\[fix] \[broker] delete topic failed if disabled system topic

\[fix]\[client] Fix topic list watcher fail log

\[fix]\[broker] Fix index generator is not rollback after entries are failed added.

\[improve]\[cli]Upgrade python to python3

\[fix]\[cli] Fix Pulsar admin tool is ignoring tls-trust-cert path arg

\[fix] \[broker] Topic close failure leaves subscription in a permanent fence state

\[fix]\[broker]\[branch-2.11] Fix geo-replication admin

\[improve] Simplify enabling Broker, WS Proxy hostname verification

\[fix]\[broker] Filter system topic when getting topic list by binary proto.

\[feat]\[txn] Support for idempotent commit and abort, Solution1.

\[fix]\[ci]\[branch-2.11] Fix broken CI

\[fix]\[client] Broker address resolution wrong if connect through a multi-dns names proxy

\[fix]\[test] ProxyWithAuthorizationTest remove SAN from test certs

\[improve]\[broker]\[branch-2.11] Improve tls config on replication client and cluster cli

\[fix] \[client] fix memory leak if enabled pooled messages

\[fix]\[txn]fix receive duplicated messages due to pendingAcks in PendingAckHandle

\[fix]\[broker] Fixed history load not releasing

\[fix]\[broker] Allow proxy to pass same role for authRole and originalRole

\[fix]\[broker] Fix geo-replication admin

\[fix] \[test] Wrong mock-fail of the test ManagedLedgerErrorsTest.recoverLongTimeAfterMultipleWriteErrors

\[fix]\[broker] Terminate the async call chain when the condition isn't met for resetCursor

\[fix]\[proxy] Fix using wrong client version in pulsar proxy

\[improve]\[broker] Use shrink map for trackerCache

\[fix] \[broker] Make the service name resolver cache of PulsarWebResource expire after access

\[fix]\[authentication] Store the original authentication data

\[fix]\[broker] Copy command fields and fix potential thread-safety in ServerCnx

\[fix]\[client] Fix authentication not update after changing the serviceUrl

\[fix]\[broker] Make ServerCnx#originalAuthData volatile

\[fix]\[broker] Make authentication refresh threadsafe

\[fix] \[broker] Incorrect service name selection logic

\[fix]\[proxy] Fix JKS TLS transport

\[fix]\[client] Fix load the trust store file

\[fix]\[fn] Fix k8s merge runtime opts bug

\[improve]\[txn] Allow superusers to abort transactions

\[improve]\[broker] Require authRole is proxyRole to set originalPrincipal

\[fix]\[broker] Expect msgs after server initiated CloseProducer

\[fix] \[ml] topic load fail by ledger lost

\[revert]\[misc] "modify check waitingForPingResponse with volatile (#12615)"

\[improve]\[broker] Add test to verify authRole cannot change

\[fix]\[ml] Reset individualDeletedMessagesSerializedSize after acked all messages.

\[improve] Upgrade lombok to 1.8.26

\[improve] Upgrade to zk 3.8.1

\[fix]\[broker] Fix loadbalance score calculation problem

\[improve]\[misc] Upgrade Netty to 4.1.87.Final

\[fix]\[ml] Fix potential NPE cause future never complete.

\[fix]\[authorization] Fix the return value of canConsumeAsync

\[improve] \[broker] Print warn log if compaction failure

\[fix] \[ml] Fix the incorrect total size if use ML interceptor

\[fix]\[broker] Replace sync call in async call chain of AdminResource#internalCreatePartitionedTopic

\[fix]\[broker] Release Netty buffer in finally block in ServerCnx#handleLastMessageIdFromCompactedLedger

\[fix]\[client] Fix async completion in ConsumerImpl#processPossibleToDLQ

\[fix]\[broker] Replace sync method in NamespacesBase#internalDeleteNamespaceBundleAsync

\[fix]\[broker] Fix concurrency bug in PersistentTopicsBase#internalGetReplicatedSubscriptionStatus

\[fix] Close TransactionBuffer when create persistent topic timeout

\[fix]\[io] Update Elasticsearch sink idle cnx timeout to 30s

\[fix] \[ml] messagesConsumedCounter of NonDurableCursor was initialized incorrectly

\[fix]\[client] Fix reader listener can't auto ack with pooled message.

\[fix] \[ml] The atomicity of multiple fields of ml is broken

\[improve]\[broker] Added isActive in ManagedCursorImpl

\[improve] Upgrade wildfly-elytron (used by debezium) to fix CVE-2022-3143

\[fix]\[proxy] Only go to connecting state once

\[fix]\[client] Set fields earlier for correct ClientCnx initialization

\[improve]\[broker] ServerCnx: go to Failed state when auth fails

\[improve]\[txn] Handle changeToReadyState failure correctly in TC client

\[fix]\[broker] AbstractBatchedMetadataStore - use AlreadyClosedException instead of IllegalStateException

\[fix] \[ml] Topics stats shows msgBacklog but there reality no backlog

\[cleanup]\[broker] Validate originalPrincipal earlier in ServerCnx

\[fix]\[sec] Exclude log4j from openmldb

\[fix]\[txn] Catch and log runtime exceptions in async operations

\[fix] \[broker] Counter of pending send messages in Replicator incorrect if schema future not complete

\[improve]\[broker] Follow up #19230 to tighten the validation scope

\[fix]\[broker] Support deleting partitioned topics with the keyword `-partition-`

\[fix]\[broker] Copy subscription properties during updating the topic partition number.

\[improve]\[websocket] Add ping support

\[fix]\[doc] Remove lombok plugin and define FunctionRecordBuilder explicitly

\[fix]\[txn] fix txn coordinator recover handle committing and aborting txn race condition.

\[fix]\[broker] Fix race condition while updating partition number

\[fix]\[client] Fix producer could send timeout when enable batching

\[fix]\[broker] Pass subName for subscription operations in ServerCnx

\[fix]\[build] Upgrade dependency-check-maven plugin to fix broken OWASP check

\[improve]\[broker] Add ref count for sticky hash to optimize the performance of Key\_Shared subscription

\[fix]\[broker] Close transactionBuffer after MessageDeduplication#checkStatus failed

\[fix]\[broker] Support zookeeper read-only config.

\[fix]\[txn] Always send correct transaction id in end txn response

\[improve]\[schema] Do not print error log with stacktrace for 404

\[fix]\[broker] Topic could be in fenced state forever if deletion fails

\[fix]\[broker] Fix consumer with schema cannot subscribe when AUTO\_CONSUME consumers exist

\[fix]\[build] Resolve OWASP Dependency Check false positives

\[improve]\[sec] Suppress false positive OWASP reports

\[fix]\[sec] Upgrade woodstox to 5.4.0

\[fix]\[sec] Upgrade jettison to 1.5.3

\[fix]\[client] Prevent DNS reverse lookup when physical address is an IP address

\[improve]\[admin] Improve partitioned-topic condition evaluation

\[improve]\[broker] Add logs for why namespace bundle been split

\[fix]\[admin] Fix `validatePersistencePolicies` that Namespace/Topic persistent policies cannot set to \< 0

\[improve]\[broker] Omit making a copy of CommandAck when there are no broker interceptors

\[fix]\[broker] Fix deadlock in PendingAckHandleImpl

\[fix]\[broker] Copy proto command fields into final variables in ServerCnx

\[fix]\[broker]fix multi invocation for ledger createComplete

\[fix] \[tx] \[branch-2.11] Transaction buffer recover blocked by readNext

\[improve] Add PulsarExceptionBase class that supports slf4j like parameterized string formatting

\[fix]\[txn] transaction pending ack store future not completely problem

\[fix]\[broker]Update interceptor handler exception

\[improve]\[sec] Remove snakeyaml dependency to suppress CVE-2022-1471

\[fix]\[broker] Check `operstate` when get physical NICs

\[fix] \[broker] getLastMessageId returns a wrong batch index of last message if enabled read compacted

\[fix]\[fn] Typo in method name

\[fix]\[test]\[branch-2.10] Remove testGetTopic method

\[cherry-pick]\[branch-2.11] cherry-pick fixing can not delete namespace by force (#18307)

\[fix]\[broker] Fix uncompleted future when getting the topic policies of a deleted topic

\[fix]\[broker] Fix delete system topic clean topic policy

\[fix]\[broker] Fix namespace deletion if \_\_change\_events topic has not been created yet

\[fix]\[sql] Fix message without schema issue.

\[cleanup]\[broker] Simplify extract entryMetadata code in filterEntriesForConsumer

\[fix]\[broker] Fix duplicated schemas creation

\[fix]\[broker] transactional producer created failed due to read snapshot blocked.

\[fix]\[broker] Fix PulsarRegistrationClient and ZkRegistrationClient not aware rack info problem.

\[improve]\[test] Add test `brokerReachThreshold` for ThresholdShedderTest

\[improve]\[broker] Make Consumer#equals more effective

\[fix]\[broker] Create replicated subscriptions for new partitions when needed

\[fix]\[client] For exclusive subscriptions, if two consumers are created repeatedly, the second consumer will block

\[fix]\[io] ElasticSearch sink: align null fields behaviour

\[improve]\[broker] Adjust to sample the error log

\[fix]\[broker] DnsResolverUtil.TTL should be greater than zero

\[fix]\[client] Fix multi-topic consumer stuck after redeliver messages

\[fix]\[client] Avoid redelivering duplicated messages when batching is enabled

\[fix]\[client] Fix failover/exclusive consumer with batch cumulate ack issue.

\[fix]\[broker] Correctly set byte and message out totals per subscription

\[improve]\[client] Change the get lastMessageId to debug level

\[fix]\[broker] Fix open cursor with null-initialPosition result with earliest position

\[fix]\[broker]Fix failover/exclusive consumer cumulate ack not remove msg from UnAckedMessageTracker

\[fix]\[client] Fix possible npe

\[fix]\[client] Fixes batch\_size not checked in MessageId#fromByteArrayWithTopic

\[fix]\[client] Fix the Windows absolute path not recognized in auth param string

\[fix]\[client] Set authentication when using loadConf in client and admin client

\[fix]\[client] Fix exception when calling loadConf on a ConsumerBuilder that has a KeySharedPolicy

\[fix]\[client] Support LocalDateTime Conversion

\[fix]\[broker] Fix can not delete namespace by force

\[fix]\[broker] In the trimDeletedEntries method, release the removed entry

\[fix]\[schema] Fix creating lots of versions for the same schema

\[fix]\[broker] fix delete\_when\_subscriptions\_caught\_up doesn't work while have active consumers

\[fix]\[client] Fix `IllegalThreadStateException` when using newThread in `ExecutorProvider.ExtendedThreadFactory`

\[fix] \[pulsar-client] Fix pendingLookupRequestSemaphore leak when Ser…

\[improve]\[broker] Add UncaughtExceptionHandler for every thread pool

\[improve]\[broker] invalidate lock cache when Deleted NotificationType

\[improve]\[broker] Remove locallyAcquiredLock when removeOwnership

\[improve]\[broker] Support setting `forceDeleteTenantAllowed` dynamically

\[improve]\[broker] Support setting `ForceDeleteNamespaceAllowed` dynamically

\[fix]\[meta] fix `getChildren` in MemoryMetadataStore and EtcdMetadataStore

\[fix]\[broker] AVAILABLE\_PERMITS\_UPDATER not updated, when writeAndFlush fails.

\[fix]\[broker] PulsarRegistrationClient - implement getAllBookies and follow BookieServiceInfo updates

\[fix]\[broker] Fix update authentication data

\[fix]\[function] Fix invalid metric type "gauge "

\[fix]\[fn] fix function failed to start if no `typeClassName` provided in `FunctionDetails`

\[improve]\[client] Support MAX\_ACK\_GROUP\_SIZE configurable

\[improve]\[schema] Change update schema auth from tenant to produce

\[fix]\[broker] Update the log print content of createSubscriptions

\[improve]\[broker] Reduce unnecessary persistence of markdelete

\[fix]\[broker] Fix the order of resource close in the InMemoryDelayedDeliveryTracker

\[fix]\[broker] Fix incorrect bundle split count metric

\[fix]\[broker]unify time unit at dropping the backlog on a topic

\[improve]\[connector] JDBC sink: allow any jdbc driver

Make BookieId work with PulsarRegistrationDriver (second take)

\[fix]\[broker]Fix mutex never released when trimming

Allow to configure and disable the size of lookahead for detecting fixed delays in messages

\[fix]\[admin] returns 4xx error when pulsar-worker-service is disabled and trying to access it

\[fix]\[loadbalance] Fix the wrong NIC speed rate unit.

\[improve]\[broker]Improve PersistentMessageExpiryMonitor expire speed when ledger not existed

\[fix]\[client] moving get sequenceId into the sync code segment

\[refactor]\[java] Unify the acknowledge process for batch and non-batch message IDs

\[fix]\[proxy] Fix refresh client auth

\[fix]\[cli] Check numMessages after incrementing counter

\[improve]\[java-client]Add init capacity for messages in BatchMessageContainerImpl

\[fix]\[doc]fix comments for exposeManagedLedgerMetricsInPrometheus field

\[Improve]\[auth]Update authentication failed metrics report

\[fix]\[sec] Bump snakeyaml to 1.32 for CVE-2022-38752

\[fix]\[cli] Quit PerformanceConsumer after receiving numMessages messages

\[bugfix] ManagedLedger: move to FENCED state in case of BadVersionException

\[fix]\[broker] Fix create ns

\[fix]\[metrics]wrong metrics text generated when label\_cluster specified

\[fix]\[broker] Fix executeWithRetry result is null

\[fix]\[cli] Fix mbeans to json

\[fix]\[bookie] Correctly handle list configuration values

\[fix]\[build] duplicate entry when merging services

\[C++] Reset `havePendingPingRequest` flag for any data received from broker

\[improve]\[pulsar-proxy] Update proxy lookup throw exception type

Issue 17588: Allow deletion of a namespace that was left in deleted status

\[improve]\[test] Remove WhiteBox for ElasticSearchSinkTests

\[improve]\[test] remove WhiteBox on MockZooKeeper

\[improve]\[ci] Remove post commit builds and add manual workflows trigger

\[ci] Move more tests to the flaky suite

\[ci] Skip run flaky tests suite for cpp-only or doc-only changes

\[ci] Extract FLAKY tests suite to a dedicated workflow

\[ci] Reduce runners load for pulls that affect only doc or cpp changes

\[fix]\[tiered-storage] Don't cleanup data when offload met Metastore exception

\[improve]\[ci] Optimize "Pulsar Bot" workflow

\[improve]\[ci] Remove "Cancel duplicate workflows" workflow

\[fix]\[flaky-test]NamespaceServiceTest.flaky/testModularLoadManagerRemoveBundleAndLoad

\[fix]\[license] Update the log4j version in the presto license file

\[fix]\[sec] bump snakeyaml to 1.31 fix CVE-2022-25857

\[improve]\[broker] Improve cursor.getNumberOfEntries if isUnackedRangesOpenCacheSetEnabled=true

\[fix]\[client]Duplicate messages when use MultiTopicsConsumerImpl

\[improve]\[cli] Using separate TLS config on the compactor

\[improve]\[txn] Add getState in transaction for client API

\[fix]\[broker] Remove timestamp from Prometheus metrics

\[improve]\[broker] Using `handle` instead of `handleAsync` to avoid using common pool thread

\[fix]\[broker] fix can not revoke permission after update topic partition

\[improve]\[admin] PulsarAdminBuilderImpl overrides timeout properties passed through config map

\[fix]\[broker]fix catching ConflictException when update topic partition

\[fix]\[broker] Fix Npe thrown by splitBundle

\[fix]\[flaky] Fix flakyness in testAutoSchemaFunction

\[improve]\[broker]Only create extended partitions when updating partition number

\[improve]\[schema]autoSkipNonRecoverableData work for schema Ledger

\[improve]\[ci] Replace test reporting with less verbose solution

\[improve]\[broker]Recover as much cursor data as possible, even if entry is invalid

\[fix]\[broker]ManagedLedger metrics fail cause of zero period

\[refactor]\[java] Improve docs and code quality about KeyValueSchema usages

\[fix]\[broker]fail to update partition meta of topic due to ConflictException: subscription already exists for topic

\[broker]\[fix]Fix update topic remove properties

\[improve]\[build] Avoid building image multiple times

\[fix]\[broker]fix arithmetic exception for LeastResourceUsageWithWeight strategy

\[improve]\[docker] Switch to Temurin JDK

\[fix]\[broker] Skip not connectable URL when get redirectionUrl

\[improve]\[admin] Unset namespace policy to improve deleting namespace.

\[fix]\[broker] Fix delivery the message that has been acknowledged

\[fix]\[txn] Correct the prompt message

\[improve]\[ci] Skip unnecessary tests when there are only cpp/python related changes

\[Fix]\[flaky-test] Fix testConsumeTxnMessage

\[Improve]\[broker]Reduce GetReplicatedSubscriptionStatus local REST call

\[fix]\[ml] fix NPE

\[improve] clean the empty topicAuthenticationMap in zk when revoke permission

\[pulsar-io] KCA: handle kafka's logical schemas

\[improve]\[broker] ServerCnx: log at warning level when topic not found

\[fix]\[storage] Autorecovery default reppDnsResolverClass to ZkBookieRackAffinityMapping

\[fix]\[broker]\[functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)

\[refactor]\[broker] Arrange cleanup operation for topic creation

\[monitor]\[txn] Add metrics for transaction

Issue 14583: \[python client]: Improve garbage collection of producer/consumer/reader objects

\[feature]\[txn] Fix individual ack batch message with transaction abort redevlier duplicate messages

\[Improvement] Recycler should use io.netty.recycler.maxCapacityPerThread instead of io.netty.recycler.maxCapacity.default as maxCapacityPerThread configuration in Netty 4.1.x

\[Issue 12844]\[pulsar-client-cpp] Excessive locking cause significant performance degradation

### KoP

Increase timeout for KafkaListenerNameTest

\[Schema] Fix wrong response when querying schema by subject and version

\[bugfix] Fix regression with the upgrade to 2.8.x: Count corrently the number of partitions in ProduceRequest

\[Schema Registry] Fix the delete operation not work

\[Schema] Return 401 error when no HTTP authentication is configured

Return a correct metadata response when topic is not found

\[test] Add kafka admin api unit test

Improve getting started guide

\[fix]\[schema] Return JSON error message for failed request

### pulsarctl

Fix vulnerabilities

### Function Mesh Worker Service

fix ci in branch 2.11

fix build branch 2.11

adopt new api branch 2.10.3

Bump function-mesh to v0.13.0

support start\stop\restar function\sink\source


# StreamNative Weekly Release Notes v2.11.1.1
Source: https://docs.streamnative.io/release-notes/pulsar/v2.11/component-changelogs-v2.11.1.1



# StreamNative Weekly Release Notes v2.11.1.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.11.1.1](https://github.com/streamnative/pulsar/releases/tag/v2.11.1.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.11.1.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.11.1.1/images/sha256-52d72693f57c58b3298e0c2212280e91940bf3df3f2e96e6093c4016da8e273e)

## General Changes

### Apache Pulsar

\[fix]\[broker] Fix NPE cause by topic publish rate limiter.

\[fix]\[monitor] topic with double quote breaks the prometheus format

\[fix]\[txn] optimize the ack/send future in TransactionImpl

\[fix]\[broker] Fix the reason label of authentication metrics

\[fix]\[txn] Fix transaction is not aborted when send or ACK failed

\[fix]\[test] Fix flaky testCreateTopicWithZombieReplicatorCursor

\[fix]\[broker] Ignore and remove the replicator cursor when the remote cluster is absent

\[fix]\[broker]Make LedgerOffloaderFactory can load the old nar.

\[fix]\[broker] Fix can't send ErrorCommand when message is null value

\[fix]\[broker] Fix the thread safety issue of BrokerData#getTimeAverageData access

\[fix]\[client] Fix NPE when acknowledging multiple messages

\[fix] \[broker] error TimeUnit to record publish latency

\[fix]\[broker] Fix the behavior of delayed message in Key\_Shared mode

\[improve]\[broker] Get lowest PositionImpl from NavigableSet

\[fix]\[broker] Fix `RoaringBitmap.contains` can't check value 65535

\[improve] \[broker] Skip split boundle if only one broker

\[fix] \[broker] Fix infinite ack of Replicator after topic is closed

\[fix]\[monitor] Fix the partitioned publisher topic stat aggregation bug

\[improve]\[build] Upgrade dependencies to reduce CVE

4a9d75c1b1 Fix checkstyle and compile issue

\[fix] \[broker] Producer created by replicator is not displayed in topic stats

\[fix]\[client] Release the orphan producers after the primary consumer is closed

\[fix]\[broker] Fix Return value of getPartitionedStats doesn't contain subscription type

### KoP

\[fix] Unify fetch offset topic name

\[improve] Enable transaction and schema registry on docker compose

Fix pulsar entry formatter encode zero timestamp record caused exception

Upgrade org.json and kaml to non vulnerable versions

6c9fc0e6 Fix javadoc plugin source version

### StreamNative Pulsar Plugins

Upgrade jackson version

### Function Mesh Worker Service

branch2.11 fix list namespaced custom object call


# StreamNative Weekly Release Notes v2.11.1.2
Source: https://docs.streamnative.io/release-notes/pulsar/v2.11/component-changelogs-v2.11.1.2



# StreamNative Weekly Release Notes v2.11.1.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.11.1.2](https://github.com/streamnative/pulsar/releases/tag/v2.11.1.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.11.1.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.11.1.2/images/sha256-6006b1bbcc42a912538bc6f3f1904a84c734215f3f443c0c5b84ba4c1478234d)

## General Changes

### Apache Pulsar

6fe835ec4b Fix compile issue

\[fix]\[broker] Fix entry filter feature for the non-persistent topic

\[improve]\[monitor] Add JVM start time metric

\[improve]\[admin] Return BAD\_REQUEST on cluster data is null for createCluster

\[fix] \[broker] In Key\_Shared mode: remove unnecessary mechanisms of message skip to avoid unnecessary consumption stuck

\[fix]\[broker] fix consume stuck of shared streaming dispatcher

\[improve]\[cli] Allow pulser-client consume create a replicated subscription

\[fix]\[broker] Fix class name typo `PrecisPublishLimiter` to "Precise"

\[fix]\[broker]Fix deadlock of metadata store

\[improve]\[fn] Use functions classloader in TopicSchema.newSchemaInstance() to fix ClassNotFoundException when using custom SerDe classes. (targeted for master)

\[fix]\[test] Use delta when comparing doubles in checkLoadReportNicSpeed

\[fix]\[ml] Fix ledger left in OPEN state when enable `inactiveLedgerRollOverTimeMs`

\[fix]\[broker] Fix default bundle size used while setting bookie affinity

### KoP

Fix list offsets for times failure when ledgers are removed by a rollover operation

Fix flaky-test: KafkaNonPartitionedTopicTest.testNonPartitionedTopic

Use brokerClientTlsEnabled to configure pulsar client

\[fix]\[schema] Fix the schema registry tests and enabled the schema tests in CI

### StreamNative Pulsar Plugins

21e18bd5 Fix compile issue

Upgrade go version

Support publish json content type message

Unify the dependencies version between Pulsar and plugins

### Function Mesh Worker Service

fix yq

fix e2e pulsar install

Reuse existing configs

Show windowConfig when get function

Cleanup functions/sinks/sources after delete


# StreamNative Weekly Release Notes v2.11.1.3
Source: https://docs.streamnative.io/release-notes/pulsar/v2.11/component-changelogs-v2.11.1.3



# StreamNative Weekly Release Notes v2.11.1.3

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.11.1.3](https://github.com/streamnative/pulsar/releases/tag/v2.11.1.3)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.11.1.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.11.1.3/images/sha256-c241999c58e1a55ca457d125557f976716081d2c99b69583dccc4d8b4957d282)

## General Changes

### Apache Pulsar

93028b423d Remove python client build

([#20597)](https://github.com/apache/pulsar/pull/20597))) Revert "\[fix]\[broker] Fix NPE when reset Replicator's cursor by position.

Fix flaky test `testSplitBundleForMultiTimes`.

\[fix]\[broker] Update new bundle-range to policies after bundle split

0e1731c75d Fix license header issue

\[fix]\[broker] Revert "Skip loading broker interceptor when disableBrokerInterceptors is true #20422"

f9faef834d Fix cherry-pick #20595 causes compile issue

\[fix] \[client] Messages lost when consumer reconnect

\[fix]\[meta] Bookie Info lost by notification race condition.

\[fix]\[broker] Topic policy can not be work well if replay policy message has any exception.

\[fix]\[broker] Fix NPE when reset Replicator's cursor by position.

\[fix]\[client] Make the whole grabCnx() progress atomic

\[fix]\[sec] Upgrade snappy-java to address multiple CVEs

\[fix]\[io] Close the kafka source connector got stuck

\[fix]\[fn] Exit JVM when main thread throws exception

b45c2ed61b Fix cherry-pick #20605 cause compile issue.

\[fix]\[sql] Remove useless configuration for Pulsar SQL

\[fix]\[meta] Adding the missed bookie id in the registration manager.

\[fix]\[broker] Fix the publish latency spike from the contention of MessageDeduplication

\[fix]\[client]Fix deadlock issue of consumer while using multiple IO threads

\[improve]\[test] Disable disk usage threshold & geoip download and enable logging for Elastic Testcontainers

\[fix]\[misc] Use ubuntu 22.04 for Pulsar images

\[cleanup]\[broker] Validate authz earlier in delete subscription logic

445eacb25e Fix license headers to match JAVADOC\_STYLE

\[improve]\[broker] Support cgroup v2 by using `jdk.internal.platform.Metrics` in Pulsar Loadbalancer

\[fix]\[admin] Report earliest msg in partitioned backlog

\[fix]\[test] Replace test call to Auth0 with call to WireMock

\[fix] \[Perf] PerformanceProducer do not produce expected number of messages.

\[fix]\[offload] Filesystem offloader class not found hadoop-hdfs-client

\[fix]\[cli] Fulfill add-opens to function-localrunner also

\[fix]\[broker]fix the publish latency spike issue with large number of producers

\[fix]\[broker] Release orphan replicator after topic closed

\[Fix]\[txn] Unwrap the completion exception.

\[fix]\[offload] fix offload metrics error

\[improve]\[broker] Save createIfMissing in TopicLoadingContext

\[fix]\[fn] Make KubernetesRuntime translate characters in function tenant, namespace, and name during function removal to avoid label errors

\[fix]\[broker] REST Client Producer fails with TLS only

\[fix]\[ml] There are two same-named managed ledgers in the one broker

\[fix]\[broker] Restore solution for certain topic unloading race conditions

\[fix]\[fn] Configure pulsar admin for TLS

\[fix]\[build] duplicate entry when merging services

49d81af324 \[fix]\[build] Don't publish docker image with "latest" tag to docker repository

a38d82b487 Bump version to 2.11.2-SNAPSHOT

\[fix]\[sec] Upgrade Guava to 32.0.0 to address CVE-2023-2976

\[fix]\[fn] Go functions need to use static grpcPort in k8s runtime

\[fix]\[io] Close the kafka source connector if there is uncaught exception

\[fix] \[broker] do not filter system topic while shedding.

\[fix]\[client] Fix where the function getMsgNumInReceiverQueue always returns 0 when using message listener

373294bfa0 Fix cherry-pick #19929 cause license header issue

\[fix]\[client] Cache empty schema version in ProducerImpl schemaCache.

\[fix]\[fn]Reset idle timer correctly

\[fix]\[broker] Fix skip message API when hole messages exists

\[fix]\[test] Fix test `testThreadSwitchOfZkMetadataStore`

\[fix]\[fn] Fix function update error

\[fix]\[fn] Go functions must retrieve consumers by non-particioned topic ID

\[fix]\[broker] Skip loading broker interceptor when disableBrokerInterceptors is true

\[fix] \[meta]Switch to the metadata store thread after zk operation

\[improve]\[misc] Upgrade Netty to 4.1.93.Final

\[improve]\[misc] Upgrade Netty to 4.1.89.Final

\[improve]\[misc] Upgrade Netty to 4.1.87.Final

\[fix]\[fn] Fix JavaInstanceStarter inferring type class name error

\[improve] \[broker] Avoid `PersistentSubscription.expireMessages` logic check backlog twice.

a609f2baff Fix cherry-pick #18620 caused license header issue

\[fix]\[broker] Invalidate metadata children cache after key deleted

\[fix]\[broker] If ledger lost, cursor mark delete position can not forward

\[fix]\[sec] Upgrade sqlite-jdbc to resolve CVE-2023-32697

\[improve]\[ci] Speed up OWASP dependency check in Pulsar CI workflow

\[fix]\[ci] Update nar maven plugin version to fix excessive downloads

\[fix]\[broker] partitioned \_\_change\_events topic is policy topic

\[fix]\[fn] Make pulsar-admin support update py/go with package url

### AoP

\[ci] Ignore the jms1\_1 test first

### MoP

Bump Netty version to 4.1.93.Final

### KoP

\[branch-2.11] Fix failed GroupMetadataManagerTest

Fix flaky OffsetTopicWriteTimeoutTest

\[transactions] Better handling of network exceptions while sending TX markers

\[transaction] Producer state manager snapshot recovery - Part-1: Add snapshot I/O buffer

Reduce the offset commit timeout to 5 seconds and make it configurable

Fix flaky CompactedPartitionedTopicTest.testClose

\[fix] not respone for PRODUCE when acks=0

\[debug] add better log for CONCURRENT\_TRANSACTIONS error

\[perf]\[improvement] Improvements for PulsarEntryFormatter

\[bugfix] Fix decode pulsar format batch records timestamp

\[refactor] Decoupling the offset topic I/O from GroupMetadataManager

\[fix]\[transaction] TransactionMarkerRequestCompletionHandler retries on UNKNOWN\_SERVER\_ERROR

Prevent double-release on timeout

\[improve] Get size from byteBuf earlier to prevent unnecessary retention

\[improvement] hide scary InterruptedException in KopEventManager during broker shutdown

Document the tlsEnabled configuration for legacy KoP versions

\[bugfix]\[transactions] Make TxnTransitMetadata.topicPartitions immutable

\[bugfix]\[transactions] Prevent ConcurrentModificationException in getProducer()

\[bugfix]\[transactions] Release memory in TransactionMarkerChannelHandler

fix: remove topic.getManagedLedger().asyncDeleteCursor

\[feat]\[schema] Support getting schema string by id

\[fix] Fix read unstable messages

### pulsarctl

Fixed remove auth plugin suffix

Removed error char

### StreamNative Pulsar Plugins

\[fix]\[test] Reinstall the component `ca-certificates-java` in the DockerFile of detector test

Fix NPE when token used (Vault authentication) doesn't exists

Fix charts repo

Enable broker interceptors in integration tests.

Bump Go version `1.18` for `pulsarctl-plugins`

Enable broker interceptors for AuditLog test

### Cloud Pulsar Plugins

\[branch-2.11] Bump Pulsar to `2.11.1.2-arrowstreet`

fixed check styles and metric method name

Added revocation check in AuthenticationProviderApiKeys.

Added RevocationList

fixed license headers for pulsar-broker-auth-apikeys project

Added RevocationClient

Renamed Authentication/AuthorizationProviderOAuth to Authentication/AuthorizationProviderApiKeys

a copy of pulsar-broker-oauth2 for the new plugin work for api-key project

Release pulsar broker api keys

### Function Mesh Worker Service

Support manual semantics

a6ab0a17 Fix ci

Create VolumeMounts based on PVC

Support http protocol

Make log config works independent on CustomRuntimeOptions

Add support for node affinities and VolumeClaimTemplates

Support set log config name and key

Support liveness probe

fix auth e2e

set value schema when pulsar-client produce

Freeznet/use local registry for e2e

fix Not enough non-faulty bookies available

Implement trigger

Fix restartFunctions and enhance ci

Add imagePullSecrets to CustomRuntimeOptions

release function-mesh 0.14.0

Use k8s namespace to fetch resources

### Aws EventBridge Connector

Fix incorrect doc file name.


# StreamNative Weekly Release Notes v2.11.1.4
Source: https://docs.streamnative.io/release-notes/pulsar/v2.11/component-changelogs-v2.11.1.4



# StreamNative Weekly Release Notes v2.11.1.4

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.11.1.4](https://github.com/streamnative/pulsar/releases/tag/v2.11.1.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.11.1.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.11.1.4/images/sha256-fb61ba272ddc6b5dcdca88946bac4b8e328c8a170f15bcce35a6a50e75c95839)

## General Changes

### Apache Pulsar

\[fix]\[broker] Fix namespace deletion if \_\_change\_events topic has not been created yet

\[fix]\[broker] Fix get topic policies as null during clean cache

\[fix]\[client] Fix subscribing pattern topics through Proxy not working

\[fix] \[broker] Can not receive any messages after switch to standby cluster

\[fix]\[broker] Avoid throwing RestException in BrokerService

\[fix] \[io] elastic-search sink connector not support JSON.String schema.

\[improve] \[broker] Add consumer-id into the log when doing subscribe.

\[fix]\[io]\[branch-2.11] Not restart instance when kafka source poll ex…

da8b69fd45 Fix return the earliest position when query position by timestamp. #20457

\[fix]\[sec] Upgrade Guava to 32.1.1 to address CVE-2023-2976

\[fix]\[build] Update base image to 22.04 and remove the python client building

\[fix]\[schema] Only handle exception when there has

\[Revert]\[build] Ubuntu 22.04 was not compatible with Python Client Wheel

Revert "\[fix]\[build] Update Python Wheel to 3.10"

\[fix]\[ws] Remove unnecessary ping/pong implementation

\[fix]\[build] Update Python Wheel to 3.10

\[fix] \[txn] fix consumer can receive aborted txn message when readType is replay

### StreamNative Pulsar Plugins

\[rest] Fix bytebuf twice release

Supplemental testing for broker E2E compression

Support `manualDecompression` request flag

Support broker do compression.

Support publishing and consuming by rest

Support E2E compression - P2

Support E2E compression - P1

Specify antrun version to avoid api breaking

### Cloud Pulsar Plugins

\[branch-2.11]\[improve] Add cache for parse claims Jwt

Fix authenticate http request

\[improve] Move authz parse jwt body logic to authenticaiton state init stage

### Aws EventBridge Connector

Improve prerequisites docs.


# StreamNative Weekly Release Notes v2.11.2.1
Source: https://docs.streamnative.io/release-notes/pulsar/v2.11/component-changelogs-v2.11.2.1



# StreamNative Weekly Release Notes v2.11.2.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.11.2.1](https://github.com/streamnative/pulsar/releases/tag/v2.11.2.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.11.2.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.11.2.1/images/sha256-996ecff0123258e3929b9edbe81826be4d0bfcd248f966246213361797e96668)

## General Changes

### Apache Pulsar

\[improve]\[offload] Extend the offload policies to allow specifying more conf

\[improve]\[offload] Create offload resources lazily

\[improve]\[offload] Support to configure more offload driver

\[fix]\[build] Upgrade PyYaml version to 6.0.1

\[improve] \[ws] add cryptoKeyReaderFactoryClassName into the file websocket.conf

\[fix] \[cli] the variable producerName of BatchMsgContainer is null

### AoP

remove useless test code

### MoP

Fix `IllegalReferenceCountException` exception to break the callback

Change log level to debug

Add Cache for event writer.

847206d Fix connect timeout ms

### SN KoP

\[branch-2.11] Fix failed GroupMetadataManagerTest

Fix flaky OffsetTopicWriteTimeoutTest

\[transactions] Better handling of network exceptions while sending TX markers

\[transaction] Producer state manager snapshot recovery - Part-1: Add snapshot I/O buffer

Reduce the offset commit timeout to 5 seconds and make it configurable

Fix flaky CompactedPartitionedTopicTest.testClose

\[fix] not respone for PRODUCE when acks=0

\[debug] add better log for CONCURRENT\_TRANSACTIONS error

\[perf]\[improvement] Improvements for PulsarEntryFormatter

\[bugfix] Fix decode pulsar format batch records timestamp

\[refactor] Decoupling the offset topic I/O from GroupMetadataManager

\[fix]\[transaction] TransactionMarkerRequestCompletionHandler retries on UNKNOWN\_SERVER\_ERROR

Prevent double-release on timeout

\[improve] Get size from byteBuf earlier to prevent unnecessary retention

\[improvement] hide scary InterruptedException in KopEventManager during broker shutdown

Document the tlsEnabled configuration for legacy KoP versions

\[bugfix]\[transactions] Make TxnTransitMetadata.topicPartitions immutable

\[bugfix]\[transactions] Prevent ConcurrentModificationException in getProducer()

\[bugfix]\[transactions] Release memory in TransactionMarkerChannelHandler

fix: remove topic.getManagedLedger().asyncDeleteCursor

\[feat]\[schema] Support getting schema string by id

\[fix] Fix read unstable messages


# StreamNative Weekly Release Notes v2.11.2.2
Source: https://docs.streamnative.io/release-notes/pulsar/v2.11/component-changelogs-v2.11.2.2



# StreamNative Weekly Release Notes v2.11.2.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.11.2.2](https://github.com/streamnative/pulsar/releases/tag/v2.11.2.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.11.2.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.11.2.2/images/sha256-b36a32b24ef00b89a5c1e15cd52a7dbd329c804c9f40d7ebe51852b7e70393ba)

## General Changes

### Apache Pulsar

\[branch-2.11]\[fix]\[broker] Fix inconsensus namespace policies by getPoliciesIfCached

\[fix]\[broker] Avoid infinite bundle unloading

\[branch-2.10]\[fix]\[broker] Inconsistent behaviour for topic auto\_creation

\[fix]\[broker] In replication scenario, remote consumer could not be registered if there has no message was sent

### MoP

([#1047)](https://github.com/streamnative/mop/pull/1047))) Revert "Add ping request for adapter channel

Fix close reader NPE.

462c5b6 Improve test.

Add ping request for adapter channel

Fix mock object

Fix bundle is being unload IllegalState exception

Improve log to avoid too much error

Fix publish latency unit

Fix NPE cased by inflating message

Fix lookup issue for MQTT-5

### Function Mesh Worker Service

bump k8s to 1.23.17

remove java17 grammar for backward comp

add service account annotation if is created via sn cloud service account


# StreamNative Weekly Release Notes v2.11.2.3
Source: https://docs.streamnative.io/release-notes/pulsar/v2.11/component-changelogs-v2.11.2.3



## StreamNative Weekly Release Notes v2.11.2.3

#### General Changes

### SN KoP

\[build] Fix oauth client release

Bump snakeyaml from 1.31 to 1.32

\[oauthclient] Create a zero-dependencies jar: - remove Async HTTP client and use the standard JDK Http client - shade and relocate Jackson Databind, used for JSON

Revert "\[oauthclient] Create a zero-dependencies jar: - remove Async HTTP client and use the standard JDK Http client - shade and relocate Jackson Databind, used for JSON

c1106ef0 \[branch-2.11] Unify project version to 3.0.0-SNAPSHOT

ce297ae2 \[branch-2.11] Fix wrong project version in proxy module

\[Transaction] Fix initTransaction might wait until request timeout

Fix wrong offset increment for Sarama message set

\[improve]\[oauthclient] Support decode base64 format credentials URL

\[CI] Upload surefire artifacts when tests failed

\[proxy] Add a trivial proxy extension implementation and test framework

\[oauthclient] Create a zero-dependencies jar: - remove Async HTTP client and use the standard JDK Http client - shade and relocate Jackson Databind, used for JSON

\[docs] Add docs for OAuth credentials

Update README.md

\[security] Communications between broker inherit BrokerClient configuration

\[improvement] Save resources on the BK threads by not accessing the metrics context

\[bugfix] AppendRecordsContext cannot be Recyclable

\[improve] Pass group ID to authorizer when using OAuth

\[CI] Speed up CI test and fix flaky test

Optimize getHeadersFromMetadata, replace Java streams with for loop

\[improvement] Remove expensive useless String.format() in canConsumeAsync

### pulsarctl

Fix unknown property partitionedIndex

### StreamNative Pulsar Plugins

Bump branch-2.11 rely on pulsarctl branch-2.11

### Cloud Pulsar Plugins

Support audience list

Do not catch parseClaimsJwt method exception

### Function Mesh Worker Service

Remove enableStateStore from CustomRuntimeOptions

allow by-pass the class loader from connector package

Bump pulsar version to 2.11.1.3

release function-mesh 0.15.0

Use state store to query/put state

Add missing config

Handle delete response when failed to delete k8s object

4e4dc670 Use pulsarctl runner image

Add missed permissions checks

Append version to the description field of ConnectorDefinition

Respect --update-auth-data parameter

Fix permission error in ci's Dockerfile

prevent cleanup fail the action

expose k8s 404 and other rest errors with correct error code

### Aws EventBridge Connector

Refactor docs struct and content.

Improve config docs.


# StreamNative Weekly Release Notes v2.11.2.4
Source: https://docs.streamnative.io/release-notes/pulsar/v2.11/component-changelogs-v2.11.2.4



## StreamNative Weekly Release Notes v2.11.2.4

#### General Changes

### Cloud Pulsar Plugins

Fixed pulsar api key audience list issuer


# StreamNative Weekly Release Notes v2.11.2.5
Source: https://docs.streamnative.io/release-notes/pulsar/v2.11/component-changelogs-v2.11.2.5



## StreamNative Weekly Release Notes v2.11.2.5

#### General Changes

### Aws EventBridge Connector

Fix docs format for Note.

Fix wrong words in docs


# StreamNative Weekly Release Notes v2.11.2.6
Source: https://docs.streamnative.io/release-notes/pulsar/v2.11/component-changelogs-v2.11.2.6



# StreamNative Weekly Release Notes v2.11.2.6

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.11.2.6](https://github.com/streamnative/pulsar/releases/tag/v2.11.2.6)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.11.2.6/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.11.2.6/images/sha256-660b04b8cee73b94b2cc72688088fa5ca1b2bb4c75c481a0444c838b4ba0cb8f)

## General Changes

### Apache Pulsar

\[fix]\[auto-recovery] Improve to the ReplicaitonWorker performance by deleting invalid underreplication nodes

\[fix]\[meta] Fix deadlock in AutoRecovery.

524f20d138 \[fix]\[test]Fix a compilation issue

\[fix]\[client] Avoid ack hole for chunk message

\[fix]\[client] Fix consumer can't consume resent chunked messages

\[fix]\[broker]Fix chunked messages will be filtered by duplicating

\[improve] \[broker] Improve cache handling for partitioned topic metadata when doing lookup

\[improve] Introduce the sync() API to ensure consistency on reads during critical metadata operation paths

\[fix]\[broker] Make sure all inflight writes have finished before completion of compaction

\[fix]\[broker] Fix can't stop phase-two of compaction even though messageId read reaches lastReadId

([#20763)](https://github.com/apache/pulsar/pull/20763))) Revert "\[fix]\[broker] Fix get topic policies as null during clean cache

\[fix]\[broker] Fix get topic policies as null during clean cache

\[fix] \[bk] Correctct the bookie info after ZK client is reconnected

### Function Mesh Worker Service

d4dd33df Bump fm to 0.16.0

Fix error that `--retain-[key-]ordering` not working

Revert "Remove enableStateStore from CustomRuntimeOptions

Fix producerConfig cannot be updated error

Expose some internal errors to users

Fix typos and broken links

Fix error that --batch-builder doesn't work for functions

Add FM Worker service content


# V2.11.2.7
Source: https://docs.streamnative.io/release-notes/pulsar/v2.11/v2.11.2.7



# StreamNative Weekly Release Notes v2.11.2.7

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.11.2.7](https://github.com/streamnative/pulsar/releases/tag/v2.11.2.7)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.11.2.7/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.11.2.7/images/sha256-9dec238bd07bc419338f9d4b2da13fb3716ed5d48de8a917180de6a1698a260b)

## General Changes

### Apache Pulsar

\[branch-2.11] Fix compatibility issues

([#21041)](https://github.com/apache/pulsar/pull/21041))) Revert "\[fix]\[broker] Fix potential case cause retention policy not working on topic level

([#21137)](https://github.com/apache/pulsar/pull/21137))) Revert "\[fix]\[broker] Fix web tls url null cause NPE

\[fix]\[client] Fix logging problem in pulsar client

\[fix]\[proxy] Fix Proxy 502 gateway error when it is configured with Keystore TLS and admin API is called

\[fix]\[io] Fix --retain\[-key]-ordering not working error for sink

\[fix]\[fn] Fix ProducerConfig cannot update error

\[fix]\[fn] Fix the --batch-builder not working error for functions

\[fix]\[broker] Fix potential case cause retention policy not working on topic level

\[fix]\[broker] Fix web tls url null cause NPE

\[improve] \[broker] Improve logs for troubleshooting

\[fix]\[client] Fix cannot retry chunk messages and send to DLQ

218faf9523 \[cleanup]\[build] Remove useless file

### AMQP1\_0 Connector

Remove wrong COPY command.

\[CI] Adjust CI to test the corresponding Pulsar image version

Fix integration test.

### AWS SQS Connector

Improve sqs source doc.

Improve sqs sink docs.

### StreamNative Pulsar Plugins

Implement Oxia State Store

### Function Mesh Worker Service

Change connectorSearchIntervalSeconds default value to 600s.

Fix possible NPE errors

Set retain\[Key]Ordering to false if it is null

Load connector definition from ConnectorCataLog CRD.

Support json format logs and yaml format log config file

Change integration test ci trigger mode to pull\_request.

Support using sidecar to send logs to pulsar

Use AuthConfig.GenericAuth field to replace auth secret


# V2.11.2.8
Source: https://docs.streamnative.io/release-notes/pulsar/v2.11/v2.11.2.8



# StreamNative Weekly Release Notes v2.11.2.8

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.11.2.8](https://github.com/streamnative/pulsar/releases/tag/v2.11.2.8)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.11.2.8/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.11.2.8/images/sha256-e4223064096c28c1362d9e32a66074ed7943c9da7e02dad97165fc7b64d2cb5d)

## General Changes

### Apache Pulsar

\[fix]\[broker] Fix write duplicate entries into the compacted ledger after RawReader reconnects

\[fix]\[broker] Backport fix UniformLoadShedder selecet wrong overloadbroker and underloadbroker

\[fix] \[broker] Make specified producer could override the previous one

\[improve] \[broker] improve read entry error log for troubleshooting

\[fix] \[client] fix same producer/consumer use more than one connection per broker

\[fix]\[client] Fix repeat consume when using n-ack and batched messages

### pulsarctl

Support status check for pulsarctl command

*StreamNative Pulsar Plugins*

Fix OxiaStateStoreProviderImpl int to long error

### Cloud Pulsar Plugins

Fix REST API interceptor check for creating partitioned topic with properties

\[fix] Maximum topic limit


# V2.11.3.1
Source: https://docs.streamnative.io/release-notes/pulsar/v2.11/v2.11.3.1



# StreamNative Weekly Release Notes v2.11.3.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.11.3.1](https://github.com/streamnative/pulsar/releases/tag/v2.11.3.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.11.3.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.11.3.1/images/sha256-906fecb07126a84e9ac9139dbcaada3b0d10aaa8a52eae85f883d001267aae66)

## General Changes

### Apache Pulsar

cf5d7cba74 \[branch-2.11] Fix license header #21704.

\[fix] \[broker] Update topic policies as much as possible when some ex was thrown

\[fix]\[fn] Fix Deadlock in Functions Worker LeaderService

\[improve]\[proxy] Fix comment about enableProxyStatsEndpoints

79f16f4862 Upgrade OWASP dependency check maven plugin version

\[fix]\[broker] Fix the issue of topics possibly being deleted.

\[improve]\[broker] Upgrade bookkeeper to 4.15.5

\[fix]\[broker] Fix issue with consumer read uncommitted messages from compacted topic

\[fix]\[broker] Fix typo in the config key

\[improve]\[broker] Support not retaining null-key message during topic compaction

\[fix] \[broker] network package lost if enable haProxyProtocolEnabled

\[fix] \[ml] Fix orphan scheduled task for ledger create timeout check

\[fix] \[broker] Fix thousands orphan PersistentTopic caused OOM

\[fix]\[client] Fix producer could send timeout when enable batching

\[fix]\[broker] Fix memory leak during topic compaction

\[fix]\[broker] Fix incorrect unack count when using shared subscription on non-persistent topic

\[fix]\[ml] Fix unfinished callback when deleting managed ledger

\[fix]\[broker] Avoid pass null role in MultiRolesTokenAuthorizationProvider

\[fix]\[txn] OpRequestSend reuse problem cause tbClient commitTxnOnTopic timeout unexpectedly

\[fix]\[broker] Fix setReplicatedSubscriptionStatus incorrect behavior

\[fix]\[client] Fix print error log 'Auto getting partitions failed' when expend partition.

\[fix]\[broker] Fix the deadlock when using BookieRackAffinityMapping with rackaware policy

\[fix]\[broker] Fix resource\_quota\_zpath

dd4f566dbb \[fix] \[build] License define and imports

\[fix] \[log] fix the vague response if topic not found

\[improve] \[broker] Let the producer request success at the first time if the previous one is inactive

\[fix]\[broker] Correct schema deletion for parititioned topic

### AoP

Add rabbitmq amqp-client dependency for test

improve dependencies

### KoP

\[branch-2.11] Fix retryable error not handled well when appending messages

\[branch-2.11] Fix service unit not ready caused UnknownServerException

### Cloud Storage Connector

Update nick-invision to nick-fields

### AMQP1\_0 Connector

7185898 Fix format errors for note.

Refactor create a connector section docs.

Fix not success upload image

### AWS SQS Connector

Update nick-invision to nick-fields

5ed80f2 Fix format errors for note.

Refactor create a connector section docs.

Try fix auto labeling.

### AWS Lambda Connector

Update nick-invision to nick-fields

a57e737 Fix auto label bot not work.

### StreamNative Pulsar Plugins

Fix the packages cloud storage failed to find gs schema

\[pulsarctl-plugin] Bump client-go to `0.20.15`

### Cloud Pulsar Plugins

\[rest-api-interceptor] Ignore system topics for max topic count check

### Function Mesh Worker Service

Support load docsLink and iconLink for connector catalog.

update retry github action owner

Fallback to reason field if the lastState's message is empty

make memory padding configurable

Fix getSinkList and getSourceList impl to avoid showing fields details.

### Google Pub / Sub Connector

Update nick-invision to nick-fields

Fix auto label bot not work.

### Google BigQuery Sink Connector

Update nick-invision to nick-fields

d438b34 Fix format errors for note.

Refactor create a connector section docs.

Fix auto label bot not work.

### Snowflake Connector

Update nick-invision to nick-fields

### Aws EventBridge Connector

Update nick-invision to nick-fields

Fix typos in doc

Refactor create a connector section docs.

Fix auto label bot not work.

Fix some docs and deprecated eventBusResourceName config.

### Activemq Connector

Update nick-invision to nick-fields


# V2.11.3.2
Source: https://docs.streamnative.io/release-notes/pulsar/v2.11/v2.11.3.2



# StreamNative Weekly Release Notes v2.11.3.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.11.3.2](https://github.com/streamnative/pulsar/releases/tag/v2.11.3.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.11.3.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.11.3.2/images/sha256-cab8cb70e7b9163c1e075d2fba484887c249817de5a6187adfeafdeeb6e7a35d)

## General Changes

### Apache Pulsar

\[fix]\[broker] Fix lookupRequestSemaphore leak when topic not found

\[fix]\[test] Fix PerformanceProducer send count error

Fix String wrong format

\[fix]\[broker]Fix NonPersistentDispatcherMultipleConsumers ArrayIndexOutOfBoundsException

\[fix]\[broker] fix the wrong value of BrokerSrevice.maxUnackedMsgsPerDispatcher

\[fix] \[broker] Fix write all compacted out entry into compacted topic

\[fix]\[sec] Exclude avro from hadoop-client

\[fix]\[sec] Bump avro version to 1.11.3 for CVE-2023-39410

\[fix]\[misc] Bump GRPC version to 1.55.3 to fix CVE

\[fix]\[sec] Upgrade Netty to 4.1.100 to address CVE-2023-44487

95e1de78eb cve: exclude ch.qos.logback in canal.protocol \* resolve CVE-2023-6378

\[fix]\[broker]Delete compacted ledger when topic is deleted

\[fix] \[ml] Fix retry mechanism of deleting ledgers to invalidate

### MoP

Add test for resubscribe

Fix unsubscribe topic cause the test failed.

remove subs from subscription manager on unsubscribe

call equals on formatted strings since they will never be null

Add filter system topic when using EventCenter

### Cloud Storage Connector

Update base image

Update base image

### AMQP1\_0 Connector

Update base image

### AWS SQS Connector

Update base image

### AWS Lambda Connector

update-base-image

### StreamNative Pulsar Plugins

Fix the dependency conflict with pulsar broker

([#1054)](https://github.com/streamnative/sn-pulsar-plugins/pull/1054))) Revert "\[detector] Separate E2E latency detector per broker

([#1094)](https://github.com/streamnative/sn-pulsar-plugins/pull/1094))) Revert "Extend receive timeout to avoid context timeout

([#1100)](https://github.com/streamnative/sn-pulsar-plugins/pull/1100))) Revert "Support pulsar detector dashboard

([#1292)](https://github.com/streamnative/sn-pulsar-plugins/pull/1292))) Revert "\[fix]\[detector] Cleanup inactive broker's e2e detector

\[branch-2.11]\[cve] Update aws-java-sdk

\[fix]\[cve] Exclude logback from zookeeper

ae638df5 Fix failed TestE2ELatencyWithMetrics

\[fix]\[detector] Cleanup inactive broker's e2e detector

Support pulsar detector dashboard

Extend receive timeout to avoid context timeout

\[detector] Separate E2E latency detector per broker

Include one older txn log file in the backup when needed

### Function Mesh Worker Service

Use oxia:0.2 image for testing

### Google BigQuery Sink Connector

Update docker base

### Snowflake Connector

Update base image


# V2.11.3.3
Source: https://docs.streamnative.io/release-notes/pulsar/v2.11/v2.11.3.3



## StreamNative Weekly Release Notes v2.11.3.3

#### General Changes

### Apache Pulsar

\[fix]\[test] testModularLoadManagerRemoveBundleAndLoad

\[fix]\[test] Make base test class method protected so that it passes ReportUnannotatedMethods validation

bd50912378 Fix LICENSE

\[fix]\[sec] Upgrade Jetty to 9.4.54.v20240208 to address CVE-2024-22201

\[improve]\[fn] Add configuration for connector & functions package url sources

fdf2be1468 Adjust license header format

\[improve]\[broker] Add fine-grain authorization to retention admin API

\[fix] \[client] fix huge permits if acked a half batched message

\[fix] \[broker] Enabling batch causes negative unackedMessages due to ack and delivery concurrency

\[fix] \[broker] Replication stopped due to unload topic failed

\[fix] \[broker] Fix can not subscribe partitioned topic with a suffix-matched regexp

\[fix] \[broker] Fix break change: could not subscribe partitioned topic with a suffix-matched regexp due to a mistake of PIP-145

\[improve]\[build] Upgrade Apache ZooKeeper to 3.9.1

307a158d65 Bump version to 2.11.4-SNAPSHOT

\[fix]\[test] Fix test testTransactionBufferMetrics

\[fix]\[sec] Upgrade Jetty to 9.4.53 to address CVE-2023-44487

7f72fd4049 Revert changes to functions\_worker.conf used in system tests

f376b3a982 Fix LICENSE for shell

e2094d4874 Update LICENSE files

\[improve]\[fn] Optimize Function Worker startup by lazy loading and direct zip/bytecode access

\[fix]\[sec] Upgrade commons-compress to 1.26.0

\[fix]\[broker] Support running docker container with gid != 0

\[fix]\[broker]\[branch-3.1] Avoid PublishRateLimiter use an already closed RateLimiter

\[fix]\[broker] Sanitize values before logging in apply-config-from-env.py script

### KoP

\[test] Add test for abort transaction with Kafka admin

### AWS Lambda Connector

Enable unit tests for weekly release

### pulsarctl

\[branch-2.11]\[cve] Update golang.org/x/net to v0.19.0

### StreamNative Pulsar Plugins

95190160 \[fix]\[sec] Upgrade commons-compress to 1.26.0

\[fix]\[cve] Exclude avro,nimbus-jose-jwt,thirdparty from org.apache.hadoop

### Function Mesh Worker Service

19bd6e63 Fix ci

0075cc86 Deprecate classloader

reduce integration test image size with slim base image

bump function-mesh to 0.19.0

clean up the disk

Ignore exception when connector customize catalogs is empty.


# V2.11.3.4
Source: https://docs.streamnative.io/release-notes/pulsar/v2.11/v2.11.3.4



# StreamNative Weekly Release Notes v2.11.3.4

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.11.3.4](https://github.com/streamnative/pulsar/releases/tag/v2.11.3.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.11.3.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.11.3.4/images/sha256-b00765876f31d12e23295c41c2c8cdcd6bd86025b2bd5b8b66c54076e28b4918)

## General Changes

### KoP

\[CI] Fix docker-compose command not found

### AMQP1\_0 Connector

Auth SN docker hub

### AWS SQS Connector

Auth SN docker hub

### pulsarctl

Auth SN docker hub

Add docker hub login

### StreamNative Pulsar Plugins

\[branch-2.11] Fix code license check

Auth SN dockerhub

Update license error message for 2.11

Cherry pick license to branch 2.11

\[branch-2.11] Update go dependencies to fix CVEs

a4360fa1 \[fix]\[sec] Exclude `nimbus-jose-jwt` and `hadoop-shaded-protobuf_3_7` for module pulsar-tools and pulsar-metadata-tool

Use an old version of the sn/charts

### Function Mesh Worker Service

allow passing allowed runtimeFlags for java runtime

Check null value before use VpaSpec

Auth SN docker hub

0817fd16 Cleanup disk

Validate functions\&connectors package url

### Google Pub / Sub Connector

316cbce Add puul\_request trigger condition


# V2.11.4.1
Source: https://docs.streamnative.io/release-notes/pulsar/v2.11/v2.11.4.1



# StreamNative Weekly Release Notes v2.11.4.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v2.11.4.1](https://github.com/streamnative/pulsar/releases/tag/v2.11.4.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/2.11.4.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/2.11.4.1/images/sha256-8130c859af2b255507aaa2957e0568a749e06c466d9972211d3b3c18bad0a9d1)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/2.11.4.1/images/sha256-985eb5d182bb4b81baabe3ec49be6a16ceedb9b82ba3af05f3eb7eb2733e3fb8)

## General Changes

### Apache Pulsar

\[fix]\[broker] Fix NPE causing dispatching to stop when using Key\_Shared mode and allowOutOfOrderDelivery=true

\[improve]\[build] Upgrade OWASP Dependency check version to 9.1.0

\[improve]\[broker] Optimize gzip compression for /metrics endpoint by sharing/caching compressed result

ece1684113 \[fix]\[client]\[branch-2.11] Fix cherry-picking issue in #22393, address SpotBugs failure

\[fix]\[io] Kafka Source connector maybe stuck

\[fix]\[io] Config autoCommitEnabled when it disabled

\[fix]\[broker] Optimize /metrics, fix unbounded request queue issue and fix race conditions in metricsBufferResponse mode

ef20f2bdcd Remove unused import in PendingAckHandleImpl

\[improve]\[broker] Improve Gzip compression, allow excluding specific paths or disabling it

\[improve]\[test] Replace usage of curl in Java test and fix stream leaks

\[improve] \[broker] Servlet support response compression

\[fix] \[broker] Prevent long deduplication cursor backlog so that topic loading wouldn't timeout

\[fix]\[txn]Handle exceptions in the transaction pending ack init

\[fix]\[client] Fix client side memory leak when call MessageImpl.create and fix imprecise client-side metrics: pendingMessagesUpDownCounter, pendingBytesUpDownCounter, latencyHistogram

\[fix]\[broker] Fix invalid condition in logging exceptions

### KoP

Add metrics documents for network in/out bytes

### Cloud Storage Connector

cve: upgrade depend version

Bump org.apache.commons:commons-compress from 1.21 to 1.26.0

Bump jackson-databind from 2.13.4.1 to 2.13.4.2

Bump json-smart from 2.4.7 to 2.4.9

Use the Apache images to run tests, in order to avoid permission issues.

### AWS SQS Connector

9f0ff02 remove pull runner image

### AWS Lambda Connector

Fix pulsar-functions base images.

### StreamNative Pulsar Plugins

Increase Oxia Client create timeout to 60s

\[test] Fix metadata integration test

### Function Mesh Worker Service

3eb1fed4 Add brokerAdditionalServlet

### Google Pub / Sub Connector

chore: bump to JDK 17

Auth SN docker hub


# StreamNative Weekly Release Notes v3.0.0.1
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/component-changelogs-v3.0.0.1



# StreamNative Weekly Release Notes v3.0.0.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.0.1](https://github.com/streamnative/pulsar/releases/tag/v3.0.0.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.0.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.0.1/images/sha256-ed40206e8de15ea00a39792b4f8d0425e4635fb9bb6b385b6baf109dc146123d)

## General Changes


# StreamNative Weekly Release Notes v3.0.0.2
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/component-changelogs-v3.0.0.2



# StreamNative Weekly Release Notes v3.0.0.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.0.2](https://github.com/streamnative/pulsar/releases/tag/v3.0.0.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.0.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.0.2/images/sha256-6fd828834597df9672cd5ec44b20d4bec7cacd08049c16790611393dd3f17e62)

## General Changes

### Apache Pulsar

\[fix]\[fn] Go functions must retrieve consumers by non-particioned topic ID

\[fix]\[fn] Fix function update error

\[improve] \[broker] Avoid `PersistentSubscription.expireMessages` logic check backlog twice.

\[improve]\[monitor] Add JVM start time metric

\[fix]\[test] Fix SegmentAbortedTxnProcessorTest

\[improve]\[admin] Return BAD\_REQUEST on cluster data is null for createCluster

\[improve]\[cli] Allow pulser-client consume create a replicated subscription

\[improve]\[broker] Gracefully shut down load balancer extension

\[improve]\[fn] Use functions classloader in TopicSchema.newSchemaInstance() to fix ClassNotFoundException when using custom SerDe classes. (targeted for master)

\[fix] \[meta]Switch to the metadata store thread after zk operation

\[fix]\[broker] Skip loading broker interceptor when disableBrokerInterceptors is true

\[Fix]\[txn] Unwrap the completion exception.

\[fix]\[broker] If ledger lost, cursor mark delete position can not forward

\[fix]\[broker] Fix ledger cachemiss size metric

\[fix]\[broker] Invalidate metadata children cache after key deleted

\[fix]\[fn] Fix JavaInstanceStarter inferring type class name error

\[fix]\[broker] pre-create non-partitioned system topics for load balance extension

\[fix]\[broker] Fix broker load manager class filter NPE

\[fix]\[broker] managedLedger.getConfig().getProperties().putAll(properties) NPE

\[fix] \[broker] In Key\_Shared mode: remove unnecessary mechanisms of message skip to avoid unnecessary consumption stuck

\[fix]\[broker] Use user-specified bundle size on creating a namespace anti-affinity group with the default local policies

\[fix]\[broker]Fix deadlock of metadata store

\[fix]\[doc] Correcting spelling mistakes

7b74d89f6a correcting spelling mistakes

\[fix]\[ml] Fix ledger left in OPEN state when enable `inactiveLedgerRollOverTimeMs`

650d66c356 \[fix]\[client] thread-safe seek

([#20242)](https://github.com/apache/pulsar/pull/20242))) Revert "\[fix]\[client] Seek should be thread-safe

\[fix]\[broker] Fix class name typo `PrecisPublishLimiter` to "Precise"

\[fix]\[broker] Allow Access to System Topic Metadata for Reader Creation Post-Namespace Deletion

\[fix]\[build] Fix publish image script

\[fix]\[fn] Correct TLS cert config translation from broker to fn worker

\[fix]\[broker] Fix NPE cause by topic publish rate limiter.

\[fix]\[fn] Support multiple input topics for Go runtime

\[fix]\[broker] Fix default bundle size used while setting bookie affinity

\[fix]\[io] add protobuf ByteString to pulsar-io jdbc core

\[fix]\[broker] Fix `UnsupportedOperationException` when update topic properties.

\[fix]\[client] Seek should be thread-safe

\[improve]\[misc] Upgrade Netty to 4.1.93.Final

\[improve]\[build] Upgrade maven surefire plugin and other build/test plugins/libs including TestNG version

\[fix]\[sec] Upgrade sqlite-jdbc to resolve CVE-2023-32697

\[improve]\[ci] Replace handmade action to configure Gradle Enterprise

\[improve]\[build] Capture local build scans on ge.apache.org to benefit from deep build insights

\[improve]\[ci] Speed up OWASP dependency check in Pulsar CI workflow

\[improve]\[ci] Split Pulsar IO unit test job to multiple jobs

\[improve]\[ci] Disable Maven http connection pooling on CI also for newer Maven versions

\[fix]\[ci] Update nar maven plugin version to fix excessive downloads

\[fix]\[broker] partitioned \_\_change\_events topic is policy topic

\[fix]\[fn] Make pulsar-admin support update py/go with package url

\[feat] OIDC: support JWKS refresh for missing Key ID

\[fix]\[txn] Implement compatibility for transaction buffer segmented snapshot feature upgrade

\[fix]\[test] Use delta when comparing doubles in checkLoadReportNicSpeed

\[fix]\[monitor] topic with double quote breaks the prometheus format

### KoP

\[feat]\[schema] Support getting schema string by id

\[Snyk] Security upgrade io.streamnative:pulsar-broker from 3.0.0.1-rc1 to 3.0.0.1

Fix list offsets for times failure when ledgers are removed by a rollover operation

Fix flaky-test: KafkaNonPartitionedTopicTest.testNonPartitionedTopic

\[fix] Fix read unstable messages

### pulsarctl

Fixed remove auth plugin suffix

Removed error char

Bump pulsar version to 3.0.0.1

### StreamNative Pulsar Plugins

Fix charts repo

Support publish json content type message

### Cloud Pulsar Plugins

fixed check styles and metric method name

Added revocation check in AuthenticationProviderApiKeys.

Added RevocationList

fixed license headers for pulsar-broker-auth-apikeys project

Added RevocationClient

Renamed Authentication/AuthorizationProviderOAuth to Authentication/AuthorizationProviderApiKeys

a copy of pulsar-broker-oauth2 for the new plugin work for api-key project

Release pulsar broker api keys

### Function Mesh Worker Service

Fix restartFunctions and enhance ci

Add imagePullSecrets to CustomRuntimeOptions

release function-mesh 0.14.0

Use k8s namespace to fetch resources

fix yq

fix e2e pulsar install

Reuse existing configs

Show windowConfig when get function


# StreamNative Weekly Release Notes v3.0.0.3
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/component-changelogs-v3.0.0.3



# StreamNative Weekly Release Notes v3.0.0.3

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.0.3](https://github.com/streamnative/pulsar/releases/tag/v3.0.0.3)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.0.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.0.3/images/sha256-fb26d07ea0a3efd7276e89b46471616a023ba84788dd15222ba460507db8864a)

## General Changes

### Apache Pulsar

\[fix]\[broker] getOwnedServiceUnits NPE

\[fix]\[broker] release orphan replicator after topic closed

\[fix]\[admin] Report earliest msg in partitioned backlog

\[fix]\[broker] Handle heartbeat namespace in ExtensibleLoadManager

\[improve]\[broker] Handle get owned namespaces admin API in ExtensibleLoadManager

\[improve]\[broker] Emit the namespace bundle listener event on extensible load manager

\[feat]\[broker]PIP-255 Part-1: Add listener interface for namespace service

\[fix]\[broker] Fix redirect loop when using ExtensibleLoadManager and list in bundle admin API

\[fix]\[broker] new load balancer system topic should not be auto-created now

\[fix]\[misc] Use ubuntu 22.04 for Pulsar images

\[fix]\[fn] Make KubernetesRuntime translate characters in function tenant, namespace, and name during function removal to avoid label errors

\[fix]\[authentication] Improve AuthenticationFilter response

\[cleanup]\[broker] Validate authz earlier in delete subscription logic

\[fix]\[build] Configure git-commit-id-plugin to skip git describe

\[fix]\[broker] REST Client Producer fails with TLS only

\[fix]\[broker] Disable EntryFilters for system topics

\[improve]\[build] Upgrade Testcontainers to 1.18.3 & docker-java to 3.3.0

\[fix]\[test] Reduce flakiness of AdminApi2Test

\[fix]\[broker] Restore solution for certain topic unloading race conditions

\[fix]\[ml] There are two same-named managed ledgers in the one broker

\[improve]\[broker] Do not expose bucketDelayedIndexStats

\[fix]\[offload] fix offload metrics error

\[fix]\[client] Fix where the function getMsgNumInReceiverQueue always returns 0 when using message listener

\[fix]\[cli] Fix logging noise while admin tool exit

\[fix] \[broker] do not filter system topic while shedding.

\[fix]\[fn] TLS args admin download command use zero arity

\[fix]\[cli] Fulfill add-opens to function-localrunner also

\[fix]\[broker] Fix skip message API when hole messages exists

\[fix]\[build] Fix the pulsar-all image may use the wrong upstream image

398a781a45 Bump version to 3.0.1-SNAPSHOT

\[fix]\[ci] Fix OWASP dependency check suppressions

\[fix]\[test] Replace calls to Auth0 with calls to wiremock

\[improve]\[ci] Increase Maven max heap size to 1024m in all GHA workflows

\[fix]\[fn] Go functions need to use static grpcPort in k8s runtime

\[fix]\[fn] Support customizing TLS config for function download command

\[fix]\[test] Replace test call to Auth0 with call to WireMock

\[fix]\[test] Remove dependency on httpbin.org service in FunctionCommonTest

\[fix]\[sec] Upgrade Guava to 32.0.0 to address CVE-2023-2976

\[fix]\[io] Close the kafka source connector if there is uncaught exception

\[fix]\[fn]Reset idle timer correctly

### MoP

eeb7338 Upgrade mqtt.codec.version from 4.1.89 to 4.1.93

### KoP

\[fix] not respone for PRODUCE when acks=0

\[refactor] Decoupling the offset topic I/O from GroupMetadataManager

\[debug] add better log for CONCURRENT\_TRANSACTIONS error

\[perf]\[improvement] Improvements for PulsarEntryFormatter

\[bugfix] Fix decode pulsar format batch records timestamp

\[fix]\[transaction] TransactionMarkerRequestCompletionHandler retries on UNKNOWN\_SERVER\_ERROR

Prevent double-release on timeout

\[improve] Get size from byteBuf earlier to prevent unnecessary retention

\[improvement] hide scary InterruptedException in KopEventManager during broker shutdown

Document the tlsEnabled configuration for legacy KoP versions

\[bugfix]\[transactions] Make TxnTransitMetadata.topicPartitions immutable

\[bugfix]\[transactions] Prevent ConcurrentModificationException in getProducer()

\[bugfix]\[transactions] Release memory in TransactionMarkerChannelHandler

fix: remove topic.getManagedLedger().asyncDeleteCursor

### StreamNative Pulsar Plugins

Fix NPE when token used (Vault authentication) doesn't exists

### Function Mesh Worker Service

Create VolumeMounts based on PVC

Use SNBOT token

Support http protocol

Add github server to maven

Cleanup functions/sinks/sources after delete

Make log config works independent on CustomRuntimeOptions

Add support for node affinities and VolumeClaimTemplates

Support set log config name and key

Support liveness probe

fix auth e2e

set value schema when pulsar-client produce

Freeznet/use local registry for e2e

fix Not enough non-faulty bookies available

fix branch-3.0 build

Implement trigger


# StreamNative Weekly Release Notes v3.0.0.4
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/component-changelogs-v3.0.0.4



# StreamNative Weekly Release Notes v3.0.0.4

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.0.4](https://github.com/streamnative/pulsar/releases/tag/v3.0.0.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.0.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.0.4/images/sha256-c2de684b34ef0f5e81449ed1aaa643e6d96c92098dd6dadeedbe7d38259575ae)

## General Changes

### Apache Pulsar

([#20597)](https://github.com/apache/pulsar/pull/20597))) Revert "\[fix]\[broker] Fix NPE when reset Replicator's cursor by position.

\[fix]\[broker] Revert "Skip loading broker interceptor when disableBrokerInterceptors is true #20422"

\[improve]\[broker] Make ExtensibleLoadManagerImpl's broker filter pure async

\[fix]\[test] Fix flaky test ExtensibleLoadManagerTest.testIsolationPolicy

\[improve]\[test] Add integration test for ExtensibleLoadManager

\[fix]\[broker] Added the skipped message handler for ServiceUnitStateChannel

\[fix]\[client] Make the whole grabCnx() progress atomic

\[improve]\[broker] Make get list from bundle Admin API async

\[fix]\[sql] Remove useless configuration for Pulsar SQL

\[fix]\[io] Close the kafka source connector got stuck

\[fix]\[txn] Use PulsarResource check for topic existence instead of brokerservice.getTopic()

\[fix]\[offload] Filesystem offloader class not found hadoop-hdfs-client

\[fix] \[Perf] PerformanceProducer do not produce expected number of messages.

\[fix]\[broker] Fix return the earliest position when query position by timestamp.

\[fix]\[broker] Topic policy can not be work well if replay policy message has any exception.

\[fix]\[broker] Fix NPE when reset Replicator's cursor by position.

\[fix]\[meta] Bookie Info lost by notification race condition.

\[fix]\[meta] Adding the missed bookie id in the registration manager.

\[improve]\[broker] Upgrade bookkeeper to 4.16.2

\[fix]\[broker]fix the publish latency spike issue with large number of producers

\[fix]\[fn] Exit JVM when main thread throws exception

\[fix]\[client]Fix deadlock issue of consumer while using multiple IO threads

\[fix]\[broker] Fix the publish latency spike from the contention of MessageDeduplication

\[fix] \[admin] set ns level backlog quota does not take effect if retention exists

\[improve]\[test] Disable disk usage threshold & geoip download and enable logging for Elastic Testcontainers

\[improve]\[bk] Add integration test with bookie http server enabled

\[improve]\[broker] Support cgroup v2 by using `jdk.internal.platform.Metrics` in Pulsar Loadbalancer

### AoP

\[ci] Ignore the jms1\_1 test first

### KoP

\[transaction] Implement producer state manager recovery

Fix flaky OffsetTopicWriteTimeoutTest

\[transactions] Better handling of network exceptions while sending TX markers

\[transaction] Producer state manager snapshot recovery - Part-1: Add snapshot I/O buffer

Reduce the offset commit timeout to 5 seconds and make it configurable

Fix flaky CompactedPartitionedTopicTest.testClose

\[improvements] Use the TopicEventListener API

\[improvement] Remove expensive useless String.format() in canConsumeAsync

### StreamNative Pulsar Plugins

\[fix]\[test] Reinstall the component `ca-certificates-java` in the DockerFile of detector test

### Function Mesh Worker Service

Support manual semantics


# StreamNative Weekly Release Notes v3.0.1.1
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/component-changelogs-v3.0.1.1



# StreamNative Weekly Release Notes v3.0.1.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.1.1](https://github.com/streamnative/pulsar/releases/tag/v3.0.1.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.1.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.1.1/images/sha256-7a32c245ac8f1de8748dfc1615c519cd87b6bfa1e0b40704b7de8c9236e66161)

## General Changes

### Apache Pulsar

\[fix]\[broker] fix ModularLoadManagerImpl always delete active bundle-data. sec ver.

\[fix]\[client] Fix RawReader hasMessageAvailable returns true when no messages

\[fix]\[meta] Fix deadlock in AutoRecovery.

\[fix]\[broker] Fix incorrect unack msk count when dup ack a message

\[fix]\[broker] Fix compaction subscription delete by inactive subscription check.

\[fix] \[admin] Fix get topic stats fail if a subscription catch up concurrently

\[fix]\[broker] Fix inconsensus namespace policies by `getPoliciesIfCached`

\[improve]\[broker] Avoid print redirect exception log when get list from bundle

\[fix]\[broker] Inconsistent behaviour for topic auto\_creation

\[improve] \[ws] add cryptoKeyReaderFactoryClassName into the file websocket.conf

\[improve]\[offload] Extend the offload policies to allow specifying more conf

\[improve]\[client] Disable polling pattern topics when TopicListWatcher is enabled.

\[fix]\[test] Close the resource after the test

\[improve]\[offload] Create offload resources lazily

\[fix] \[broker] Can not receive any messages after switch to standby cluster

\[fix]\[broker] Fix get topic policies as null during clean cache

\[fix]\[broker] Avoid throwing RestException in BrokerService

\[fix]\[client] Fix subscribing pattern topics through Proxy not working

\[improve]\[offload] Support to configure more offload driver

\[fix]\[schema] Only handle exception when there has

\[fix] \[client] Messages lost when consumer reconnect

\[improve]\[admin] Remove duplicate topics name when `deleteNamespace`

\[fix] \[txn] fix consumer can receive aborted txn message when readType is replay

\[fix] \[io] elastic-search sink connector not support JSON.String schema.

\[improve] \[broker] Add consumer-id into the log when doing subscribe.

\[fix]\[broker]Check that the super user role is in the MultiRolesTokenAuthorizationProvider plugin

\[fix]\[broker] Gracefully shutdown does not work with admin cli in standalone

\[improve]\[sql] Fix the wrong format of the logs

\[fix]\[client] Fix perf-producer get OOM with high publish latency

\[fix]\[broker] fix MessageDeduplication throw NPE when enable broker dedup and set namespace disable deduplication.

\[fix]\[io] Update test certs for Elasticsearch

\[fix]\[broker] Fix message loss during topic compaction

([#20980)](https://github.com/apache/pulsar/pull/20980))) Revert "\[fix]\[broker] Fix message loss during topic compaction

\[fix]\[broker] Fix message loss during topic compaction

\[fix]\[broker] Fix incorrect number of read compacted entries

\[fix] \[ml] fix discontinuous ledger deletion

\[fix]\[broker] In replication scenario, remote consumer could not be registered if there has no message was sent

\[improve]\[broker] Add annotation for topic compaction strategy

\[fix]\[broker] Avoid infinite bundle unloading

\[improve]\[broker] Add broker filter sync method back to guarantee the API compatibility

\[fix]\[build] Upgrade PyYaml version to 6.0.1

\[fix] \[cli] the variable producerName of BatchMsgContainer is null

\[fix]\[broker] call ServerCnx#closeProducer from correct thread

\[fix]\[io]\[branch-3.0] Not restart instance when kafka source poll exception.

\[fix]\[test] Fix resource leak in PulsarTestContext

\[fix]\[test] Fix flaky PersistentSubscriptionTest

\[fix]\[sec] Upgrade snappy-java to address multiple CVEs

f3bb89d4a6 Revert Add listener interface for namespace service #20406

\[fix] Ignore openIDTokenIssuerTrustCertsFilePath conf when blank

\[fix]\[ws] Remove unnecessary ping/pong implementation

### AoP

remove useless test code

### MoP

Fix connection timeout ms

Fix dispatch docker build

Support fast build docker image

Support batch ack in broker

Upgrade pulsar verison to 3.0.0.4

([#1047)](https://github.com/streamnative/mop/pull/1047))) Revert "Add ping request for adapter channel

Fix close reader NPE.

113a653 Improve test.

Add ping request for adapter channel

Fix mock object

Fix publish latency unit

Improve log to avoid too much error

Fix bundle is being unload IllegalState exception

Fix NPE cased by inflating message

Fix lookup issue for MQTT-5

Fix `IllegalReferenceCountException` exception to break the callback

Change log level to debug

Add Cache for event writer.

843fdf1 Fix connect timeout ms

### StreamNative Pulsar Plugins

\[rest] Fix bytebuf twice release

Specify antrun version to avoid api breaking

Supplemental testing for broker E2E compression

Support `manualDecompression` request flag

Support broker do compression.

Support publishing and consuming by rest

Support E2E compression - P2

Support E2E compression - P1

### Cloud Pulsar Plugins

Fixed pulsar api key audience list issuer

Support audience list

Do not catch parseClaimsJwt method exception

\[improve] Add cache for parse claims Jwt

Fix authenticate http request

\[improve] Move authz parse jwt body logic to authenticaiton state init stage

### Function Mesh Worker Service

Remove enableStateStore from CustomRuntimeOptions

allow by-pass the class loader from connector package

dependencies alignment

release function-mesh 0.15.0

Use state store to query/put state

Add missing config

Handle delete response when failed to delete k8s object

bd9f2b03 Use pulsarctl runner image

Add missed permissions checks

Append version to the description field of ConnectorDefinition

Respect --update-auth-data parameter

expose k8s 404 and other rest errors with correct error code

bump k8s to 1.23.17

remove java17 grammar for backward comp

Bump k8s to 18.0.0

add service account annotation if is created via sn cloud service account

Fix permission error in ci's Dockerfile

prevent cleanup fail the action

### Aws EventBridge Connector

Refactor docs struct and content.

Improve config docs.

Improve prerequisites docs.


# StreamNative Weekly Release Notes v3.0.1.2
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/component-changelogs-v3.0.1.2



# StreamNative Weekly Release Notes v3.0.1.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.1.2](https://github.com/streamnative/pulsar/releases/tag/v3.0.1.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.1.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.1.2/images/sha256-388d3a124fc52af46779443867faea67c42e8ca01eb841b538397f7ba3cab954)

## General Changes

### Apache Pulsar

\[improve] \[broker] Improve cache handling for partitioned topic metadata when doing lookup

\[fix]\[broker] Fix potential case cause retention policy not working on topic level

\[fix] \[broker] Producer is blocked on creation because backlog exceeded on topic, when dedup is enabled and no producer is there

\[fix]\[broker] Make sure all inflight writes have finished before completion of compaction

\[fix]\[broker] Fix can't stop phase-two of compaction even though messageId read reaches lastReadId

([#20763)](https://github.com/apache/pulsar/pull/20763))) Revert "\[fix]\[broker] Fix get topic policies as null during clean cache

\[fix]\[broker] Fix get topic policies as null during clean cache

\[fix] \[bk] Correctct the bookie info after ZK client is reconnected

\[fix]\[broker] Use MessageDigest.isEqual when comparing digests

\[improve]\[proxy] Support disabling metrics endpoint

\[fix]\[sec] Upgrade Netty to 4.1.94.Final to address CVE-2023-34462

### Function Mesh Worker Service

Bump fm to 0.16.0

Fix error that `--retain-[key-]ordering` not working

Revert "Remove enableStateStore from CustomRuntimeOptions

Fix producerConfig cannot be updated error

Expose some internal errors to users

Fix typos and broken links

Fix error that --batch-builder doesn't work for functions

Add FM Worker service content

### Aws EventBridge Connector

Fix docs format for Note.

Fix wrong words in docs


# V3.0.1.5
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.1.5



# StreamNative Weekly Release Notes v3.0.1.5

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.1.5](https://github.com/streamnative/pulsar/releases/tag/v3.0.1.5)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.1.5/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.1.5/images/sha256-4c3099ad7e42bc36337af8b0a4d7a548d9c450996ae14e3509426a7775b4e1f9)

## General Changes

### Apache Pulsar

\[fix]\[broker] Fix unload operation stuck when use ExtensibleLoadManager

\[fix]\[broker]\[branch-3.0] Fix inconsistent topic policy

\[fix]\[sec] Upgrade Zookeeper to 3.8.3 to address CVE-2023-44981

\[fix]\[sec] Upgrade Netty to 4.1.100 to address CVE-2023-44487

\[fix]\[sec] Upgrade Jetty to 9.4.53 to address CVE-2023-44487

\[fix]\[test] Fix LocalBookkeeperEnsemble resource leak in tests

\[fix]\[broker] Fix heartbeat namespace create event topic and cannot delete heartbeat topic

\[fix]\[broker] Fix heartbeat namespace create transaction internal topic

\[improve]\[broker] use ConcurrentHashMap in ServiceUnitStateChannel and avoid recursive update error

\[fix] \[test] \[branch-3.0] Fix AutoRecovery flaky test.

\[fix]\[test] Fix AuditorLedgerCheckerTest flaky test.

\[fix]\[broker] Fix lookup heartbeat and sla namespace bundle when using extensible load manager

\[improve]\[ci] Add new CI unit test group "Broker Group 4" with cluster migration tests

\[fix] \[auto-recovery] Fix PulsarLedgerUnderreplicationManager notify problem.

\[fix]\[test] Fix resource leaks with Pulsar Functions tests

\[fix]\[test] Fix some resource leaks in compaction tests

\[feat]\[sql] Support UUID for json and avro

\[fix]\[test] Fix a resource leak in ClusterMigrationTest

\[fix] \[bk-client] Fix bk client MinNumRacksPerWriteQuorum and EnforceMinNumRacksPerWriteQuorum not work problem.

\[fix]\[ci] Fix docker image building by releasing more disk space before building

\[fix] \[ml] fix wrong msg backlog of non-durable cursor after trim ledgers

\[fix] \[ml] Reader can set read-pos to a deleted ledger

\[fix] \[broker] fix flaky test PatternTopicsConsumerImplTest

\[fix]\[sec] Fix MultiRoles token provider when using anonymous clients

\[fix]\[test] Fix flaky test NarUnpackerTest

\[fix] \[metadata] Fix zookeeper related flacky test

([#21231)](https://github.com/apache/pulsar/pull/21231))) Revert "\[fix]\[broker] Fix inconsistent topic policy

\[improve] \[auto-recovery] Migrate the replication testing from BookKeeper to Pulsar.

\[fix]\[broker] Fix inconsistent topic policy

\[fix]\[broker] rackaware policy is ineffective when delete zk rack info after bkclient initialize

\[fix]\[broker] fix bug caused by optimistic locking

\[improve] \[broker] Not close the socket if lookup failed caused by bundle unloading or metadata ex

\[fix] \[client] fix reader.hasMessageAvailable return false when incoming queue is not empty

\[improve] \[broker] Print warn log if ssl handshake error & print ledger id when switch ledger

\[fix]\[ml] Fix thread safe issue with RangeCache.put and RangeCache.clear

### KoP

\[fix] Run messageReadStats metrics registerFailedEvent execute on netty thread

Ignore metadata init exception to avoid rolling upgrade from failing

Ignore the flaky MultiLedgerTest.testListOffsetForEmptyRolloverLedger

Fix txn marker to the offset topic cannot be read

### AMQP1\_0 Connector

c49dc72 Cherr-picked from #721: Improve sink and source connector docs.

### Cloud Pulsar Plugins

\[rest-api-interceptor] Ignore system topics for max topic count check

### Function Mesh Worker Service

disable golang runtime by default

allow submit very long name resources

Set usingInsecureAuth to false by default

Add configs: javaOpts/labels/logConfig to CustomRuntimeOptions

### Google BigQuery Sink Connector

da966d6 Cherr-picked by #395 and #409: improve sources docs.

### Aws EventBridge Connector

\[fix] Fix getting wrong metadata value of `sequence_id` and `producer_name`


# V3.0.1.6
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.1.6



# StreamNative Weekly Release Notes v3.0.1.6

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.1.6](https://github.com/streamnative/pulsar/releases/tag/v3.0.1.6)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.1.6/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.1.6/images/sha256-aa9bb84c0c5ead9d26726af5bb375b9e028c10843ec4fa710775a646b2ab46f6)

## General Changes

### Apache Pulsar

\[fix]\[broker] \[fix]\[broker] Ignore individual acknowledgment for CompactorSubscription when an entry has been filtered

\[fix]\[broker] Fix MultiRoles token provider NPE when using anonymous clients

\[fix] \[build] rename schema\_example.conf to schema\_example.json

ca67fe950c Revert "Release 3.0.2"

\[fix] \[build] Fix in-correct license definetion

\[fix]\[proxy] Move status endpoint out of auth coverage

\[fix]\[sec] Upgrade snappy-java to 1.1.10.5

\[feat]\[meta] Upgrade to jetcd to 0.7.5

\[fix]\[sec] Bump avro version to 1.11.3 for CVE-2023-39410

### MoP

3d350a1 Exclude grpc dependency in the test module

### KoP

\[fix]\[schema-registry] Fix conflict schema version

### StreamNative Pulsar Plugins

\[audit-log]\[fix] Fix audit log producer cache

### Function Mesh Worker Service

Fix state store and add tests for oxia state store

Bump function-mesh to v0.18.0

766b9d48 Bump function mesh to v0.17.0


# V3.0.1.7
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.1.7



# StreamNative Weekly Release Notes v3.0.1.7

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.1.7](https://github.com/streamnative/pulsar/releases/tag/v3.0.1.7)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.1.7/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.1.7/images/sha256-91ecfc16909ccae05252d434e8e3cd7f395f83f6de821539cccec975538ad242)

## General Changes

### Apache Pulsar

\[fix]\[broker] Fix resource\_quota\_zpath

\[fix]\[broker] Fix setReplicatedSubscriptionStatus incorrect behavior

\[improve] \[broker] Let the producer request success at the first time if the previous one is inactive

\[fix]\[broker] Correct schema deletion for parititioned topic

\[fix] \[broker] Delete topic timeout due to NPE

\[fix]\[broker] Fix issue with consumer read uncommitted messages from compacted topic

\[fix]\[broker] Duplicate LedgerOffloader creation when namespace/topic…

dc198df524 Revert "Release 3.0.2"

\[fix]\[broker] Fix create topic with different auto creation strategies causes race condition

\[fix]\[broker] Fix namespace bundle stuck in unloading status

\[fix] \[build] Remove test testNoOrphanTopicIfInitFailed

([#21270)](https://github.com/apache/pulsar/pull/21270))) Revert "\[fix]\[client] Avert extensive time consumption during table view construction

\[fix]\[client] Fix print error log 'Auto getting partitions failed' when expend partition.

\[fix] \[broker] Fix thousands orphan PersistentTopic caused OOM

\[fix] \[ml] Fix orphan scheduled task for ledger create timeout check

\[fix]\[broker] Fix failure while creating non-durable cursor with inactive managed-ledger

\[fix]\[ml] Fix unfinished callback when deleting managed ledger

\[fix]\[client] Avert extensive time consumption during table view construction

\[fix]\[broker] Fix the deadlock when using BookieRackAffinityMapping with rackaware policy

\[fix]\[broker] Fix PulsarService/BrokerService shutdown when brokerShutdownTimeoutMs=0

\[fix]\[txn] OpRequestSend reuse problem cause tbClient commitTxnOnTopic timeout unexpectedly

\[fix]\[broker] Avoid pass null role in MultiRolesTokenAuthorizationProvider

\[fix]\[txn] Ack all message ids when ack chunk messages with transaction.

\[fix]\[build] Fix apt download issue in building the docker image

\[fix] \[broker] Make the new exclusive consumer instead the inactive one faster

### KoP

Fix NPE for OffsetsFetch v7 or earlier requests

Apply restrict checkstyle rules for indents and spaces

\[Branch-3.0]\[transaction] Make the list offset request aware of the read-committed isolation level

\[schema-registry]\[fix] Fix schema ID generation logic

\[fix] Use log end offset as the earliest offset when topic is empty

### Cloud Storage Connector

Optimize Azure blob storage connector config validation logic

Add azure blob storage sink connector docs.

Add Google Cloud Storage sink connector docs.

Add AWS s3 sink connector docs.

### AWS Lambda Connector

Make AWS Lambda sink connector private

### StreamNative Pulsar Plugins

Fix cve

### Function Mesh Worker Service

Fallback to reason field if the lastState's message is empty

make memory padding configurable

Fix getSinkList and getSourceList impl to avoid showing fields details.

67e808c0 Use local registry for sn-java and generic runner images

Support read customize connector catalogs.

Support generic runtime

Append version to the description field of ConnectorDefinition for load from connector catalog

### Lakehouse Connector

Update snappy dependency

### Snowflake Connector

Improve snowflake sink connector doc

Fix wrong value for configuration `processingGuarantees`

### Aws EventBridge Connector

Fix some docs and deprecated eventBusResourceName config.


# V3.0.1.8
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.1.8



# StreamNative Weekly Release Notes v3.0.1.8

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.1.8](https://github.com/streamnative/pulsar/releases/tag/v3.0.1.8)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.1.8/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.1.8/images/sha256-7360df5ef30913a5c26268600e0c6f0983e701e47f6a228370b047a14141af62)

## General Changes

### Apache Pulsar

f87e657bc0 Set project version 3.0.3-SNAPSHOT

\[fix]\[sec] Upgrade Bouncycastle to 1.75 to address CVE-2023-33201

\[improve]\[build] Upgrade Apache ZooKeeper to 3.9.1

\[fix]\[sec] Upgrade rabbitmq client to address CVE-2023-46120

\[fix] \[log] fix the vague response if topic not found

### AoP

Add rabbitmq amqp-client dependency for test

### MoP

fe09c41 Fixed EventCenterTest by disabling topic level policies

### Cloud Storage Connector

Update permission describe for AWS S3.

### AWS SQS Connector

Try fix auto labeling.

### AWS Lambda Connector

34e026f Fix auto label bot not work.

### pulsarctl

Update golang.org/x/net

### StreamNative Pulsar Plugins

Remove shaded protobuf for hadoop-common also from pulsar-tools

Fix the metadata tool CI

Removed shaded Protobuf dependency from hadoop

1dc2e621 Fixed more Go x/net version update

Fix the dependency conflict with pulsar broker

Fix detector go dep

a5d45ac5 Fixed go mod tidy

Update aws-java-sdk

Update go dependencies to fix CVEs

### Google Pub / Sub Connector

Fix auto label bot not work.

### Google BigQuery Sink Connector

Fix auto label bot not work.

### Aws EventBridge Connector

Fix auto label bot not work.


# V3.0.10.5
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.10.5



## StreamNative Weekly Release Notes v3.0.10.5

#### General Changes

### Apache Pulsar

([#24317](https://github.com/apache/pulsar/pull/24317)) \[fix]\[io]\[branch-3.0]Pulsar-SQL: Fix classcast ex when decode decimal value

([#24313](https://github.com/apache/pulsar/pull/24313)) \[fix]\[broker] Fix potential deadlock when creating partitioned topic

([#24293](https://github.com/apache/pulsar/pull/24293)) \[fix]\[broker] fix wrong method name checkTopicExists.

([#24307](https://github.com/apache/pulsar/pull/24307)) \[fix]\[build] Ensure that buildtools is Java 8 compatible and fix remaining compatibility issue

([#24304](https://github.com/apache/pulsar/pull/24304)) \[fix]\[test] Simplify BetweenTestClassesListenerAdapter and fix issue with BeforeTest/AfterTest annotations

([#24289](https://github.com/apache/pulsar/pull/24289)) \[improve]\[io] Add configuration parameter for disabling aggregation for Kinesis Producers

([#24302](https://github.com/apache/pulsar/pull/24302)) \[improve] Upgrade pulsar-client-python to 3.7.0 in Docker image

([#24299](https://github.com/apache/pulsar/pull/24299)) \[fix]\[test] Fix more Netty ByteBuf leaks in tests

([#24297](https://github.com/apache/pulsar/pull/24297)) \[fix]\[io] Fix SyntaxWarning in Pulsar Python functions

([#24282](https://github.com/apache/pulsar/pull/24282)) \[fix]\[client] Fix producer publishing getting stuck after message with incompatible schema is discarded

([#24283](https://github.com/apache/pulsar/pull/24283)) \[cleanup]\[test] Remove unused parameter from deleteNamespaceWithRetry method in MockedPulsarServiceBaseTest

([#24263](https://github.com/apache/pulsar/pull/24263)) \[improve]\[build] Upgrade zstd version from 1.5.2-3 to 1.5.7-3

([#24281](https://github.com/apache/pulsar/pull/24281)) \[fix]\[test] Fix multiple ByteBuf leaks in tests

([#24275](https://github.com/apache/pulsar/pull/24275)) \[fix]\[broker] Fix HashedWheelTimer leak in PulsarService by stopping it in shutdown

([#24274](https://github.com/apache/pulsar/pull/24274)) \[fix]\[misc] Fix ByteBuf leak in SchemaUtils

([#24254](https://github.com/apache/pulsar/pull/24254)) \[fix]\[broker]Fix incorrect priority between topic policies and global topic policies

([#24266](https://github.com/apache/pulsar/pull/24266)) \[improve]\[ci] Disable detailed console logging for integration tests in CI

([#24261](https://github.com/apache/pulsar/pull/24261)) \[fix]\[test] Fix flaky ManagedCursorTest.testLastActiveAfterResetCursor and disable failing SchemaTest

([#24244](https://github.com/apache/pulsar/pull/24244)) \[fix]\[test] Fix flaky ManagedCursorTest.testSkipEntriesWithIndividualDeletedMessages

([#24248](https://github.com/apache/pulsar/pull/24248)) \[improve]\[io]\[kca] support fully-qualified topic names in source records

([#24260](https://github.com/apache/pulsar/pull/24260)) \[improve]\[build] Upgrade Gradle Develocity Maven Extension dependencies

([#24258](https://github.com/apache/pulsar/pull/24258)) \[fix]\[test] Fix TestNG BetweenTestClassesListenerAdapter listener

([#24257](https://github.com/apache/pulsar/pull/24257)) \[fix]\[broker] Unregister non-static metrics collectors registered in Prometheus default registry

bebc3b0d6a Fix checkstyle issue in previous cherry-pick c2d33cc

([#24178](https://github.com/apache/pulsar/pull/24178)) \[fix]\[broker]fix memory leak, messages lost, incorrect replication state if using multiple schema versions(auto\_produce)

([#24219](https://github.com/apache/pulsar/pull/24219)) \[improve]\[broker]Improve the feature "Optimize subscription seek (cursor reset) by timestamp": search less entries

([#23919](https://github.com/apache/pulsar/pull/23919)) \[fix]\[broker] Fix seeking by timestamp can be reset the cursor position to earliest

([#22792](https://github.com/apache/pulsar/pull/22792)) \[improve]\[broker] Optimize subscription seek (cursor reset) by timestamp

([#24243](https://github.com/apache/pulsar/pull/24243)) \[improve]\[build] Upgrade SpotBugs to 4.9.x

([#24240](https://github.com/apache/pulsar/pull/24240)) \[improve]\[build] Upgrade to jacoco 0.8.13

([#24237](https://github.com/apache/pulsar/pull/24237)) \[improve]\[build] Upgrade Lombok to 1.18.38 to support JDK 24

([#24221](https://github.com/apache/pulsar/pull/24221)) \[improve]\[io] support kafka connect transforms and predicates

([#24230](https://github.com/apache/pulsar/pull/24230)) \[improve]\[client]Improve transaction log when a TXN command timeout

([#24223](https://github.com/apache/pulsar/pull/24223)) \[fix]\[broker] Orphan schema after disabled a cluster for a namespace

([#24228](https://github.com/apache/pulsar/pull/24228)) \[fix]\[broker] Fix ByteBuf memory leak in REST API for publishing messages

b4e1c93d2c Fix presto-distribution/LICENSE

([#24184](https://github.com/apache/pulsar/pull/24184)) \[fix]\[client] Fix incorrect producer.getPendingQueueSize due to incomplete queue implementation

([#24214](https://github.com/apache/pulsar/pull/24214)) \[improve] Upgrade Netty to 4.1.121.Final

([#24212](https://github.com/apache/pulsar/pull/24212)) \[fix]\[test] Fix flaky BatchMessageWithBatchIndexLevelTest.testBatchMessageAck

([#24218](https://github.com/apache/pulsar/pull/24218)) \[fix]\[test] Fix multiple resource leaks in tests

([#24187](https://github.com/apache/pulsar/pull/24187)) \[improve]\[client] validate ClientConfigurationData earlier to avoid resource leaks

([#24216](https://github.com/apache/pulsar/pull/24216)) \[fix]\[broker] Fix HealthChecker deadlock in shutdown

([#24209](https://github.com/apache/pulsar/pull/24209)) \[fix]\[broker] Fix tenant creation and update with null value

([#24192](https://github.com/apache/pulsar/pull/24192)) \[fix]\[admin] Backlog quota's policy is null which causes a NPE

([#24210](https://github.com/apache/pulsar/pull/24210)) \[fix]\[broker] Fix broker shutdown delay by resolving hanging health checks

([#24207](https://github.com/apache/pulsar/pull/24207)) \[fix]\[broker] Fix compaction service log's wrong condition

([#24204](https://github.com/apache/pulsar/pull/24204)) \[fix]\[test] Fix resource leaks in ProxyTest and fix invalid tests

([#24201](https://github.com/apache/pulsar/pull/24201)) \[improve]\[io] Upgrade Kafka client and compatible Confluent platform version

([#24118)](https://github.com/apache/pulsar/pull/24118))) Revert "\[fix]\[broker] Add topic consistency check

([#24154)](https://github.com/apache/pulsar/pull/24154))) Revert "\[fix]\[broker] Directly query single topic existence when the topic is partitioned

([#24032](https://github.com/apache/pulsar/pull/24032)) \[fix]\[broker] Fix missing validation when setting retention policy on topic level

([#24098](https://github.com/apache/pulsar/pull/24098)) \[fix]\[ml] Skip deleting cursor if it was already deleted before calling unsubscribe

([#24181](https://github.com/apache/pulsar/pull/24181)) \[fix]\[proxy] Fix incorrect client error when calling get topic metadata

([#24158](https://github.com/apache/pulsar/pull/24158)) \[fix]\[proxy] Propagate client connection feature flags through Pulsar Proxy to Broker

([#24103](https://github.com/apache/pulsar/pull/24103)) \[fix]\[schema] Reject unsupported Avro schema types during schema registration

([#24091](https://github.com/apache/pulsar/pull/24091)) \[fix]\[broker] Fix some problems in calculate totalAvailableBookies in method getExcludedBookiesWithIsolationGroups when some bookies belongs to multiple isolation groups.

([#21320](https://github.com/apache/pulsar/pull/21320)) \[fix]\[bk] Fix the var name for IsolationGroups

([#24171](https://github.com/apache/pulsar/pull/24171)) \[improve]\[test] Use configured session timeout for MockZooKeeper and TestZKServer in PulsarTestContext

([#24172](https://github.com/apache/pulsar/pull/24172)) \[fix]\[test] Improve reliability of IncrementPartitionsTest

([#24170](https://github.com/apache/pulsar/pull/24170)) \[fix]\[test]flaky-test:ManagedLedgerInterceptorImplTest.testManagedLedgerPayloadInputProcessorFailure

([#23980](https://github.com/apache/pulsar/pull/23980)) \[fix]\[broker] Consumer stuck when delete subscription \_\_compaction failed

([#24167](https://github.com/apache/pulsar/pull/24167)) \[fix]\[ml] Fix ML thread blocking issue in internalGetPartitionedStats API

([#24166](https://github.com/apache/pulsar/pull/24166)) \[fix]\[test] Fix invalid test CompactionTest.testDeleteCompactedLedgerWithSlowAck

([#24150](https://github.com/apache/pulsar/pull/24150)) \[fix]\[broker] The feature brokerDeleteInactivePartitionedTopicMetadataEnabled leaves orphan topic policies and topic schemas

([#24154](https://github.com/apache/pulsar/pull/24154)) \[fix]\[broker] Directly query single topic existence when the topic is partitioned

([#24118](https://github.com/apache/pulsar/pull/24118)) \[fix]\[broker] Add topic consistency check

([#24056](https://github.com/apache/pulsar/pull/24056)) \[fix]\[test] Update partitioned topic subscription assertions in IncrementPartitionsTest

([#24033](https://github.com/apache/pulsar/pull/24033)) \[cleanup]\[misc] Add override annotation

([#24161](https://github.com/apache/pulsar/pull/24161)) \[fix]\[test] Fix flaky BrokerServiceChaosTest.testFetchPartitionedTopicMetadataWithCacheRefresh

([#24162](https://github.com/apache/pulsar/pull/24162)) \[fix]\[test] Fix flaky BrokerServiceChaosTest

1035accffd Bump version to next snapshot version

([#24097](https://github.com/apache/pulsar/pull/24097)) \[fix] \[broker] topics infinitely failed to delete after remove cluster from replicated clusters modifying when using partitioned system topic

([#22261](https://github.com/apache/pulsar/pull/22261)) \[fix] Bump google.golang.org/protobuf from 1.32.0 to 1.33.0 in /pulsar-function-go

([#24132](https://github.com/apache/pulsar/pull/24132)) \[fix]\[io] Fix KinesisSink json flattening for AVRO's SchemaType.BYTES

([#20984](https://github.com/apache/pulsar/pull/20984)) \[fix]\[broker] Fix get outdated compactedTopicContext after compactionHorizon has been updated

([#20697](https://github.com/apache/pulsar/pull/20697)) \[improve]\[broker] Improve CompactedTopicImpl lock

([#24131](https://github.com/apache/pulsar/pull/24131)) \[fix]\[ml] Return 1 when bytes size is 0 or negative for entry count estimation

([#24128](https://github.com/apache/pulsar/pull/24128)) \[improve]\[io] Enhance Kafka connector logging with focused bootstrap server information

([#24125](https://github.com/apache/pulsar/pull/24125)) \[fix]\[ml] Don't estimate number of entries when ledgers are empty, return 1 instead

([#24123](https://github.com/apache/pulsar/pull/24123)) \[improve]\[client] Prevent NullPointException when closing ClientCredentialsFlow

([#24124](https://github.com/apache/pulsar/pull/24124)) \[improve]\[io] Remove sleep when sourceTask.poll of kafka return null

([#24116](https://github.com/apache/pulsar/pull/24116)) \[improve]\[broker] Change topic exists log to warn

([#24104](https://github.com/apache/pulsar/pull/24104)) \[fix]\[client] Pattern subscription regression when broker-side evaluation is disabled

([#24100](https://github.com/apache/pulsar/pull/24100)) \[fix]\[client] Fix consumer leak when thread is interrupted before subscribe completes

([#24089](https://github.com/apache/pulsar/pull/24089)) \[fix]\[ml] Fix issues in estimateEntryCountBySize

([#24073](https://github.com/apache/pulsar/pull/24073)) \[improve]\[broker] Optimize message expiration rate repeated update issues

([#24087](https://github.com/apache/pulsar/pull/24087)) \[fix]\[broker] Avoid IllegalStateException when marker\_type field is not set in publishing

([#24083](https://github.com/apache/pulsar/pull/24083)) \[fix]\[ci] Bump dependency-check to 12.1.0 to fix OWASP Dependency Check job

([#24082](https://github.com/apache/pulsar/pull/24082)) \[clean]\[client] Clean code for the construction of retry/dead letter topic name

([#24079](https://github.com/apache/pulsar/pull/24079)) \[fix]\[broker] Fix NPE while publishing Metadata-Event with not init producer

([#24080](https://github.com/apache/pulsar/pull/24080)) \[fix]\[broker] Fix Metadata event synchronizer should not fail with bad version

([#24081](https://github.com/apache/pulsar/pull/24081)) \[fix]\[broker] Fix Metadata Event Synchronizer producer creation retry so that the producer gets created eventually

([#24048](https://github.com/apache/pulsar/pull/24048)) \[fix]\[broker] Fix UnsupportedOperationException while setting subscription level dispatch rate policy

([#24054](https://github.com/apache/pulsar/pull/24054)) \[fix]\[ml] Corrected pulsar\_storage\_size metric to not multiply offloaded storage by the write quorum

([#24067](https://github.com/apache/pulsar/pull/24067)) \[fix]\[broker] http metric endpoint get compaction latency stats always be 0

([#24064](https://github.com/apache/pulsar/pull/24064)) \[improve]\[broker] Optimize ThresholdShedder with improved boundary checks and parameter reuse

([#24055](https://github.com/apache/pulsar/pull/24055)) \[fix] Avoid negative estimated entry count

([#24060](https://github.com/apache/pulsar/pull/24060)) \[improve]\[monitor] Add version=0.0.4 to /metrics content type for Prometheus 3.x compatibility

([#24059](https://github.com/apache/pulsar/pull/24059)) \[fix]\[client] Copy eventTime to retry letter topic and DLQ messages

([#24061](https://github.com/apache/pulsar/pull/24061)) \[fix]\[client] Fix building broken batched message when publishing

([#24063](https://github.com/apache/pulsar/pull/24063)) \[fix]\[broker]Fix failed consumption after loaded up a terminated topic

([#24072](https://github.com/apache/pulsar/pull/24072)) \[fix]\[broker] Pattern subscription doesn't work when the pattern excludes the topic domain.

ebce3b07ed Fix presto LICENSE after Netty 4.1.119.Final upgrade

([#24049](https://github.com/apache/pulsar/pull/24049)) \[improve] Upgrade Netty to 4.1.119.Final

([#23975](https://github.com/apache/pulsar/pull/23975)) \[fix]\[broker] Add expire check for replicator

([#24023](https://github.com/apache/pulsar/pull/24023)) \[fix]\[doc] fix doc related to chunk message feature.

8437af98eb Bump version to next snapshot version

([#23962](https://github.com/apache/pulsar/pull/23962)) \[improve]\[ml] Use lock-free queue in InflightReadsLimiter since there's no concurrent access

([#23978](https://github.com/apache/pulsar/pull/23978)) \[improve]\[cli] Support additional msg metadata for V1 topic on peek message cmd

([#24014](https://github.com/apache/pulsar/pull/24014)) \[fix]\[broker] Fix BucketDelayedDeliveryTracker thread safety

([#24019](https://github.com/apache/pulsar/pull/24019)) \[fix]\[test]Fix flaky test V1\_ProducerConsumerTest.testConcurrentConsumerReceiveWhileReconnect

([#24011](https://github.com/apache/pulsar/pull/24011)) \[fix]\[test] Fix flaky test OneWayReplicatorUsingGlobalZKTest.testConfigReplicationStartAt

([#23931](https://github.com/apache/pulsar/pull/23931)) \[improve] \[broker] Make the estimated entry size more accurate

([#24004](https://github.com/apache/pulsar/pull/24004)) \[improve]\[ci] Upgrade Gradle Develocity Maven Extension to 1.23.1

([#23697](https://github.com/apache/pulsar/pull/23697)) \[fix]\[broker] Geo Replication lost messages or frequently fails due to Deduplication is not appropriate for Geo-Replication

([#24006](https://github.com/apache/pulsar/pull/24006)) \[fix]\[broker] fix broker identifying incorrect stuck topic

([#23286](https://github.com/apache/pulsar/pull/23286)) \[improve]\[broker] Fix non-persistent system topic schema compatibility

([#23881](https://github.com/apache/pulsar/pull/23881)) \[improve]\[fn] Set default tenant and namespace for ListFunctions cmd

([#23730](https://github.com/apache/pulsar/pull/23730)) \[fix]\[admin] Verify is policies read only before revoke permissions on topic

([#24003](https://github.com/apache/pulsar/pull/24003)) \[improve]\[test] Upgrade Testcontainers to 1.20.4 and docker-java to 3.4.0

### KoP

([#1230](https://github.com/streamnative/sn-kop/pull/1230)) Change log level from error to info for successful authorization in SimpleAclAuthorizer

### Function Mesh Worker Service

Fix resource error during update and get connectors

Fix trigger function not support partitioned input topics error

Find specified ServiceAccount using oauth2's client role and use it when exist

validate function-mesh v0.24.1

exclude lz4-java for CVE reasons


# V3.0.10.6
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.10.6



## StreamNative Weekly Release Notes v3.0.10.6

#### General Changes

### Apache Pulsar

([#23594](https://github.com/apache/pulsar/pull/23594)) \[fix] \[broker] No longer allow creating subscription that contains slash

232f0ef492 remove unnecessary codes

([#24366](https://github.com/apache/pulsar/pull/24366)) \[fix]\[broker]Fix deadlock when compaction and topic deletion execute concurrently

([#24350](https://github.com/apache/pulsar/pull/24350)) \[fix]\[broker] Fix issue that topic policies was deleted after a sub topic deleted, even if the partitioned topic still exists

([#24384](https://github.com/apache/pulsar/pull/24384)) \[fix]\[ml]Revert a behavior change of releasing idle offloaded ledger handle: only release idle BlobStoreBackedReadHandle

([#24397](https://github.com/apache/pulsar/pull/24397)) \[improve]\[misc] Upgrade Netty to 4.1.122.Final and tcnative to 2.0.72.Final

([#24391](https://github.com/apache/pulsar/pull/24391)) \[improve]\[broker] Add managedCursor/LedgerInfoCompressionType settings to broker.conf

([#24392](https://github.com/apache/pulsar/pull/24392)) \[improve]\[broker] Make maxBatchDeletedIndexToPersist configurable and document other related configs

([#24386](https://github.com/apache/pulsar/pull/24386)) \[improve]\[broker] Added synchronized for sendMessages in Non-Persistent message dispatchers

([#24381](https://github.com/apache/pulsar/pull/24381)) \[improve]\[ml]Release idle offloaded read handle only the ref count is 0

([#19783](https://github.com/apache/pulsar/pull/19783)) \[improve]\[offloaders] Automatically evict Offloaded Ledgers from memory

([#24360](https://github.com/apache/pulsar/pull/24360)) \[fix]\[broker] expose consumer name for partitioned topic stats

([#24359](https://github.com/apache/pulsar/pull/24359)) \[improve]\[broker]Improve the log when encountered in-flight read limitation

([#24354](https://github.com/apache/pulsar/pull/24354)) \[fix]\[io] Acknowledge RabbitMQ message after processing the message successfully

([#24352](https://github.com/apache/pulsar/pull/24352)) \[fix]\[broker] Ignore metadata changes when broker is not in the Started state

([#24190](https://github.com/apache/pulsar/pull/24190)) \[fix]\[broker] Resolve the issue of frequent updates in message expiration deletion rate

([#24338](https://github.com/apache/pulsar/pull/24338)) \[fix]\[ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent issue

([#24331](https://github.com/apache/pulsar/pull/24331)) \[fix]\[offload] Complete the future outside of the reading loop in BlobStoreBackedReadHandleImplV2.readAsync

([#24324](https://github.com/apache/pulsar/pull/24324)) \[fix]\[test] Fix flaky AutoScaledReceiverQueueSizeTest.testNegativeClientMemory

([#24316](https://github.com/apache/pulsar/pull/24316)) \[fix]\[io] Fix kinesis avro bytes handling

a75d16fa6e Fix checkstyle errors after previous cherry-picks

([#24344](https://github.com/apache/pulsar/pull/24344)) \[improve]\[ml] Offload ledgers without check ledger length

([#24286](https://github.com/apache/pulsar/pull/24286)) \[fix]\[broker]Non-global topic policies and global topic policies overwrite each other

([#24279](https://github.com/apache/pulsar/pull/24279)) \[fix]\[broker]Global topic policies do not affect after unloading topic and persistence global topic policies never affect

([#24349](https://github.com/apache/pulsar/pull/24349)) \[fix]\[io]\[branch-3.0] Backport Kinesis Sink custom native executable support #23762

([#24317](https://github.com/apache/pulsar/pull/24317)) \[fix]\[io]\[branch-3.0]Pulsar-SQL: Fix classcast ex when decode decimal value

([#24313](https://github.com/apache/pulsar/pull/24313)) \[fix]\[broker] Fix potential deadlock when creating partitioned topic

([#24293](https://github.com/apache/pulsar/pull/24293)) \[fix]\[broker] fix wrong method name checkTopicExists.

([#24307](https://github.com/apache/pulsar/pull/24307)) \[fix]\[build] Ensure that buildtools is Java 8 compatible and fix remaining compatibility issue

([#24304](https://github.com/apache/pulsar/pull/24304)) \[fix]\[test] Simplify BetweenTestClassesListenerAdapter and fix issue with BeforeTest/AfterTest annotations

([#24289](https://github.com/apache/pulsar/pull/24289)) \[improve]\[io] Add configuration parameter for disabling aggregation for Kinesis Producers

([#24302](https://github.com/apache/pulsar/pull/24302)) \[improve] Upgrade pulsar-client-python to 3.7.0 in Docker image

([#24299](https://github.com/apache/pulsar/pull/24299)) \[fix]\[test] Fix more Netty ByteBuf leaks in tests

([#24297](https://github.com/apache/pulsar/pull/24297)) \[fix]\[io] Fix SyntaxWarning in Pulsar Python functions

([#24282](https://github.com/apache/pulsar/pull/24282)) \[fix]\[client] Fix producer publishing getting stuck after message with incompatible schema is discarded

([#24283](https://github.com/apache/pulsar/pull/24283)) \[cleanup]\[test] Remove unused parameter from deleteNamespaceWithRetry method in MockedPulsarServiceBaseTest

([#24263](https://github.com/apache/pulsar/pull/24263)) \[improve]\[build] Upgrade zstd version from 1.5.2-3 to 1.5.7-3

([#24281](https://github.com/apache/pulsar/pull/24281)) \[fix]\[test] Fix multiple ByteBuf leaks in tests

([#24275](https://github.com/apache/pulsar/pull/24275)) \[fix]\[broker] Fix HashedWheelTimer leak in PulsarService by stopping it in shutdown

([#24274](https://github.com/apache/pulsar/pull/24274)) \[fix]\[misc] Fix ByteBuf leak in SchemaUtils

([#24254](https://github.com/apache/pulsar/pull/24254)) \[fix]\[broker]Fix incorrect priority between topic policies and global topic policies

([#24266](https://github.com/apache/pulsar/pull/24266)) \[improve]\[ci] Disable detailed console logging for integration tests in CI

([#24261](https://github.com/apache/pulsar/pull/24261)) \[fix]\[test] Fix flaky ManagedCursorTest.testLastActiveAfterResetCursor and disable failing SchemaTest

([#24244](https://github.com/apache/pulsar/pull/24244)) \[fix]\[test] Fix flaky ManagedCursorTest.testSkipEntriesWithIndividualDeletedMessages

([#24248](https://github.com/apache/pulsar/pull/24248)) \[improve]\[io]\[kca] support fully-qualified topic names in source records

([#24260](https://github.com/apache/pulsar/pull/24260)) \[improve]\[build] Upgrade Gradle Develocity Maven Extension dependencies

([#24258](https://github.com/apache/pulsar/pull/24258)) \[fix]\[test] Fix TestNG BetweenTestClassesListenerAdapter listener

([#24257](https://github.com/apache/pulsar/pull/24257)) \[fix]\[broker] Unregister non-static metrics collectors registered in Prometheus default registry

bebc3b0d6a Fix checkstyle issue in previous cherry-pick c2d33cc

([#24178](https://github.com/apache/pulsar/pull/24178)) \[fix]\[broker]fix memory leak, messages lost, incorrect replication state if using multiple schema versions(auto\_produce)

([#24219](https://github.com/apache/pulsar/pull/24219)) \[improve]\[broker]Improve the feature "Optimize subscription seek (cursor reset) by timestamp": search less entries

([#23919](https://github.com/apache/pulsar/pull/23919)) \[fix]\[broker] Fix seeking by timestamp can be reset the cursor position to earliest

([#22792](https://github.com/apache/pulsar/pull/22792)) \[improve]\[broker] Optimize subscription seek (cursor reset) by timestamp

([#24243](https://github.com/apache/pulsar/pull/24243)) \[improve]\[build] Upgrade SpotBugs to 4.9.x

([#24240](https://github.com/apache/pulsar/pull/24240)) \[improve]\[build] Upgrade to jacoco 0.8.13

([#24237](https://github.com/apache/pulsar/pull/24237)) \[improve]\[build] Upgrade Lombok to 1.18.38 to support JDK 24

([#24221](https://github.com/apache/pulsar/pull/24221)) \[improve]\[io] support kafka connect transforms and predicates

([#24230](https://github.com/apache/pulsar/pull/24230)) \[improve]\[client]Improve transaction log when a TXN command timeout

([#24223](https://github.com/apache/pulsar/pull/24223)) \[fix]\[broker] Orphan schema after disabled a cluster for a namespace

([#24228](https://github.com/apache/pulsar/pull/24228)) \[fix]\[broker] Fix ByteBuf memory leak in REST API for publishing messages

b4e1c93d2c Fix presto-distribution/LICENSE

([#24184](https://github.com/apache/pulsar/pull/24184)) \[fix]\[client] Fix incorrect producer.getPendingQueueSize due to incomplete queue implementation

([#24214](https://github.com/apache/pulsar/pull/24214)) \[improve] Upgrade Netty to 4.1.121.Final

([#24212](https://github.com/apache/pulsar/pull/24212)) \[fix]\[test] Fix flaky BatchMessageWithBatchIndexLevelTest.testBatchMessageAck

([#24218](https://github.com/apache/pulsar/pull/24218)) \[fix]\[test] Fix multiple resource leaks in tests

([#24187](https://github.com/apache/pulsar/pull/24187)) \[improve]\[client] validate ClientConfigurationData earlier to avoid resource leaks

([#24216](https://github.com/apache/pulsar/pull/24216)) \[fix]\[broker] Fix HealthChecker deadlock in shutdown

([#24209](https://github.com/apache/pulsar/pull/24209)) \[fix]\[broker] Fix tenant creation and update with null value

([#24192](https://github.com/apache/pulsar/pull/24192)) \[fix]\[admin] Backlog quota's policy is null which causes a NPE

([#24210](https://github.com/apache/pulsar/pull/24210)) \[fix]\[broker] Fix broker shutdown delay by resolving hanging health checks

([#24207](https://github.com/apache/pulsar/pull/24207)) \[fix]\[broker] Fix compaction service log's wrong condition

([#24204](https://github.com/apache/pulsar/pull/24204)) \[fix]\[test] Fix resource leaks in ProxyTest and fix invalid tests

([#24201](https://github.com/apache/pulsar/pull/24201)) \[improve]\[io] Upgrade Kafka client and compatible Confluent platform version

([#24118)](https://github.com/apache/pulsar/pull/24118))) Revert "\[fix]\[broker] Add topic consistency check

([#24154)](https://github.com/apache/pulsar/pull/24154))) Revert "\[fix]\[broker] Directly query single topic existence when the topic is partitioned

([#24032](https://github.com/apache/pulsar/pull/24032)) \[fix]\[broker] Fix missing validation when setting retention policy on topic level

([#24098](https://github.com/apache/pulsar/pull/24098)) \[fix]\[ml] Skip deleting cursor if it was already deleted before calling unsubscribe

([#24181](https://github.com/apache/pulsar/pull/24181)) \[fix]\[proxy] Fix incorrect client error when calling get topic metadata

([#24158](https://github.com/apache/pulsar/pull/24158)) \[fix]\[proxy] Propagate client connection feature flags through Pulsar Proxy to Broker

([#24103](https://github.com/apache/pulsar/pull/24103)) \[fix]\[schema] Reject unsupported Avro schema types during schema registration

([#24091](https://github.com/apache/pulsar/pull/24091)) \[fix]\[broker] Fix some problems in calculate totalAvailableBookies in method getExcludedBookiesWithIsolationGroups when some bookies belongs to multiple isolation groups.

([#21320](https://github.com/apache/pulsar/pull/21320)) \[fix]\[bk] Fix the var name for IsolationGroups

([#24171](https://github.com/apache/pulsar/pull/24171)) \[improve]\[test] Use configured session timeout for MockZooKeeper and TestZKServer in PulsarTestContext

([#24172](https://github.com/apache/pulsar/pull/24172)) \[fix]\[test] Improve reliability of IncrementPartitionsTest

([#24170](https://github.com/apache/pulsar/pull/24170)) \[fix]\[test]flaky-test:ManagedLedgerInterceptorImplTest.testManagedLedgerPayloadInputProcessorFailure

([#23980](https://github.com/apache/pulsar/pull/23980)) \[fix]\[broker] Consumer stuck when delete subscription \_\_compaction failed

([#24167](https://github.com/apache/pulsar/pull/24167)) \[fix]\[ml] Fix ML thread blocking issue in internalGetPartitionedStats API

([#24166](https://github.com/apache/pulsar/pull/24166)) \[fix]\[test] Fix invalid test CompactionTest.testDeleteCompactedLedgerWithSlowAck

([#24150](https://github.com/apache/pulsar/pull/24150)) \[fix]\[broker] The feature brokerDeleteInactivePartitionedTopicMetadataEnabled leaves orphan topic policies and topic schemas

([#24154](https://github.com/apache/pulsar/pull/24154)) \[fix]\[broker] Directly query single topic existence when the topic is partitioned

([#24118](https://github.com/apache/pulsar/pull/24118)) \[fix]\[broker] Add topic consistency check

([#24056](https://github.com/apache/pulsar/pull/24056)) \[fix]\[test] Update partitioned topic subscription assertions in IncrementPartitionsTest

([#24033](https://github.com/apache/pulsar/pull/24033)) \[cleanup]\[misc] Add override annotation

([#24161](https://github.com/apache/pulsar/pull/24161)) \[fix]\[test] Fix flaky BrokerServiceChaosTest.testFetchPartitionedTopicMetadataWithCacheRefresh

([#24162](https://github.com/apache/pulsar/pull/24162)) \[fix]\[test] Fix flaky BrokerServiceChaosTest

1035accffd Bump version to next snapshot version

([#24097](https://github.com/apache/pulsar/pull/24097)) \[fix] \[broker] topics infinitely failed to delete after remove cluster from replicated clusters modifying when using partitioned system topic

([#22261](https://github.com/apache/pulsar/pull/22261)) \[fix] Bump google.golang.org/protobuf from 1.32.0 to 1.33.0 in /pulsar-function-go

([#24132](https://github.com/apache/pulsar/pull/24132)) \[fix]\[io] Fix KinesisSink json flattening for AVRO's SchemaType.BYTES

([#20984](https://github.com/apache/pulsar/pull/20984)) \[fix]\[broker] Fix get outdated compactedTopicContext after compactionHorizon has been updated

([#20697](https://github.com/apache/pulsar/pull/20697)) \[improve]\[broker] Improve CompactedTopicImpl lock

([#24131](https://github.com/apache/pulsar/pull/24131)) \[fix]\[ml] Return 1 when bytes size is 0 or negative for entry count estimation

([#24128](https://github.com/apache/pulsar/pull/24128)) \[improve]\[io] Enhance Kafka connector logging with focused bootstrap server information

([#24125](https://github.com/apache/pulsar/pull/24125)) \[fix]\[ml] Don't estimate number of entries when ledgers are empty, return 1 instead

([#24123](https://github.com/apache/pulsar/pull/24123)) \[improve]\[client] Prevent NullPointException when closing ClientCredentialsFlow

([#24124](https://github.com/apache/pulsar/pull/24124)) \[improve]\[io] Remove sleep when sourceTask.poll of kafka return null

([#24116](https://github.com/apache/pulsar/pull/24116)) \[improve]\[broker] Change topic exists log to warn

([#24104](https://github.com/apache/pulsar/pull/24104)) \[fix]\[client] Pattern subscription regression when broker-side evaluation is disabled

([#24100](https://github.com/apache/pulsar/pull/24100)) \[fix]\[client] Fix consumer leak when thread is interrupted before subscribe completes

([#24089](https://github.com/apache/pulsar/pull/24089)) \[fix]\[ml] Fix issues in estimateEntryCountBySize

([#24073](https://github.com/apache/pulsar/pull/24073)) \[improve]\[broker] Optimize message expiration rate repeated update issues

([#24087](https://github.com/apache/pulsar/pull/24087)) \[fix]\[broker] Avoid IllegalStateException when marker\_type field is not set in publishing

([#24083](https://github.com/apache/pulsar/pull/24083)) \[fix]\[ci] Bump dependency-check to 12.1.0 to fix OWASP Dependency Check job

([#24082](https://github.com/apache/pulsar/pull/24082)) \[clean]\[client] Clean code for the construction of retry/dead letter topic name

([#24079](https://github.com/apache/pulsar/pull/24079)) \[fix]\[broker] Fix NPE while publishing Metadata-Event with not init producer

([#24080](https://github.com/apache/pulsar/pull/24080)) \[fix]\[broker] Fix Metadata event synchronizer should not fail with bad version

([#24081](https://github.com/apache/pulsar/pull/24081)) \[fix]\[broker] Fix Metadata Event Synchronizer producer creation retry so that the producer gets created eventually

([#24048](https://github.com/apache/pulsar/pull/24048)) \[fix]\[broker] Fix UnsupportedOperationException while setting subscription level dispatch rate policy

([#24054](https://github.com/apache/pulsar/pull/24054)) \[fix]\[ml] Corrected pulsar\_storage\_size metric to not multiply offloaded storage by the write quorum

([#24067](https://github.com/apache/pulsar/pull/24067)) \[fix]\[broker] http metric endpoint get compaction latency stats always be 0

([#24064](https://github.com/apache/pulsar/pull/24064)) \[improve]\[broker] Optimize ThresholdShedder with improved boundary checks and parameter reuse

([#24055](https://github.com/apache/pulsar/pull/24055)) \[fix] Avoid negative estimated entry count

([#24060](https://github.com/apache/pulsar/pull/24060)) \[improve]\[monitor] Add version=0.0.4 to /metrics content type for Prometheus 3.x compatibility

([#24059](https://github.com/apache/pulsar/pull/24059)) \[fix]\[client] Copy eventTime to retry letter topic and DLQ messages

([#24061](https://github.com/apache/pulsar/pull/24061)) \[fix]\[client] Fix building broken batched message when publishing

([#24063](https://github.com/apache/pulsar/pull/24063)) \[fix]\[broker]Fix failed consumption after loaded up a terminated topic

([#24072](https://github.com/apache/pulsar/pull/24072)) \[fix]\[broker] Pattern subscription doesn't work when the pattern excludes the topic domain.

ebce3b07ed Fix presto LICENSE after Netty 4.1.119.Final upgrade

([#24049](https://github.com/apache/pulsar/pull/24049)) \[improve] Upgrade Netty to 4.1.119.Final

([#23975](https://github.com/apache/pulsar/pull/23975)) \[fix]\[broker] Add expire check for replicator

([#24023](https://github.com/apache/pulsar/pull/24023)) \[fix]\[doc] fix doc related to chunk message feature.

8437af98eb Bump version to next snapshot version

([#23962](https://github.com/apache/pulsar/pull/23962)) \[improve]\[ml] Use lock-free queue in InflightReadsLimiter since there's no concurrent access

([#23978](https://github.com/apache/pulsar/pull/23978)) \[improve]\[cli] Support additional msg metadata for V1 topic on peek message cmd

([#24014](https://github.com/apache/pulsar/pull/24014)) \[fix]\[broker] Fix BucketDelayedDeliveryTracker thread safety

([#24019](https://github.com/apache/pulsar/pull/24019)) \[fix]\[test]Fix flaky test V1\_ProducerConsumerTest.testConcurrentConsumerReceiveWhileReconnect

([#24011](https://github.com/apache/pulsar/pull/24011)) \[fix]\[test] Fix flaky test OneWayReplicatorUsingGlobalZKTest.testConfigReplicationStartAt

([#23931](https://github.com/apache/pulsar/pull/23931)) \[improve] \[broker] Make the estimated entry size more accurate

([#24004](https://github.com/apache/pulsar/pull/24004)) \[improve]\[ci] Upgrade Gradle Develocity Maven Extension to 1.23.1

([#23697](https://github.com/apache/pulsar/pull/23697)) \[fix]\[broker] Geo Replication lost messages or frequently fails due to Deduplication is not appropriate for Geo-Replication

([#24006](https://github.com/apache/pulsar/pull/24006)) \[fix]\[broker] fix broker identifying incorrect stuck topic

([#23286](https://github.com/apache/pulsar/pull/23286)) \[improve]\[broker] Fix non-persistent system topic schema compatibility

([#23881](https://github.com/apache/pulsar/pull/23881)) \[improve]\[fn] Set default tenant and namespace for ListFunctions cmd

([#23730](https://github.com/apache/pulsar/pull/23730)) \[fix]\[admin] Verify is policies read only before revoke permissions on topic

([#24003](https://github.com/apache/pulsar/pull/24003)) \[improve]\[test] Upgrade Testcontainers to 1.20.4 and docker-java to 3.4.0

### KoP

\[refactor] Create PartitionLog only after ProducerStateManager#recover is done

### StreamNative Pulsar Plugins

Add test to verify the sts module

Fix the backup tool can not use sts to authenticate

### pulsarctl

e7d5e82 Use snstage docker image

([#1778](https://github.com/streamnative/pulsarctl/pull/1778)) Fix jwt cve

([#1711](https://github.com/streamnative/pulsarctl/pull/1711)) fix code check

([#1704](https://github.com/streamnative/pulsarctl/pull/1704)) feat: Subscription get message by id json output

([#1699](https://github.com/streamnative/pulsarctl/pull/1699)) Update subscription get message by id typo lederId to ledgerId

([#1585](https://github.com/streamnative/pulsarctl/pull/1585)) fix: upgrade golang version to fix CVE

([#1587](https://github.com/streamnative/pulsarctl/pull/1587)) Setup go version to 1.22 fix cve

([#1549](https://github.com/streamnative/pulsarctl/pull/1549)) update pulsar-client-go to master latest commit

2af1258 fix ci

([#1537](https://github.com/streamnative/pulsarctl/pull/1537)) Bump the pulsar-client-go to the master version

6f25051 Fix TestDeleteNonExistPartitionedTopic

([#1509](https://github.com/streamnative/pulsarctl/pull/1509)) Fix json marshal error for Secrets and UserConfigs when creating/updating functions

([#1455](https://github.com/streamnative/pulsarctl/pull/1455)) Support create token with headers

([#1478](https://github.com/streamnative/pulsarctl/pull/1478)) Upgrade the dependency version to fix vulnerabilities

([#1447](https://github.com/streamnative/pulsarctl/pull/1447)) Add trivy scan workflow to avoid vulnerabilities

([#1451](https://github.com/streamnative/pulsarctl/pull/1451)) \[fix] Upgrade go version to 1.21 to fix CVE-2023-24538

([#1362](https://github.com/streamnative/pulsarctl/pull/1362)) fix source test typo

([#1360](https://github.com/streamnative/pulsarctl/pull/1360)) fix source test

([#1419](https://github.com/streamnative/pulsarctl/pull/1419)) Auth SN docker hub

([#1405](https://github.com/streamnative/pulsarctl/pull/1405)) Support no auth context

([#1393](https://github.com/streamnative/pulsarctl/pull/1393)) fix token

([#1402](https://github.com/streamnative/pulsarctl/pull/1402)) Add docker hub login

([#1398](https://github.com/streamnative/pulsarctl/pull/1398)) Auth SN docker hub

5cb0593 Disable bk unit test and fix it later ---

([#1257](https://github.com/streamnative/pulsarctl/pull/1257)) Add method to mark bookie readonly

([#1328](https://github.com/streamnative/pulsarctl/pull/1328)) Build arm64 linux executable binary artifact

([#1305](https://github.com/streamnative/pulsarctl/pull/1305)) Update jose2go to fix GHSA-mhpq-9638-x6pw

([#1268](https://github.com/streamnative/pulsarctl/pull/1268)) Update golang.org/x/net

([#1205](https://github.com/streamnative/pulsarctl/pull/1205)) Replace apache pulsar client go repo on 3.0 branch

([#1084](https://github.com/streamnative/pulsarctl/pull/1084)) Fixed remove auth plugin suffix

([#1072](https://github.com/streamnative/pulsarctl/pull/1072)) Removed error char

([#1065](https://github.com/streamnative/pulsarctl/pull/1065)) Bump pulsar version to 3.0.0.1

([#1067](https://github.com/streamnative/pulsarctl/pull/1067)) fix: Fix TestUpdateTopicNotExist and TestUpdateNonPartitionedTopic

### Function Mesh Worker Service

Support invalid name


# V3.0.10.7
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.10.7



## StreamNative Weekly Release Notes v3.0.10.7

#### General Changes

### Apache Pulsar

([#24552](https://github.com/apache/pulsar/pull/24552)) \[improve]\[test] Remove EntryCacheCreator from ManagedLedgerFactoryImpl

([#24544](https://github.com/apache/pulsar/pull/24544)) \[improve] Upgrade pulsar-client-python to 3.8.0 in Docker image

([#24516](https://github.com/apache/pulsar/pull/24516)) \[fix]\[broker] Fix exclusive producer creation when last shared producer closes

([#24506](https://github.com/apache/pulsar/pull/24506)) \[fix]\[broker] Fix duplicate increment of ADD\_OP\_COUNT\_UPDATER in OpAddEntry

([#24543](https://github.com/apache/pulsar/pull/24543)) \[fix]\[broker] Fix matching of topicsPattern for topic names which contain non-ascii characters

([#24539](https://github.com/apache/pulsar/pull/24539)) \[fix]\[client] Close orphan producer or consumer when the creation is interrupted

([#24517](https://github.com/apache/pulsar/pull/24517)) \[fix]\[client] Fix ClientCnx handleSendError NPE

([#24515](https://github.com/apache/pulsar/pull/24515)) \[fix]\[ml] Fix asyncReadEntries might never complete if empty entries are read from BK

([#24525](https://github.com/apache/pulsar/pull/24525)) \[improve]\[misc] Optimize topic list hashing so that potentially large String allocation is avoided

([#24528](https://github.com/apache/pulsar/pull/24528)) \[fix]\[client] Fix issue in auto releasing of idle connection with topics pattern consumer

([#24529](https://github.com/apache/pulsar/pull/24529)) \[fix]\[proxy] Fix default value of connectionMaxIdleSeconds in Pulsar Proxy

([#24476](https://github.com/apache/pulsar/pull/24476)) \[fix]\[client] NPE in MultiTopicsConsumerImpl.negativeAcknowledge

([#24465](https://github.com/apache/pulsar/pull/24465)) \[fix]\[proxy] Fix proxy OOM by replacing TopicName with a simple conversion method

([#22495](https://github.com/apache/pulsar/pull/22495)) \[fix]\[test] Move ExtensibleLoadManagerImplTest to flaky tests

([#21642](https://github.com/apache/pulsar/pull/21642)) \[fix]\[test] Fix flaky test SimpleProducerConsumerStatTest#testPartitionTopicStats

([#24453](https://github.com/apache/pulsar/pull/24453)) \[fix]\[broker] replication does not work due to the mixed and repetitive sending of user messages and replication markers

([#24424](https://github.com/apache/pulsar/pull/24424)) \[fix]\[broker] Fix the non-persistenttopic's replicator always get error "Producer send queue is full" if set a small value of the config replicationProducerQueueSize

([#24189](https://github.com/apache/pulsar/pull/24189)) \[fix]\[broker]excessive replication speed leads to error: Producer send queue is full

([#22674](https://github.com/apache/pulsar/pull/22674)) \[Fix]\[broker] Limit replication rate based on bytes

([#20931](https://github.com/apache/pulsar/pull/20931)) \[fix]\[broker] Fix ack hole in cursor for geo-replication

([#24443](https://github.com/apache/pulsar/pull/24443)) \[fix]\[txn] Fix negative unacknowledged messages in transactions by ensuring that the batch size is added into CommandAck

([#24421](https://github.com/apache/pulsar/pull/24421)) \[fix]\[build] Add missing `<name>` to submodules

([#24441](https://github.com/apache/pulsar/pull/24441)) \[fix]\[ml] Enhance OpFindNewest to support skip non-recoverable data

([#24459](https://github.com/apache/pulsar/pull/24459)) \[improve]\[broker] change to warn log level for ack validation error

([#24434](https://github.com/apache/pulsar/pull/24434)) \[improve]\[broker] Improve the log when namespace bundle is not available

([#24432](https://github.com/apache/pulsar/pull/24432)) \[fix]\[ml]Still got BK ledger, even though it has been deleted after offloaded

([#21467](https://github.com/apache/pulsar/pull/21467)) \[fix]\[test] Cleanup resources if starting PulsarService fails in PulsarTestContext

([#24351](https://github.com/apache/pulsar/pull/24351)) \[improve]\[broker] Deny removing local cluster from topic level replicated cluster policy

([#24419](https://github.com/apache/pulsar/pull/24419)) \[fix]\[broker] Once the cluster is configured incorrectly, the broker maintains the incorrect cluster configuration even if you removed it

([#24404](https://github.com/apache/pulsar/pull/24404)) \[fix]\[client] Prevent NPE when seeking with null topic in TopicMessageId

([#24405](https://github.com/apache/pulsar/pull/24405)) \[fix]\[ml]Received more than once callback when calling cursor.delete

([#24406](https://github.com/apache/pulsar/pull/24406)) \[fix]\[ml] Cursor ignores the position that has an empty ack-set if disabled deletionAtBatchIndexLevelEnabled

([#24401](https://github.com/apache/pulsar/pull/24401)) \[fix]\[txn] Fix deadlock when loading transaction buffer snapshot

([#24402](https://github.com/apache/pulsar/pull/24402)) \[fix]\[client] Fix some potential resource leak

([#20629](https://github.com/apache/pulsar/pull/20629)) \[improve]\[test] Fix flaky test SimpleProducerConsumerStatTest#testMsgRateExpired

([#21629](https://github.com/apache/pulsar/pull/21629)) \[fix]\[build] Fix potential insufficient protostuff-related configs

([#23594](https://github.com/apache/pulsar/pull/23594)) \[fix] \[broker] No longer allow creating subscription that contains slash

([#24366](https://github.com/apache/pulsar/pull/24366)) \[fix]\[broker]Fix deadlock when compaction and topic deletion execute concurrently

([#24350](https://github.com/apache/pulsar/pull/24350)) \[fix]\[broker] Fix issue that topic policies was deleted after a sub topic deleted, even if the partitioned topic still exists

([#24384](https://github.com/apache/pulsar/pull/24384)) \[fix]\[ml]Revert a behavior change of releasing idle offloaded ledger handle: only release idle BlobStoreBackedReadHandle

([#24397](https://github.com/apache/pulsar/pull/24397)) \[improve]\[misc] Upgrade Netty to 4.1.122.Final and tcnative to 2.0.72.Final

([#24391](https://github.com/apache/pulsar/pull/24391)) \[improve]\[broker] Add managedCursor/LedgerInfoCompressionType settings to broker.conf

([#24392](https://github.com/apache/pulsar/pull/24392)) \[improve]\[broker] Make maxBatchDeletedIndexToPersist configurable and document other related configs

([#24386](https://github.com/apache/pulsar/pull/24386)) \[improve]\[broker] Added synchronized for sendMessages in Non-Persistent message dispatchers

([#24381](https://github.com/apache/pulsar/pull/24381)) \[improve]\[ml]Release idle offloaded read handle only the ref count is 0

([#19783](https://github.com/apache/pulsar/pull/19783)) \[improve]\[offloaders] Automatically evict Offloaded Ledgers from memory

([#24360](https://github.com/apache/pulsar/pull/24360)) \[fix]\[broker] expose consumer name for partitioned topic stats

([#24359](https://github.com/apache/pulsar/pull/24359)) \[improve]\[broker]Improve the log when encountered in-flight read limitation

([#24354](https://github.com/apache/pulsar/pull/24354)) \[fix]\[io] Acknowledge RabbitMQ message after processing the message successfully

([#24352](https://github.com/apache/pulsar/pull/24352)) \[fix]\[broker] Ignore metadata changes when broker is not in the Started state

([#24190](https://github.com/apache/pulsar/pull/24190)) \[fix]\[broker] Resolve the issue of frequent updates in message expiration deletion rate

([#24338](https://github.com/apache/pulsar/pull/24338)) \[fix]\[ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent issue

([#24331](https://github.com/apache/pulsar/pull/24331)) \[fix]\[offload] Complete the future outside of the reading loop in BlobStoreBackedReadHandleImplV2.readAsync

([#24324](https://github.com/apache/pulsar/pull/24324)) \[fix]\[test] Fix flaky AutoScaledReceiverQueueSizeTest.testNegativeClientMemory

([#24316](https://github.com/apache/pulsar/pull/24316)) \[fix]\[io] Fix kinesis avro bytes handling

([#24344](https://github.com/apache/pulsar/pull/24344)) \[improve]\[ml] Offload ledgers without check ledger length

([#24286](https://github.com/apache/pulsar/pull/24286)) \[fix]\[broker]Non-global topic policies and global topic policies overwrite each other

([#24279](https://github.com/apache/pulsar/pull/24279)) \[fix]\[broker]Global topic policies do not affect after unloading topic and persistence global topic policies never affect

([#24349](https://github.com/apache/pulsar/pull/24349)) \[fix]\[io]\[branch-3.0] Backport Kinesis Sink custom native executable support #23762

([#24317](https://github.com/apache/pulsar/pull/24317)) \[fix]\[io]\[branch-3.0]Pulsar-SQL: Fix classcast ex when decode decimal value

([#24313](https://github.com/apache/pulsar/pull/24313)) \[fix]\[broker] Fix potential deadlock when creating partitioned topic

([#24293](https://github.com/apache/pulsar/pull/24293)) \[fix]\[broker] fix wrong method name checkTopicExists.

([#24307](https://github.com/apache/pulsar/pull/24307)) \[fix]\[build] Ensure that buildtools is Java 8 compatible and fix remaining compatibility issue

([#24304](https://github.com/apache/pulsar/pull/24304)) \[fix]\[test] Simplify BetweenTestClassesListenerAdapter and fix issue with BeforeTest/AfterTest annotations

([#24289](https://github.com/apache/pulsar/pull/24289)) \[improve]\[io] Add configuration parameter for disabling aggregation for Kinesis Producers

([#24302](https://github.com/apache/pulsar/pull/24302)) \[improve] Upgrade pulsar-client-python to 3.7.0 in Docker image

([#24299](https://github.com/apache/pulsar/pull/24299)) \[fix]\[test] Fix more Netty ByteBuf leaks in tests

([#24297](https://github.com/apache/pulsar/pull/24297)) \[fix]\[io] Fix SyntaxWarning in Pulsar Python functions

([#24282](https://github.com/apache/pulsar/pull/24282)) \[fix]\[client] Fix producer publishing getting stuck after message with incompatible schema is discarded

([#24283](https://github.com/apache/pulsar/pull/24283)) \[cleanup]\[test] Remove unused parameter from deleteNamespaceWithRetry method in MockedPulsarServiceBaseTest

([#24263](https://github.com/apache/pulsar/pull/24263)) \[improve]\[build] Upgrade zstd version from 1.5.2-3 to 1.5.7-3

([#24281](https://github.com/apache/pulsar/pull/24281)) \[fix]\[test] Fix multiple ByteBuf leaks in tests

([#24275](https://github.com/apache/pulsar/pull/24275)) \[fix]\[broker] Fix HashedWheelTimer leak in PulsarService by stopping it in shutdown

([#24274](https://github.com/apache/pulsar/pull/24274)) \[fix]\[misc] Fix ByteBuf leak in SchemaUtils

([#24254](https://github.com/apache/pulsar/pull/24254)) \[fix]\[broker]Fix incorrect priority between topic policies and global topic policies

([#24266](https://github.com/apache/pulsar/pull/24266)) \[improve]\[ci] Disable detailed console logging for integration tests in CI

([#24261](https://github.com/apache/pulsar/pull/24261)) \[fix]\[test] Fix flaky ManagedCursorTest.testLastActiveAfterResetCursor and disable failing SchemaTest

([#24244](https://github.com/apache/pulsar/pull/24244)) \[fix]\[test] Fix flaky ManagedCursorTest.testSkipEntriesWithIndividualDeletedMessages

([#24248](https://github.com/apache/pulsar/pull/24248)) \[improve]\[io]\[kca] support fully-qualified topic names in source records

([#24260](https://github.com/apache/pulsar/pull/24260)) \[improve]\[build] Upgrade Gradle Develocity Maven Extension dependencies

([#24258](https://github.com/apache/pulsar/pull/24258)) \[fix]\[test] Fix TestNG BetweenTestClassesListenerAdapter listener

([#24257](https://github.com/apache/pulsar/pull/24257)) \[fix]\[broker] Unregister non-static metrics collectors registered in Prometheus default registry

([#24178](https://github.com/apache/pulsar/pull/24178)) \[fix]\[broker]fix memory leak, messages lost, incorrect replication state if using multiple schema versions(auto\_produce)

([#24219](https://github.com/apache/pulsar/pull/24219)) \[improve]\[broker]Improve the feature "Optimize subscription seek (cursor reset) by timestamp": search less entries

([#23919](https://github.com/apache/pulsar/pull/23919)) \[fix]\[broker] Fix seeking by timestamp can be reset the cursor position to earliest

([#22792](https://github.com/apache/pulsar/pull/22792)) \[improve]\[broker] Optimize subscription seek (cursor reset) by timestamp

([#24243](https://github.com/apache/pulsar/pull/24243)) \[improve]\[build] Upgrade SpotBugs to 4.9.x

([#24240](https://github.com/apache/pulsar/pull/24240)) \[improve]\[build] Upgrade to jacoco 0.8.13

([#24237](https://github.com/apache/pulsar/pull/24237)) \[improve]\[build] Upgrade Lombok to 1.18.38 to support JDK 24

([#24221](https://github.com/apache/pulsar/pull/24221)) \[improve]\[io] support kafka connect transforms and predicates

([#24230](https://github.com/apache/pulsar/pull/24230)) \[improve]\[client]Improve transaction log when a TXN command timeout

([#24223](https://github.com/apache/pulsar/pull/24223)) \[fix]\[broker] Orphan schema after disabled a cluster for a namespace

([#24228](https://github.com/apache/pulsar/pull/24228)) \[fix]\[broker] Fix ByteBuf memory leak in REST API for publishing messages

([#24184](https://github.com/apache/pulsar/pull/24184)) \[fix]\[client] Fix incorrect producer.getPendingQueueSize due to incomplete queue implementation

([#24214](https://github.com/apache/pulsar/pull/24214)) \[improve] Upgrade Netty to 4.1.121.Final

([#24212](https://github.com/apache/pulsar/pull/24212)) \[fix]\[test] Fix flaky BatchMessageWithBatchIndexLevelTest.testBatchMessageAck

([#24218](https://github.com/apache/pulsar/pull/24218)) \[fix]\[test] Fix multiple resource leaks in tests

([#24187](https://github.com/apache/pulsar/pull/24187)) \[improve]\[client] validate ClientConfigurationData earlier to avoid resource leaks

([#24216](https://github.com/apache/pulsar/pull/24216)) \[fix]\[broker] Fix HealthChecker deadlock in shutdown

([#24209](https://github.com/apache/pulsar/pull/24209)) \[fix]\[broker] Fix tenant creation and update with null value

([#24192](https://github.com/apache/pulsar/pull/24192)) \[fix]\[admin] Backlog quota's policy is null which causes a NPE

([#24210](https://github.com/apache/pulsar/pull/24210)) \[fix]\[broker] Fix broker shutdown delay by resolving hanging health checks

([#24207](https://github.com/apache/pulsar/pull/24207)) \[fix]\[broker] Fix compaction service log's wrong condition

([#24204](https://github.com/apache/pulsar/pull/24204)) \[fix]\[test] Fix resource leaks in ProxyTest and fix invalid tests

([#24201](https://github.com/apache/pulsar/pull/24201)) \[improve]\[io] Upgrade Kafka client and compatible Confluent platform version

([#24118)](https://github.com/apache/pulsar/pull/24118))) Revert "\[fix]\[broker] Add topic consistency check

([#24154)](https://github.com/apache/pulsar/pull/24154))) Revert "\[fix]\[broker] Directly query single topic existence when the topic is partitioned

([#24032](https://github.com/apache/pulsar/pull/24032)) \[fix]\[broker] Fix missing validation when setting retention policy on topic level

([#24098](https://github.com/apache/pulsar/pull/24098)) \[fix]\[ml] Skip deleting cursor if it was already deleted before calling unsubscribe

([#24181](https://github.com/apache/pulsar/pull/24181)) \[fix]\[proxy] Fix incorrect client error when calling get topic metadata

([#24158](https://github.com/apache/pulsar/pull/24158)) \[fix]\[proxy] Propagate client connection feature flags through Pulsar Proxy to Broker

([#24103](https://github.com/apache/pulsar/pull/24103)) \[fix]\[schema] Reject unsupported Avro schema types during schema registration

([#24091](https://github.com/apache/pulsar/pull/24091)) \[fix]\[broker] Fix some problems in calculate totalAvailableBookies in method getExcludedBookiesWithIsolationGroups when some bookies belongs to multiple isolation groups.

([#21320](https://github.com/apache/pulsar/pull/21320)) \[fix]\[bk] Fix the var name for IsolationGroups

([#24171](https://github.com/apache/pulsar/pull/24171)) \[improve]\[test] Use configured session timeout for MockZooKeeper and TestZKServer in PulsarTestContext

([#24172](https://github.com/apache/pulsar/pull/24172)) \[fix]\[test] Improve reliability of IncrementPartitionsTest

([#24170](https://github.com/apache/pulsar/pull/24170)) \[fix]\[test]flaky-test:ManagedLedgerInterceptorImplTest.testManagedLedgerPayloadInputProcessorFailure

([#23980](https://github.com/apache/pulsar/pull/23980)) \[fix]\[broker] Consumer stuck when delete subscription \_\_compaction failed

([#24167](https://github.com/apache/pulsar/pull/24167)) \[fix]\[ml] Fix ML thread blocking issue in internalGetPartitionedStats API

([#24166](https://github.com/apache/pulsar/pull/24166)) \[fix]\[test] Fix invalid test CompactionTest.testDeleteCompactedLedgerWithSlowAck

([#24150](https://github.com/apache/pulsar/pull/24150)) \[fix]\[broker] The feature brokerDeleteInactivePartitionedTopicMetadataEnabled leaves orphan topic policies and topic schemas

([#24154](https://github.com/apache/pulsar/pull/24154)) \[fix]\[broker] Directly query single topic existence when the topic is partitioned

([#24118](https://github.com/apache/pulsar/pull/24118)) \[fix]\[broker] Add topic consistency check

([#24056](https://github.com/apache/pulsar/pull/24056)) \[fix]\[test] Update partitioned topic subscription assertions in IncrementPartitionsTest

([#24033](https://github.com/apache/pulsar/pull/24033)) \[cleanup]\[misc] Add override annotation

([#24161](https://github.com/apache/pulsar/pull/24161)) \[fix]\[test] Fix flaky BrokerServiceChaosTest.testFetchPartitionedTopicMetadataWithCacheRefresh

([#24162](https://github.com/apache/pulsar/pull/24162)) \[fix]\[test] Fix flaky BrokerServiceChaosTest

([#24097](https://github.com/apache/pulsar/pull/24097)) \[fix] \[broker] topics infinitely failed to delete after remove cluster from replicated clusters modifying when using partitioned system topic

([#22261](https://github.com/apache/pulsar/pull/22261)) \[fix] Bump google.golang.org/protobuf from 1.32.0 to 1.33.0 in /pulsar-function-go

([#24132](https://github.com/apache/pulsar/pull/24132)) \[fix]\[io] Fix KinesisSink json flattening for AVRO's SchemaType.BYTES

([#20984](https://github.com/apache/pulsar/pull/20984)) \[fix]\[broker] Fix get outdated compactedTopicContext after compactionHorizon has been updated

([#20697](https://github.com/apache/pulsar/pull/20697)) \[improve]\[broker] Improve CompactedTopicImpl lock

([#24131](https://github.com/apache/pulsar/pull/24131)) \[fix]\[ml] Return 1 when bytes size is 0 or negative for entry count estimation

([#24128](https://github.com/apache/pulsar/pull/24128)) \[improve]\[io] Enhance Kafka connector logging with focused bootstrap server information

([#24125](https://github.com/apache/pulsar/pull/24125)) \[fix]\[ml] Don't estimate number of entries when ledgers are empty, return 1 instead

([#24123](https://github.com/apache/pulsar/pull/24123)) \[improve]\[client] Prevent NullPointException when closing ClientCredentialsFlow

([#24124](https://github.com/apache/pulsar/pull/24124)) \[improve]\[io] Remove sleep when sourceTask.poll of kafka return null

([#24116](https://github.com/apache/pulsar/pull/24116)) \[improve]\[broker] Change topic exists log to warn

([#24104](https://github.com/apache/pulsar/pull/24104)) \[fix]\[client] Pattern subscription regression when broker-side evaluation is disabled

([#24100](https://github.com/apache/pulsar/pull/24100)) \[fix]\[client] Fix consumer leak when thread is interrupted before subscribe completes

([#24089](https://github.com/apache/pulsar/pull/24089)) \[fix]\[ml] Fix issues in estimateEntryCountBySize

([#24073](https://github.com/apache/pulsar/pull/24073)) \[improve]\[broker] Optimize message expiration rate repeated update issues

([#24087](https://github.com/apache/pulsar/pull/24087)) \[fix]\[broker] Avoid IllegalStateException when marker\_type field is not set in publishing

([#24083](https://github.com/apache/pulsar/pull/24083)) \[fix]\[ci] Bump dependency-check to 12.1.0 to fix OWASP Dependency Check job

([#24082](https://github.com/apache/pulsar/pull/24082)) \[clean]\[client] Clean code for the construction of retry/dead letter topic name

([#24079](https://github.com/apache/pulsar/pull/24079)) \[fix]\[broker] Fix NPE while publishing Metadata-Event with not init producer

([#24080](https://github.com/apache/pulsar/pull/24080)) \[fix]\[broker] Fix Metadata event synchronizer should not fail with bad version

([#24081](https://github.com/apache/pulsar/pull/24081)) \[fix]\[broker] Fix Metadata Event Synchronizer producer creation retry so that the producer gets created eventually

([#24048](https://github.com/apache/pulsar/pull/24048)) \[fix]\[broker] Fix UnsupportedOperationException while setting subscription level dispatch rate policy

([#24054](https://github.com/apache/pulsar/pull/24054)) \[fix]\[ml] Corrected pulsar\_storage\_size metric to not multiply offloaded storage by the write quorum

([#24067](https://github.com/apache/pulsar/pull/24067)) \[fix]\[broker] http metric endpoint get compaction latency stats always be 0

([#24064](https://github.com/apache/pulsar/pull/24064)) \[improve]\[broker] Optimize ThresholdShedder with improved boundary checks and parameter reuse

([#24055](https://github.com/apache/pulsar/pull/24055)) \[fix] Avoid negative estimated entry count

([#24060](https://github.com/apache/pulsar/pull/24060)) \[improve]\[monitor] Add version=0.0.4 to /metrics content type for Prometheus 3.x compatibility

([#24059](https://github.com/apache/pulsar/pull/24059)) \[fix]\[client] Copy eventTime to retry letter topic and DLQ messages

([#24061](https://github.com/apache/pulsar/pull/24061)) \[fix]\[client] Fix building broken batched message when publishing

([#24063](https://github.com/apache/pulsar/pull/24063)) \[fix]\[broker]Fix failed consumption after loaded up a terminated topic

([#24072](https://github.com/apache/pulsar/pull/24072)) \[fix]\[broker] Pattern subscription doesn't work when the pattern excludes the topic domain.

([#24049](https://github.com/apache/pulsar/pull/24049)) \[improve] Upgrade Netty to 4.1.119.Final

([#23975](https://github.com/apache/pulsar/pull/23975)) \[fix]\[broker] Add expire check for replicator

([#24023](https://github.com/apache/pulsar/pull/24023)) \[fix]\[doc] fix doc related to chunk message feature.

([#23962](https://github.com/apache/pulsar/pull/23962)) \[improve]\[ml] Use lock-free queue in InflightReadsLimiter since there's no concurrent access

([#23978](https://github.com/apache/pulsar/pull/23978)) \[improve]\[cli] Support additional msg metadata for V1 topic on peek message cmd

([#24014](https://github.com/apache/pulsar/pull/24014)) \[fix]\[broker] Fix BucketDelayedDeliveryTracker thread safety

([#24019](https://github.com/apache/pulsar/pull/24019)) \[fix]\[test]Fix flaky test V1\_ProducerConsumerTest.testConcurrentConsumerReceiveWhileReconnect

([#24011](https://github.com/apache/pulsar/pull/24011)) \[fix]\[test] Fix flaky test OneWayReplicatorUsingGlobalZKTest.testConfigReplicationStartAt

([#23931](https://github.com/apache/pulsar/pull/23931)) \[improve] \[broker] Make the estimated entry size more accurate

([#24004](https://github.com/apache/pulsar/pull/24004)) \[improve]\[ci] Upgrade Gradle Develocity Maven Extension to 1.23.1

([#23697](https://github.com/apache/pulsar/pull/23697)) \[fix]\[broker] Geo Replication lost messages or frequently fails due to Deduplication is not appropriate for Geo-Replication

([#24006](https://github.com/apache/pulsar/pull/24006)) \[fix]\[broker] fix broker identifying incorrect stuck topic

([#23286](https://github.com/apache/pulsar/pull/23286)) \[improve]\[broker] Fix non-persistent system topic schema compatibility

([#23881](https://github.com/apache/pulsar/pull/23881)) \[improve]\[fn] Set default tenant and namespace for ListFunctions cmd

([#23730](https://github.com/apache/pulsar/pull/23730)) \[fix]\[admin] Verify is policies read only before revoke permissions on topic

([#24003](https://github.com/apache/pulsar/pull/24003)) \[improve]\[test] Upgrade Testcontainers to 1.20.4 and docker-java to 3.4.0

### KoP

Fix topic name reference in AlterPartitionReassignments

### StreamNative Pulsar Plugins

1ed217a86 Update oxia version

### pulsarctl

e7d5e82 Use snstage docker image

Fix jwt cve

fix code check

feat: Subscription get message by id json output

Update subscription get message by id typo lederId to ledgerId

fix: upgrade golang version to fix CVE

Setup go version to 1.22 fix cve

update pulsar-client-go to master latest commit

2af1258 fix ci

Bump the pulsar-client-go to the master version

6f25051 Fix TestDeleteNonExistPartitionedTopic

Fix json marshal error for Secrets and UserConfigs when creating/updating functions

Support create token with headers

Upgrade the dependency version to fix vulnerabilities

Add trivy scan workflow to avoid vulnerabilities

\[fix] Upgrade go version to 1.21 to fix CVE-2023-24538

fix source test typo

fix source test

Auth SN docker hub

Support no auth context

fix token

Add docker hub login

Auth SN docker hub

5cb0593 Disable bk unit test and fix it later ---

Add method to mark bookie readonly

Build arm64 linux executable binary artifact

Update jose2go to fix GHSA-mhpq-9638-x6pw

Update golang.org/x/net

Replace apache pulsar client go repo on 3.0 branch

Fixed remove auth plugin suffix

Removed error char

Bump pulsar version to 3.0.0.1

fix: Fix TestUpdateTopicNotExist and TestUpdateNonPartitionedTopic

### Function Mesh Worker Service

Set minReplicas to parallelism when HPA is enabled

Set default VPA by default when HPA is not enabled


# V3.0.10.8
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.10.8



## StreamNative Weekly Release Notes v3.0.10.8

#### General Changes

### Apache Pulsar

([#24772](https://github.com/apache/pulsar/pull/24772)) \[fix]\[misc] Fix compareTo contract violation for NamespaceBundleStats, TimeAverageMessageData and ResourceUnitRanking

([#24769](https://github.com/apache/pulsar/pull/24769)) \[fix]\[test] Flaky-test: BrokerServiceTest.testShutDownWithMaxConcurrentUnload

([#24767](https://github.com/apache/pulsar/pull/24767)) \[fix]\[ci] Fix CI for Java 25 including upgrade of Gradle Develocity Maven extension

([#24763](https://github.com/apache/pulsar/pull/24763)) \[improve]\[build] Upgrade Lombok to 1.18.42 to fully support JDK25

([#23634](https://github.com/apache/pulsar/pull/23634)) \[improve]\[broker] If there is a deadlock in the service, the probe should return a failure because the service may be unavailable

([#24738](https://github.com/apache/pulsar/pull/24738)) \[fix]\[broker] First entry will be skipped if opening NonDurableCursor while trimmed ledger is adding first entry.

([#24753](https://github.com/apache/pulsar/pull/24753)) \[fix]\[ml]Fix EOFException after enabled topics offloading

([#24741](https://github.com/apache/pulsar/pull/24741)) \[fix]\[broker] Prevent unexpected recycle failure in dispatcher's read callback

([#20522](https://github.com/apache/pulsar/pull/20522)) \[improve]\[broker] Choose random thread for consumerFlow in PersistentDispatcherSingleActiveConsumer

([#24758](https://github.com/apache/pulsar/pull/24758)) \[fix]\[broker]\[branch-3.0] Prevent NPE in ownedBundlesCountPerNamespace on first bundle load

([#24752](https://github.com/apache/pulsar/pull/24752)) \[fix]\[client] rollback TopicListWatcher retry behavior

([#24698](https://github.com/apache/pulsar/pull/24698)) \[fix]\[client]TopicListWatcher not closed when calling PatternMultiTopicsConsumerImpl.closeAsync() method

([#24634](https://github.com/apache/pulsar/pull/24634)) Dispatcher did unnecessary sort for recentlyJoinedConsumers and printed noisy error logs

([#24730](https://github.com/apache/pulsar/pull/24730)) \[fix]\[broker] Ensure KeyShared sticky mode consumer respects assigned ranges

([#24743](https://github.com/apache/pulsar/pull/24743)) \[fix]\[client] Fix receiver queue auto-scale without memory limit

([#24731](https://github.com/apache/pulsar/pull/24731)) \[fix]\[broker] Fix cannot shutdown broker gracefully by admin api

([#24654](https://github.com/apache/pulsar/pull/24654)) \[fix]\[io] Improve Kafka Connect source offset flushing logic

([#24725](https://github.com/apache/pulsar/pull/24725)) \[fix]\[client] Avoid recycling the same ConcurrentBitSetRecyclable among different threads

([#24719](https://github.com/apache/pulsar/pull/24719)) \[fix]\[broker] Fix memory leak when metrics are updated in a thread other than FastThreadLocalThread

([#24594](https://github.com/apache/pulsar/pull/24594)) \[improve]\[build] Disable javadoc build failure

([#24580](https://github.com/apache/pulsar/pull/24580)) \[fix]\[broker]Fix never recovered metadata store bad version issue if received a large response from ZK

([#23336](https://github.com/apache/pulsar/pull/23336)) \[fix]\[client] Fix ArrayIndexOutOfBoundsException when using SameAuthParamsLookupAutoClusterFailover

([#23977](https://github.com/apache/pulsar/pull/23977)) \[fix]\[broker] Invalid regex in PulsarLedgerManager causes zk data notification to be ignored

([#24518](https://github.com/apache/pulsar/pull/24518)) ([#24671](https://github.com/apache/pulsar/pull/24671)) \[fix]\[broker]\[branch-3.0] Fix wrong backlog age metrics when the mark delete position point to a deleted ledger

([#24663](https://github.com/apache/pulsar/pull/24663)) \[fix]\[client] Skip schema validation when sending messages to DLQ to avoid infinite loop when schema validation fails on an incoming message

([#24669](https://github.com/apache/pulsar/pull/24669)) \[improve]\[io] Support specifying Kinesis KPL native binary path with 1.0 version specific path

([#24668](https://github.com/apache/pulsar/pull/24668)) \[improve]\[build] Use org.apache.nifi:nifi-nar-maven-plugin:2.1.0 with skipDocGeneration=true

([#24661](https://github.com/apache/pulsar/pull/24661)) \[improve]\[io] Upgrade AWS SDK v1 & v2, Kinesis KPL and KPC versions

([#24639](https://github.com/apache/pulsar/pull/24639)) \[fix]\[broker] Fix race condition in MetadataStoreCacheLoader causing inconsistent availableBroker list caching

([#24666](https://github.com/apache/pulsar/pull/24666)) \[improve]\[build] Increase maven resolver's sync context timeout

([#24662](https://github.com/apache/pulsar/pull/24662)) \[fix]\[client] fix ArrayIndexOutOfBoundsException in SameAuthParamsLookupAutoClusterFailover

([#24649](https://github.com/apache/pulsar/pull/24649)) \[fix]\[offload] Exclude unnecessary dependencies from tiered storage provider / offloader nar files

([#24643](https://github.com/apache/pulsar/pull/24643)) \[fix]\[broker] Add double-check for non-durable cursor creation

([#24633](https://github.com/apache/pulsar/pull/24633)) \[fix]\[test]fix flaky ZeroQueueSizeTest.testZeroQueueGetExceptionWhenReceiveBatchMessage

([#24632](https://github.com/apache/pulsar/pull/24632)) \[fix]\[test] Fix ConcurrentModificationException in Ipv4Proxy

([#24630](https://github.com/apache/pulsar/pull/24630)) \[fix]\[test]fix flaky ZeroQueueSizeTest.testZeroQueueGetExceptionWhenReceiveBatchMessage

([#24626](https://github.com/apache/pulsar/pull/24626)) \[fix]\[proxy] Fix TooLongFrameException with Pulsar Proxy

([#24621](https://github.com/apache/pulsar/pull/24621)) \[fix]\[broker] Fix duplicate watcher registration after SessionReestablished

([#24610](https://github.com/apache/pulsar/pull/24610)) \[fix]\[client]Prevent ZeroQueueConsumer from receiving batch messages when using MessagePayloadProcessor

([#24604](https://github.com/apache/pulsar/pull/24604)) \[improve]\[io] Add dependency file name information to error message when .nar file validation fails with ZipException

([#21361](https://github.com/apache/pulsar/pull/21361)) \[improve]\[broker] Optimize and clean up aggregation of topic stats

([#24601](https://github.com/apache/pulsar/pull/24601)) \[improve]\[doc] Improve the JavaDocs of sendAsync to avoid improper use

([#24599](https://github.com/apache/pulsar/pull/24599)) \[fix]\[client] Retry for unknown exceptions when creating a producer or consumer

([#24450](https://github.com/apache/pulsar/pull/24450)) \[fix]\[broker] Fix REST API to produce messages to single-partitioned topics

([#24595](https://github.com/apache/pulsar/pull/24595)) \[fix]\[ci] Fix code coverage metrics in Pulsar CI

([#24582](https://github.com/apache/pulsar/pull/24582)) \[improve]\[client] Support load RSA PKCS#8 private key

([#24535](https://github.com/apache/pulsar/pull/24535)) \[improve]\[test] Add test for dead letter topic with max unacked messages blocking

([#24532](https://github.com/apache/pulsar/pull/24532)) \[fix]\[misc] Upgrade dependencies to fix critical security vulnerabilities

([#24514](https://github.com/apache/pulsar/pull/24514)) \[improve]\[build] Bump org.apache.commons:commons-lang3 from 3.17.0 to 3.18.0

([#24586](https://github.com/apache/pulsar/pull/24586)) \[improve]\[test] Refactor the way way pulsar-io-debezium-oracle nar file is patched when building the test image

([#24590](https://github.com/apache/pulsar/pull/24590)) \[fix]\[broker] Fix flaky testReplicatorsInflightTaskListIsEmptyAfterReplicationFinished

([#24542)](https://github.com/apache/pulsar/pull/24542))) Revert "\[fix]\[broker]Fix thread safety issues in BucketDelayedDeliveryTracker with StampedLock optimistic reads

([#24554](https://github.com/apache/pulsar/pull/24554)) ([#24571](https://github.com/apache/pulsar/pull/24571)) \[fix]\[client]\[branch-4.0] Partitioned topics are unexpectedly created by client after deletion

([#24576](https://github.com/apache/pulsar/pull/24576)) \[fix]\[test] fix flaky GrowableArrayBlockingQueueTest.testPollBlockingThreadsTermination

([#24569](https://github.com/apache/pulsar/pull/24569)) \[fix]\[broker] Fix ManagedCursor state management race conditions and lifecycle issues

([#24550](https://github.com/apache/pulsar/pull/24550)) \[improve]\[client] Terminate consumer.receive() when consumer is closed

([#24473](https://github.com/apache/pulsar/pull/24473)) \[improve]\[build] replace org.apache.commons.lang to org.apache.commons.lang3

([#24560](https://github.com/apache/pulsar/pull/24560)) \[fix]\[broker] Fix maxTopicsPerNamespace might report a false failure

([#24505](https://github.com/apache/pulsar/pull/24505)) \[fix]\[test]fix flaky test BrokerServiceAutoTopicCreationTest.testDynamicConfigurationTopicAutoCreationPartitioned

([#24472](https://github.com/apache/pulsar/pull/24472)) \[fix] Prevent IllegalStateException: Field 'message' is not set

([#24542](https://github.com/apache/pulsar/pull/24542)) \[fix]\[broker]Fix thread safety issues in BucketDelayedDeliveryTracker with StampedLock optimistic reads

([#24551](https://github.com/apache/pulsar/pull/24551)) \[fix]\[broker] Fix Broker OOM due to too many waiting cursors and reuse a recycled OpReadEntry incorrectly

([#24511](https://github.com/apache/pulsar/pull/24511)) \[fix]\[broker] Fix deduplication replay might never complete for exceptions

([#24522](https://github.com/apache/pulsar/pull/24522)) \[fix]\[ml] Fix the possibility of message loss or disorder when ML PayloadProcessor processing fails

([#24451](https://github.com/apache/pulsar/pull/24451)) \[fix]\[test]\[branch-3.0] Correct topic policy loading logic and improve related tests

([#24552](https://github.com/apache/pulsar/pull/24552)) \[improve]\[test] Remove EntryCacheCreator from ManagedLedgerFactoryImpl

([#24544](https://github.com/apache/pulsar/pull/24544)) \[improve] Upgrade pulsar-client-python to 3.8.0 in Docker image

([#24516](https://github.com/apache/pulsar/pull/24516)) \[fix]\[broker] Fix exclusive producer creation when last shared producer closes

([#24506](https://github.com/apache/pulsar/pull/24506)) \[fix]\[broker] Fix duplicate increment of ADD\_OP\_COUNT\_UPDATER in OpAddEntry

([#24543](https://github.com/apache/pulsar/pull/24543)) \[fix]\[broker] Fix matching of topicsPattern for topic names which contain non-ascii characters

([#24539](https://github.com/apache/pulsar/pull/24539)) \[fix]\[client] Close orphan producer or consumer when the creation is interrupted

([#24517](https://github.com/apache/pulsar/pull/24517)) \[fix]\[client] Fix ClientCnx handleSendError NPE

([#24515](https://github.com/apache/pulsar/pull/24515)) \[fix]\[ml] Fix asyncReadEntries might never complete if empty entries are read from BK

([#24525](https://github.com/apache/pulsar/pull/24525)) \[improve]\[misc] Optimize topic list hashing so that potentially large String allocation is avoided

([#24528](https://github.com/apache/pulsar/pull/24528)) \[fix]\[client] Fix issue in auto releasing of idle connection with topics pattern consumer

([#24529](https://github.com/apache/pulsar/pull/24529)) \[fix]\[proxy] Fix default value of connectionMaxIdleSeconds in Pulsar Proxy

([#24476](https://github.com/apache/pulsar/pull/24476)) \[fix]\[client] NPE in MultiTopicsConsumerImpl.negativeAcknowledge

([#24465](https://github.com/apache/pulsar/pull/24465)) \[fix]\[proxy] Fix proxy OOM by replacing TopicName with a simple conversion method

([#22495](https://github.com/apache/pulsar/pull/22495)) \[fix]\[test] Move ExtensibleLoadManagerImplTest to flaky tests

([#21642](https://github.com/apache/pulsar/pull/21642)) \[fix]\[test] Fix flaky test SimpleProducerConsumerStatTest#testPartitionTopicStats

([#24453](https://github.com/apache/pulsar/pull/24453)) \[fix]\[broker] replication does not work due to the mixed and repetitive sending of user messages and replication markers

([#24424](https://github.com/apache/pulsar/pull/24424)) \[fix]\[broker] Fix the non-persistenttopic's replicator always get error "Producer send queue is full" if set a small value of the config replicationProducerQueueSize

([#24189](https://github.com/apache/pulsar/pull/24189)) \[fix]\[broker]excessive replication speed leads to error: Producer send queue is full

([#22674](https://github.com/apache/pulsar/pull/22674)) \[Fix]\[broker] Limit replication rate based on bytes

([#20931](https://github.com/apache/pulsar/pull/20931)) \[fix]\[broker] Fix ack hole in cursor for geo-replication

([#24443](https://github.com/apache/pulsar/pull/24443)) \[fix]\[txn] Fix negative unacknowledged messages in transactions by ensuring that the batch size is added into CommandAck

([#24421](https://github.com/apache/pulsar/pull/24421)) \[fix]\[build] Add missing `<name>` to submodules

([#24441](https://github.com/apache/pulsar/pull/24441)) \[fix]\[ml] Enhance OpFindNewest to support skip non-recoverable data

([#24459](https://github.com/apache/pulsar/pull/24459)) \[improve]\[broker] change to warn log level for ack validation error

([#24434](https://github.com/apache/pulsar/pull/24434)) \[improve]\[broker] Improve the log when namespace bundle is not available

([#24432](https://github.com/apache/pulsar/pull/24432)) \[fix]\[ml]Still got BK ledger, even though it has been deleted after offloaded

([#21467](https://github.com/apache/pulsar/pull/21467)) \[fix]\[test] Cleanup resources if starting PulsarService fails in PulsarTestContext

([#24351](https://github.com/apache/pulsar/pull/24351)) \[improve]\[broker] Deny removing local cluster from topic level replicated cluster policy

([#24419](https://github.com/apache/pulsar/pull/24419)) \[fix]\[broker] Once the cluster is configured incorrectly, the broker maintains the incorrect cluster configuration even if you removed it

([#24404](https://github.com/apache/pulsar/pull/24404)) \[fix]\[client] Prevent NPE when seeking with null topic in TopicMessageId

([#24405](https://github.com/apache/pulsar/pull/24405)) \[fix]\[ml]Received more than once callback when calling cursor.delete

([#24406](https://github.com/apache/pulsar/pull/24406)) \[fix]\[ml] Cursor ignores the position that has an empty ack-set if disabled deletionAtBatchIndexLevelEnabled

([#24401](https://github.com/apache/pulsar/pull/24401)) \[fix]\[txn] Fix deadlock when loading transaction buffer snapshot

([#24402](https://github.com/apache/pulsar/pull/24402)) \[fix]\[client] Fix some potential resource leak

([#20629](https://github.com/apache/pulsar/pull/20629)) \[improve]\[test] Fix flaky test SimpleProducerConsumerStatTest#testMsgRateExpired

([#21629](https://github.com/apache/pulsar/pull/21629)) \[fix]\[build] Fix potential insufficient protostuff-related configs

([#23594](https://github.com/apache/pulsar/pull/23594)) \[fix] \[broker] No longer allow creating subscription that contains slash

([#24366](https://github.com/apache/pulsar/pull/24366)) \[fix]\[broker]Fix deadlock when compaction and topic deletion execute concurrently

([#24350](https://github.com/apache/pulsar/pull/24350)) \[fix]\[broker] Fix issue that topic policies was deleted after a sub topic deleted, even if the partitioned topic still exists

([#24384](https://github.com/apache/pulsar/pull/24384)) \[fix]\[ml]Revert a behavior change of releasing idle offloaded ledger handle: only release idle BlobStoreBackedReadHandle

([#24397](https://github.com/apache/pulsar/pull/24397)) \[improve]\[misc] Upgrade Netty to 4.1.122.Final and tcnative to 2.0.72.Final

([#24391](https://github.com/apache/pulsar/pull/24391)) \[improve]\[broker] Add managedCursor/LedgerInfoCompressionType settings to broker.conf

([#24392](https://github.com/apache/pulsar/pull/24392)) \[improve]\[broker] Make maxBatchDeletedIndexToPersist configurable and document other related configs

([#24386](https://github.com/apache/pulsar/pull/24386)) \[improve]\[broker] Added synchronized for sendMessages in Non-Persistent message dispatchers

([#24381](https://github.com/apache/pulsar/pull/24381)) \[improve]\[ml]Release idle offloaded read handle only the ref count is 0

([#19783](https://github.com/apache/pulsar/pull/19783)) \[improve]\[offloaders] Automatically evict Offloaded Ledgers from memory

([#24360](https://github.com/apache/pulsar/pull/24360)) \[fix]\[broker] expose consumer name for partitioned topic stats

([#24359](https://github.com/apache/pulsar/pull/24359)) \[improve]\[broker]Improve the log when encountered in-flight read limitation

([#24354](https://github.com/apache/pulsar/pull/24354)) \[fix]\[io] Acknowledge RabbitMQ message after processing the message successfully

([#24352](https://github.com/apache/pulsar/pull/24352)) \[fix]\[broker] Ignore metadata changes when broker is not in the Started state

([#24190](https://github.com/apache/pulsar/pull/24190)) \[fix]\[broker] Resolve the issue of frequent updates in message expiration deletion rate

([#24338](https://github.com/apache/pulsar/pull/24338)) \[fix]\[ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent issue

([#24331](https://github.com/apache/pulsar/pull/24331)) \[fix]\[offload] Complete the future outside of the reading loop in BlobStoreBackedReadHandleImplV2.readAsync

([#24324](https://github.com/apache/pulsar/pull/24324)) \[fix]\[test] Fix flaky AutoScaledReceiverQueueSizeTest.testNegativeClientMemory

([#24316](https://github.com/apache/pulsar/pull/24316)) \[fix]\[io] Fix kinesis avro bytes handling

([#24344](https://github.com/apache/pulsar/pull/24344)) \[improve]\[ml] Offload ledgers without check ledger length

([#24286](https://github.com/apache/pulsar/pull/24286)) \[fix]\[broker]Non-global topic policies and global topic policies overwrite each other

([#24279](https://github.com/apache/pulsar/pull/24279)) \[fix]\[broker]Global topic policies do not affect after unloading topic and persistence global topic policies never affect

([#24349](https://github.com/apache/pulsar/pull/24349)) \[fix]\[io]\[branch-3.0] Backport Kinesis Sink custom native executable support #23762

([#24317](https://github.com/apache/pulsar/pull/24317)) \[fix]\[io]\[branch-3.0]Pulsar-SQL: Fix classcast ex when decode decimal value

([#24313](https://github.com/apache/pulsar/pull/24313)) \[fix]\[broker] Fix potential deadlock when creating partitioned topic

([#24293](https://github.com/apache/pulsar/pull/24293)) \[fix]\[broker] fix wrong method name checkTopicExists.

([#24307](https://github.com/apache/pulsar/pull/24307)) \[fix]\[build] Ensure that buildtools is Java 8 compatible and fix remaining compatibility issue

([#24304](https://github.com/apache/pulsar/pull/24304)) \[fix]\[test] Simplify BetweenTestClassesListenerAdapter and fix issue with BeforeTest/AfterTest annotations

([#24289](https://github.com/apache/pulsar/pull/24289)) \[improve]\[io] Add configuration parameter for disabling aggregation for Kinesis Producers

([#24302](https://github.com/apache/pulsar/pull/24302)) \[improve] Upgrade pulsar-client-python to 3.7.0 in Docker image

([#24299](https://github.com/apache/pulsar/pull/24299)) \[fix]\[test] Fix more Netty ByteBuf leaks in tests

([#24297](https://github.com/apache/pulsar/pull/24297)) \[fix]\[io] Fix SyntaxWarning in Pulsar Python functions

([#24282](https://github.com/apache/pulsar/pull/24282)) \[fix]\[client] Fix producer publishing getting stuck after message with incompatible schema is discarded

([#24283](https://github.com/apache/pulsar/pull/24283)) \[cleanup]\[test] Remove unused parameter from deleteNamespaceWithRetry method in MockedPulsarServiceBaseTest

([#24263](https://github.com/apache/pulsar/pull/24263)) \[improve]\[build] Upgrade zstd version from 1.5.2-3 to 1.5.7-3

([#24281](https://github.com/apache/pulsar/pull/24281)) \[fix]\[test] Fix multiple ByteBuf leaks in tests

([#24275](https://github.com/apache/pulsar/pull/24275)) \[fix]\[broker] Fix HashedWheelTimer leak in PulsarService by stopping it in shutdown

([#24274](https://github.com/apache/pulsar/pull/24274)) \[fix]\[misc] Fix ByteBuf leak in SchemaUtils

([#24254](https://github.com/apache/pulsar/pull/24254)) \[fix]\[broker]Fix incorrect priority between topic policies and global topic policies

([#24266](https://github.com/apache/pulsar/pull/24266)) \[improve]\[ci] Disable detailed console logging for integration tests in CI

([#24261](https://github.com/apache/pulsar/pull/24261)) \[fix]\[test] Fix flaky ManagedCursorTest.testLastActiveAfterResetCursor and disable failing SchemaTest

([#24244](https://github.com/apache/pulsar/pull/24244)) \[fix]\[test] Fix flaky ManagedCursorTest.testSkipEntriesWithIndividualDeletedMessages

([#24248](https://github.com/apache/pulsar/pull/24248)) \[improve]\[io]\[kca] support fully-qualified topic names in source records

([#24260](https://github.com/apache/pulsar/pull/24260)) \[improve]\[build] Upgrade Gradle Develocity Maven Extension dependencies

([#24258](https://github.com/apache/pulsar/pull/24258)) \[fix]\[test] Fix TestNG BetweenTestClassesListenerAdapter listener

([#24257](https://github.com/apache/pulsar/pull/24257)) \[fix]\[broker] Unregister non-static metrics collectors registered in Prometheus default registry

([#24178](https://github.com/apache/pulsar/pull/24178)) \[fix]\[broker]fix memory leak, messages lost, incorrect replication state if using multiple schema versions(auto\_produce)

([#24219](https://github.com/apache/pulsar/pull/24219)) \[improve]\[broker]Improve the feature "Optimize subscription seek (cursor reset) by timestamp": search less entries

([#23919](https://github.com/apache/pulsar/pull/23919)) \[fix]\[broker] Fix seeking by timestamp can be reset the cursor position to earliest

([#22792](https://github.com/apache/pulsar/pull/22792)) \[improve]\[broker] Optimize subscription seek (cursor reset) by timestamp

([#24243](https://github.com/apache/pulsar/pull/24243)) \[improve]\[build] Upgrade SpotBugs to 4.9.x

([#24240](https://github.com/apache/pulsar/pull/24240)) \[improve]\[build] Upgrade to jacoco 0.8.13

([#24237](https://github.com/apache/pulsar/pull/24237)) \[improve]\[build] Upgrade Lombok to 1.18.38 to support JDK 24

([#24221](https://github.com/apache/pulsar/pull/24221)) \[improve]\[io] support kafka connect transforms and predicates

([#24230](https://github.com/apache/pulsar/pull/24230)) \[improve]\[client]Improve transaction log when a TXN command timeout

([#24223](https://github.com/apache/pulsar/pull/24223)) \[fix]\[broker] Orphan schema after disabled a cluster for a namespace

([#24228](https://github.com/apache/pulsar/pull/24228)) \[fix]\[broker] Fix ByteBuf memory leak in REST API for publishing messages

([#24184](https://github.com/apache/pulsar/pull/24184)) \[fix]\[client] Fix incorrect producer.getPendingQueueSize due to incomplete queue implementation

([#24214](https://github.com/apache/pulsar/pull/24214)) \[improve] Upgrade Netty to 4.1.121.Final

([#24212](https://github.com/apache/pulsar/pull/24212)) \[fix]\[test] Fix flaky BatchMessageWithBatchIndexLevelTest.testBatchMessageAck

([#24218](https://github.com/apache/pulsar/pull/24218)) \[fix]\[test] Fix multiple resource leaks in tests

([#24187](https://github.com/apache/pulsar/pull/24187)) \[improve]\[client] validate ClientConfigurationData earlier to avoid resource leaks

([#24216](https://github.com/apache/pulsar/pull/24216)) \[fix]\[broker] Fix HealthChecker deadlock in shutdown

([#24209](https://github.com/apache/pulsar/pull/24209)) \[fix]\[broker] Fix tenant creation and update with null value

([#24192](https://github.com/apache/pulsar/pull/24192)) \[fix]\[admin] Backlog quota's policy is null which causes a NPE

([#24210](https://github.com/apache/pulsar/pull/24210)) \[fix]\[broker] Fix broker shutdown delay by resolving hanging health checks

([#24207](https://github.com/apache/pulsar/pull/24207)) \[fix]\[broker] Fix compaction service log's wrong condition

([#24204](https://github.com/apache/pulsar/pull/24204)) \[fix]\[test] Fix resource leaks in ProxyTest and fix invalid tests

([#24201](https://github.com/apache/pulsar/pull/24201)) \[improve]\[io] Upgrade Kafka client and compatible Confluent platform version

([#24118)](https://github.com/apache/pulsar/pull/24118))) Revert "\[fix]\[broker] Add topic consistency check

([#24154)](https://github.com/apache/pulsar/pull/24154))) Revert "\[fix]\[broker] Directly query single topic existence when the topic is partitioned

([#24032](https://github.com/apache/pulsar/pull/24032)) \[fix]\[broker] Fix missing validation when setting retention policy on topic level

([#24098](https://github.com/apache/pulsar/pull/24098)) \[fix]\[ml] Skip deleting cursor if it was already deleted before calling unsubscribe

([#24181](https://github.com/apache/pulsar/pull/24181)) \[fix]\[proxy] Fix incorrect client error when calling get topic metadata

([#24158](https://github.com/apache/pulsar/pull/24158)) \[fix]\[proxy] Propagate client connection feature flags through Pulsar Proxy to Broker

([#24103](https://github.com/apache/pulsar/pull/24103)) \[fix]\[schema] Reject unsupported Avro schema types during schema registration

([#24091](https://github.com/apache/pulsar/pull/24091)) \[fix]\[broker] Fix some problems in calculate totalAvailableBookies in method getExcludedBookiesWithIsolationGroups when some bookies belongs to multiple isolation groups.

([#21320](https://github.com/apache/pulsar/pull/21320)) \[fix]\[bk] Fix the var name for IsolationGroups

([#24171](https://github.com/apache/pulsar/pull/24171)) \[improve]\[test] Use configured session timeout for MockZooKeeper and TestZKServer in PulsarTestContext

([#24172](https://github.com/apache/pulsar/pull/24172)) \[fix]\[test] Improve reliability of IncrementPartitionsTest

([#24170](https://github.com/apache/pulsar/pull/24170)) \[fix]\[test]flaky-test:ManagedLedgerInterceptorImplTest.testManagedLedgerPayloadInputProcessorFailure

([#23980](https://github.com/apache/pulsar/pull/23980)) \[fix]\[broker] Consumer stuck when delete subscription \_\_compaction failed

([#24167](https://github.com/apache/pulsar/pull/24167)) \[fix]\[ml] Fix ML thread blocking issue in internalGetPartitionedStats API

([#24166](https://github.com/apache/pulsar/pull/24166)) \[fix]\[test] Fix invalid test CompactionTest.testDeleteCompactedLedgerWithSlowAck

([#24150](https://github.com/apache/pulsar/pull/24150)) \[fix]\[broker] The feature brokerDeleteInactivePartitionedTopicMetadataEnabled leaves orphan topic policies and topic schemas

([#24154](https://github.com/apache/pulsar/pull/24154)) \[fix]\[broker] Directly query single topic existence when the topic is partitioned

([#24118](https://github.com/apache/pulsar/pull/24118)) \[fix]\[broker] Add topic consistency check

([#24056](https://github.com/apache/pulsar/pull/24056)) \[fix]\[test] Update partitioned topic subscription assertions in IncrementPartitionsTest

([#24033](https://github.com/apache/pulsar/pull/24033)) \[cleanup]\[misc] Add override annotation

([#24161](https://github.com/apache/pulsar/pull/24161)) \[fix]\[test] Fix flaky BrokerServiceChaosTest.testFetchPartitionedTopicMetadataWithCacheRefresh

([#24162](https://github.com/apache/pulsar/pull/24162)) \[fix]\[test] Fix flaky BrokerServiceChaosTest

([#24097](https://github.com/apache/pulsar/pull/24097)) \[fix] \[broker] topics infinitely failed to delete after remove cluster from replicated clusters modifying when using partitioned system topic

([#22261](https://github.com/apache/pulsar/pull/22261)) \[fix] Bump google.golang.org/protobuf from 1.32.0 to 1.33.0 in /pulsar-function-go

([#24132](https://github.com/apache/pulsar/pull/24132)) \[fix]\[io] Fix KinesisSink json flattening for AVRO's SchemaType.BYTES

([#20984](https://github.com/apache/pulsar/pull/20984)) \[fix]\[broker] Fix get outdated compactedTopicContext after compactionHorizon has been updated

([#20697](https://github.com/apache/pulsar/pull/20697)) \[improve]\[broker] Improve CompactedTopicImpl lock

([#24131](https://github.com/apache/pulsar/pull/24131)) \[fix]\[ml] Return 1 when bytes size is 0 or negative for entry count estimation

([#24128](https://github.com/apache/pulsar/pull/24128)) \[improve]\[io] Enhance Kafka connector logging with focused bootstrap server information

([#24125](https://github.com/apache/pulsar/pull/24125)) \[fix]\[ml] Don't estimate number of entries when ledgers are empty, return 1 instead

([#24123](https://github.com/apache/pulsar/pull/24123)) \[improve]\[client] Prevent NullPointException when closing ClientCredentialsFlow

([#24124](https://github.com/apache/pulsar/pull/24124)) \[improve]\[io] Remove sleep when sourceTask.poll of kafka return null

([#24116](https://github.com/apache/pulsar/pull/24116)) \[improve]\[broker] Change topic exists log to warn

([#24104](https://github.com/apache/pulsar/pull/24104)) \[fix]\[client] Pattern subscription regression when broker-side evaluation is disabled

([#24100](https://github.com/apache/pulsar/pull/24100)) \[fix]\[client] Fix consumer leak when thread is interrupted before subscribe completes

([#24089](https://github.com/apache/pulsar/pull/24089)) \[fix]\[ml] Fix issues in estimateEntryCountBySize

([#24073](https://github.com/apache/pulsar/pull/24073)) \[improve]\[broker] Optimize message expiration rate repeated update issues

([#24087](https://github.com/apache/pulsar/pull/24087)) \[fix]\[broker] Avoid IllegalStateException when marker\_type field is not set in publishing

([#24083](https://github.com/apache/pulsar/pull/24083)) \[fix]\[ci] Bump dependency-check to 12.1.0 to fix OWASP Dependency Check job

([#24082](https://github.com/apache/pulsar/pull/24082)) \[clean]\[client] Clean code for the construction of retry/dead letter topic name

([#24079](https://github.com/apache/pulsar/pull/24079)) \[fix]\[broker] Fix NPE while publishing Metadata-Event with not init producer

([#24080](https://github.com/apache/pulsar/pull/24080)) \[fix]\[broker] Fix Metadata event synchronizer should not fail with bad version

([#24081](https://github.com/apache/pulsar/pull/24081)) \[fix]\[broker] Fix Metadata Event Synchronizer producer creation retry so that the producer gets created eventually

([#24048](https://github.com/apache/pulsar/pull/24048)) \[fix]\[broker] Fix UnsupportedOperationException while setting subscription level dispatch rate policy

([#24054](https://github.com/apache/pulsar/pull/24054)) \[fix]\[ml] Corrected pulsar\_storage\_size metric to not multiply offloaded storage by the write quorum

([#24067](https://github.com/apache/pulsar/pull/24067)) \[fix]\[broker] http metric endpoint get compaction latency stats always be 0

([#24064](https://github.com/apache/pulsar/pull/24064)) \[improve]\[broker] Optimize ThresholdShedder with improved boundary checks and parameter reuse

([#24055](https://github.com/apache/pulsar/pull/24055)) \[fix] Avoid negative estimated entry count

([#24060](https://github.com/apache/pulsar/pull/24060)) \[improve]\[monitor] Add version=0.0.4 to /metrics content type for Prometheus 3.x compatibility

([#24059](https://github.com/apache/pulsar/pull/24059)) \[fix]\[client] Copy eventTime to retry letter topic and DLQ messages

([#24061](https://github.com/apache/pulsar/pull/24061)) \[fix]\[client] Fix building broken batched message when publishing

([#24063](https://github.com/apache/pulsar/pull/24063)) \[fix]\[broker]Fix failed consumption after loaded up a terminated topic

([#24072](https://github.com/apache/pulsar/pull/24072)) \[fix]\[broker] Pattern subscription doesn't work when the pattern excludes the topic domain.

([#24049](https://github.com/apache/pulsar/pull/24049)) \[improve] Upgrade Netty to 4.1.119.Final

([#23975](https://github.com/apache/pulsar/pull/23975)) \[fix]\[broker] Add expire check for replicator

([#24023](https://github.com/apache/pulsar/pull/24023)) \[fix]\[doc] fix doc related to chunk message feature.

([#23962](https://github.com/apache/pulsar/pull/23962)) \[improve]\[ml] Use lock-free queue in InflightReadsLimiter since there's no concurrent access

([#23978](https://github.com/apache/pulsar/pull/23978)) \[improve]\[cli] Support additional msg metadata for V1 topic on peek message cmd

([#24014](https://github.com/apache/pulsar/pull/24014)) \[fix]\[broker] Fix BucketDelayedDeliveryTracker thread safety

([#24019](https://github.com/apache/pulsar/pull/24019)) \[fix]\[test]Fix flaky test V1\_ProducerConsumerTest.testConcurrentConsumerReceiveWhileReconnect

([#24011](https://github.com/apache/pulsar/pull/24011)) \[fix]\[test] Fix flaky test OneWayReplicatorUsingGlobalZKTest.testConfigReplicationStartAt

([#23931](https://github.com/apache/pulsar/pull/23931)) \[improve] \[broker] Make the estimated entry size more accurate

([#24004](https://github.com/apache/pulsar/pull/24004)) \[improve]\[ci] Upgrade Gradle Develocity Maven Extension to 1.23.1

([#23697](https://github.com/apache/pulsar/pull/23697)) \[fix]\[broker] Geo Replication lost messages or frequently fails due to Deduplication is not appropriate for Geo-Replication

([#24006](https://github.com/apache/pulsar/pull/24006)) \[fix]\[broker] fix broker identifying incorrect stuck topic

([#23286](https://github.com/apache/pulsar/pull/23286)) \[improve]\[broker] Fix non-persistent system topic schema compatibility

([#23881](https://github.com/apache/pulsar/pull/23881)) \[improve]\[fn] Set default tenant and namespace for ListFunctions cmd

([#23730](https://github.com/apache/pulsar/pull/23730)) \[fix]\[admin] Verify is policies read only before revoke permissions on topic

([#24003](https://github.com/apache/pulsar/pull/24003)) \[improve]\[test] Upgrade Testcontainers to 1.20.4 and docker-java to 3.4.0

### StreamNative Pulsar Plugins

bc332df40 Fix snrbac plugin test zk mock

Fix LicenseAdditionalServletTest

fix export duplicated JVM metrics on AuditLogMetrics

### pulsarctl

e7d5e82 Use snstage docker image

Fix jwt cve

fix code check

feat: Subscription get message by id json output

Update subscription get message by id typo lederId to ledgerId

fix: upgrade golang version to fix CVE

Setup go version to 1.22 fix cve

update pulsar-client-go to master latest commit

2af1258 fix ci

Bump the pulsar-client-go to the master version

6f25051 Fix TestDeleteNonExistPartitionedTopic

Fix json marshal error for Secrets and UserConfigs when creating/updating functions

Support create token with headers

Upgrade the dependency version to fix vulnerabilities

Add trivy scan workflow to avoid vulnerabilities

\[fix] Upgrade go version to 1.21 to fix CVE-2023-24538

fix source test typo

fix source test

Auth SN docker hub

Support no auth context

fix token

Add docker hub login

Auth SN docker hub

5cb0593 Disable bk unit test and fix it later ---

Add method to mark bookie readonly

Build arm64 linux executable binary artifact

Update jose2go to fix GHSA-mhpq-9638-x6pw

Update golang.org/x/net

Replace apache pulsar client go repo on 3.0 branch

Fixed remove auth plugin suffix

Removed error char

Bump pulsar version to 3.0.0.1

fix: Fix TestUpdateTopicNotExist and TestUpdateNonPartitionedTopic

### Function Mesh Worker Service

feat: support multiple mcp servers

Support trigger agent function with properties

Update function-mesh version to v0.25.0 in pom.xml

00da1b9b Fix ci

Remove ConnectRestException from mesh-worker-common module

Support input-type-class and output-type-class arguments for Functions

Support set extra env for kafka connect

Support streamable http for AgentFunction and make trigger timeout value configurable

Create a new sub module mesh-worker-common

Generate OpenAPI docs for agent-functions

1c20298f Update pulsar version to 3.0.10.6

3b1c05a8 Cleanup disk

c60e56c5 Update MeshWorkerServer

35aa29bc Fix version

Support set agent tools config

Make MeshWorker able to run standalone and load additional servlets

Support load ConnectorCatalog using label

Update error msg in status

6724eb27 Update version when release

Implement agent function


# V3.0.2.1
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.2.1



# StreamNative Weekly Release Notes v3.0.2.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.2.1](https://github.com/streamnative/pulsar/releases/tag/v3.0.2.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.2.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.2.1/images/sha256-789103fa36dc68827de1b1a0e9b714d452de5b7e0b1fdb0f73cb3f934b626f73)

## General Changes

### Apache Pulsar

\[fix]\[broker] Fix typo in the config key

\[fix]\[offload] Don't cleanup data when offload met MetaStore exception

\[improve]\[broker] Support not retaining null-key message during topic compaction

\[fix]\[broker] Do not write replicated snapshot marker when the topic which is not enable replication

\[fix]\[build] Fix Stage Docker images fail on M1 Mac

\[improve]\[broker] Print recoverBucketSnapshot log if cursorProperties are empty

\[improve]\[admin] Add clusters check when set replication clusters

\[fix]\[broker] Fix lookupRequestSemaphore leak when topic not found

\[fix]\[broker] Fix memory leak during topic compaction

\[fix]\[admin] Fix KeyValue schema compatibility check caused OOM

\[fix]\[broker] Fixed getting incorrect KeyValue schema version

\[fix]\[broker] Fix incorrect unack count when using shared subscription on non-persistent topic

\[cleanup]\[client] Fix inconsistent API annotations of `getTopicName`

### StreamNative Pulsar Plugins

42554715 Fixed new pom versions

Fix the packages cloud storage failed to find gs schema

Replace GCS hadoop connector shaded artifact

Replace hadoop-common guava-shaded dependency

\[pulsarctl-plugin] Bump client-go to `0.20.15`

### Aws EventBridge Connector

Refactor create a connector section docs.


# V3.0.2.2
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.2.2



# StreamNative Weekly Release Notes v3.0.2.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.2.2](https://github.com/streamnative/pulsar/releases/tag/v3.0.2.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.2.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.2.2/images/sha256-83dc9427045983c299dbad84f47c14bd8d22e95ad3e302805ff35392a2e1e218)

## General Changes

### Apache Pulsar

\[fix]\[test]\[branch-3.0] fix testCleanupEmptySubscriptionAuthenticationMap

\[fix]\[sec] Upgrade org.bouncycastle:bc-fips to 1.0.2.4

\[fix]\[sec] Exclude avro from hadoop-client

\[fix]\[broker] Record GeoPersistentReplicator.msgOut before producer#sendAsync

\[improve]\[broker] cleanup the empty subscriptionAuthenticationMap in zk when revoke subscription permission

\[fix]\[fn] Fix Deadlock in Functions Worker LeaderService

\[fix]\[test] Fix PerformanceProducer send count error

\[fix] \[broker] network package lost if enable haProxyProtocolEnabled

### KoP

Support configuring kopAllowedNamespaces dynamically

SNIP-112: Return short topic names for OffsetFetch requests without topics

\[Proxy] Fix thread safety issues in BrokerConnectionGroup

Add Proxy metrics

\[improve] Gets the eventExecutor from the request context to register event instead of the fixed eventExecutor

### Cloud Storage Connector

Update nick-invision to nick-fields

397926d Fix format errors for note.

Fix typos in doc

Refactor create a connector section docs.

### AMQP1\_0 Connector

6bbd23b Fix format errors for note.

Refactor create a connector section docs.

Fix not success upload image

### AWS SQS Connector

Update nick-invision to nick-fields

2e7906a Fix format errors for note.

Refactor create a connector section docs.

### AWS Lambda Connector

Update nick-invision to nick-fields

### StreamNative Pulsar Plugins

b2fc57e0 release semaphoreToPulsar in the 'finally' code block.

4109099f Fix backup tool compress problems.

### Function Mesh Worker Service

Support load docsLink and iconLink for connector catalog.

update retry github action owner

### Google Pub / Sub Connector

Update nick-invision to nick-fields

### Google BigQuery Sink Connector

Update nick-invision to nick-fields

221e7d1 Fix format errors for note.

Refactor create a connector section docs.

### Snowflake Connector

Update nick-invision to nick-fields

Refactor connector creation doc

### Aws EventBridge Connector

Update nick-invision to nick-fields

Fix typos in doc

### Activemq Connector

Update nick-invision to nick-fields


# V3.0.2.3
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.2.3



# StreamNative Weekly Release Notes v3.0.2.3

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.2.3](https://github.com/streamnative/pulsar/releases/tag/v3.0.2.3)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.2.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.2.3/images/sha256-2dfd0c568bbcdaf34593ee3ceb304837a01712c5bcc81b6f4a756815da5dbb3a)

## General Changes

### Apache Pulsar

\[fix]\[broker] Fix TableViewLoadDataStoreImpl NPE

\[fix] \[broker] Update topic policies as much as possible when some ex was thrown

\[fix]\[broker] Fixed the ExtensibleLoadManagerImpl internal system getTopic failure when the leadership changes #21764

\[fix]\[broker] Skip topic auto-creation for ExtensibleLoadManager internal topics

\[fix]\[broker] Fixed ServiceUnitStateChannel monitor to tombstone only inactive bundle states

4c915bdcdd Upgrade OWASP dependency check maven plugin version

\[fix]\[client] Fix producer thread block forever on memory limit controller

\[fix]\[broker] Fix the issue of topics possibly being deleted.

### AMQP1\_0 Connector

Fix incorrect docker images link.

### StreamNative Pulsar Plugins

Release SN-RBAC to branch-3.0

Update go x/crypto to 0.17 to fix CVE-2023-48795

### Cloud Pulsar Plugins

Fix build.sh

Release apikeys\&oauth2 module first in build.sh

Fix test pom

Release SN-RBAC to branch-3.0

### Function Mesh Worker Service

add transformFunctionEnabled and disabled by default


# V3.0.2.4
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.2.4



# StreamNative Weekly Release Notes v3.0.2.4

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.2.4](https://github.com/streamnative/pulsar/releases/tag/v3.0.2.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.2.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.2.4/images/sha256-d5205e703b6e1602cacb3708ab0abd18276997a2f326f7424bbf97c79a2eb747)

## General Changes

### Apache Pulsar

\[improve]\[io] Make connectors load sensitive fields from secrets

\[fix]\[broker] Avoid compaction task stuck when the last message to compact is a marker

\[improve]\[proxy] Fix comment about enableProxyStatsEndpoints

\[fix]\[broker] fix the wrong value of BrokerSrevice.maxUnackedMsgsPerDispatcher

\[improve] \[client] Prevent reserve memory with a negative memory size to avoid send task stuck

\[fix]\[broker]Fix NonPersistentDispatcherMultipleConsumers ArrayIndexOutOfBoundsException

\[fix]\[broker] Fix compaction/replication data loss when expire messages

\[improve]\[broker] Skip loading the NAR packages if not configured

\[improve]\[broker] Improve NamespaceUnloadStrategy error message

\[fix]\[client] Fix messages in the batch container timed out unexpectedly

\[fix] \[broker] Fix break change: could not subscribe partitioned topic with a suffix-matched regexp due to a mistake of PIP-145

\[fix] \[client] Messages lost due to TopicListWatcher reconnect

\[improve]\[broker] Don't rollover empty ledgers based on inactivity

\[fix]\[broker] Delete compacted ledger when topic is deleted

\[fix] \[ml] Fix retry mechanism of deleting ledgers to invalidate

\[improve]\[broker] defer the ownership checks if the owner is inactive (ExtensibleLoadManager)

### KoP

Fix flaky testPublishTimestampInBatch

Fix offset commit timeout error due to incorrect send timer implementation

Increase the default maxReadEntriesNum to 50

Make loading offsets synchronous to avoid race condition

Add KSN proxy dashboard

Remove unused CI workflows

### pulsarctl

Update jose2go to fix GHSA-mhpq-9638-x6pw

### StreamNative Pulsar Plugins

\[fix]\[cve] Exclude logback from zookeeper

\[fix]\[detector] Cleanup inactive broker's e2e detector

Include one older txn log file in the backup when needed

Update jose2go to 1.6.0 to address GHSA-mhpq-9638-x6pw

Upgrade the version


# V3.0.2.5
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.2.5



# StreamNative Weekly Release Notes v3.0.2.5

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.2.5](https://github.com/streamnative/pulsar/releases/tag/v3.0.2.5)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.2.5/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.2.5/images/sha256-e0d81c11e548f4b7cda875043540e82947b536f611c33991029545932869f4f5)

## General Changes

### Apache Pulsar

\[improve]\[build] Add a default username in the image

\[fix]\[broker] Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set

### KoP

Attach Kafka txn\_metadata to messageMetadata in KafkaEntryFormatter

### StreamNative Pulsar Plugins

Enhance find orphan ledger command.

### Cloud Pulsar Plugins

Fix: return error future when JWT expired

Fix: do not throws EX when calling AuthorizationProviderOAuth.isSuperUser


# V3.0.2.6
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.2.6



# StreamNative Weekly Release Notes v3.0.2.6

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.2.6](https://github.com/streamnative/pulsar/releases/tag/v3.0.2.6)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.2.6/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.2.6/images/sha256-b3d32f35bfef45dac57d0f5c11d227628c0ae1fdef41ca1c0901b5f247cf84da)

## General Changes

### Apache Pulsar

04c2da4f20 Fix testNoCleanupOffloadLedgerWhenMetadataExceptionHappens

\[fix] \[ci] \[branch-3.0] Fix the build issue from cherry-pick

\[improve]\[ml] Filter out deleted entries before read entries from ledger.

\[improve]\[broker] Avoid record inactiveproducers when deduplication is disable.

\[fix]\[broker] Fix String wrong format

\[fix] \[broker] Fix write all compacted out entry into compacted topic

\[improve]\[ci] Upgrade pulsar-client-python to 3.4.0 to avoid CVE-2023-1428

\[fix]\[broker] Fix deadlock while skip non-recoverable ledgers.

\[fix]\[client] Fix multi-topics consumer could receive old messages after seek

\[fix]\[broker] Fix getMessageById throws 500

\[fix] \[broker] Replication stopped due to unload topic failed

\[fix]\[fn] Use unified PackageManagement service to download packages

\[improve] \[proxy] Add a check for brokerServiceURL that does not support multi uri yet

\[fix]\[broker] Fix schema deletion error when deleting a partitioned topic with many partitions and schema

\[fix]\[client] Fix ConsumerBuilderImpl#subscribe silent stuck when using pulsar-client:3.0.x with jackson-annotations prior to 2.12.0

\[fix] \[broker] add timeout for health check read.

\[improve]\[broker] Do not close the socket if lookup failed due to LockBusyException

\[fix] \[broker] \[branch-3.0] Fast fix infinite HTTP call createSubscriptions caused by wrong topicName

\[fix] \[bk] Fix the BookKeeper license

\[improve] \[bk] Upgrade BookKeeper dependency to 4.16.4

\[fix]\[test] Make base test class method protected so that it passes ReportUnannotatedMethods validation

\[fix]\[broker] Restore the broker id to match the format used in existing Pulsar releases

\[fix]\[broker] Fix leader broker cannot be determined when the advertised address and advertised listeners are configured

\[fix]\[broker] Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set

\[fix]\[broker] Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set

\[fix]\[broker] Fix PulsarService.getLookupServiceAddress returns wrong port if TLS is enabled

\[cleanup] Consolidate certs in broker (and some proxy) tests

\[fix]\[test] ProxyWithoutServiceDiscoveryTest should enable authz

([#21744)](https://github.com/apache/pulsar/pull/21744))) Revert "\[fix]\[test]\[branch-3.0] fix testCleanupEmptySubscriptionAuthenticationMap

\[cleanup] Consolidate certs used in tests

\[cleanup] Deduplicate test certificates to simplify management

([#21633)](https://github.com/apache/pulsar/pull/21633))) Revert "\[fix]\[broker] Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set

### MoP

Fix broker enable dedup cause client publish msg NPE

Add test for resubscribe

Add filter system topic when using EventCenter

Fix unsubscribe topic cause the test failed.

call equals on formatted strings since they will never be null

remove subs from subscription manager on unsubscribe

### KoP

\[test] Add list consumer group cli test

### Cloud Storage Connector

Update base image

Update base image

### AMQP1\_0 Connector

Update base image

### AWS SQS Connector

Update base image

### AWS Lambda Connector

update-base-image

### pulsarctl

Add method to mark bookie readonly

Build arm64 linux executable binary artifact

### StreamNative Pulsar Plugins

5398055b update

407d4941 update

Add arm64 artifacts release

### Function Mesh Worker Service

reduce integration test image size with slim base image

clean up the disk

bump function-mesh to 0.19.0

Ignore exception when connector customize catalogs is empty.

Use oxia:0.2 image for testing

### Google BigQuery Sink Connector

Update docker base

### Snowflake Connector

Update base image


# V3.0.2.8
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.2.8



# StreamNative Weekly Release Notes v3.0.2.8

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.2.8](https://github.com/streamnative/pulsar/releases/tag/v3.0.2.8)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.2.8/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.2.8/images/sha256-d7510486eaef226caa0e791c1768d94c61c0dd77a87643c1583c08dd4abcdd59)

## General Changes

### Apache Pulsar

55788734a5 Fix compile issue

\[fix] \[txn] Get previous position by managed ledger.

\[fix] \[broker] Subscription stuck due to called Admin API analyzeSubscriptionBacklog

\[fix]\[broker]Support setting `autoSkipNonRecoverableData` dynamically in expiryMon…

\[improve]\[broker] Do not retain the data in the system topic

\[fix] \[broker] Fix can not subscribe partitioned topic with a suffix-matched regexp

\[fix]\[broker] Fix hash collision when using a consumer name that ends with a number

\[fix]\[sec] Upgrade commons-compress to 1.26.0

\[fix]\[broker] Support running docker container with gid != 0

\[fix]\[broker]\[branch-3.1] Avoid PublishRateLimiter use an already closed RateLimiter


# V3.0.2.9
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.2.9



## StreamNative Weekly Release Notes v3.0.2.9

#### General Changes

### Apache Pulsar

\[fix] \[broker]\[branch-3.0] Expire messages according to ledger close time to avoid client clock skew

\[improve]\[fn]\[branch-3.0] Add missing "exception" argument to some `log.error`

\[improve]\[broker] Consistently add fine-grain authorization to REST API

\[improve] \[broker] Do not print an Error log when responding to `HTTP-404` when calling `Admin API` and the topic does not exist.

\[fix] \[branch-3.0] Fix reader stuck when read from compacted topic with read compact mode disable

\[fix]\[broker]\[branch-3.0] Avoid consumers receiving acknowledged messages from compacted topic after reconnection

\[fix]\[broker]\[branch-3.0] Fix broker not starting when both transactions and the Extensible Load Manager are enabled

\[fix]\[sec] Upgrade Jetty to 9.4.54.v20240208 to address CVE-2024-22201

\[fix]\[txn] Fix getting last message ID when there are ongoing transactions

\[improve]\[fn] Add configuration for connector & functions package url sources

\[improve]\[broker] Add fine-grain authorization to retention admin API

\[improve]\[admin]\[branch-3.0] Expose the offload threshold in seconds to the admin

Minor Compile fix

\[fix]\[txn]Fix TopicTransactionBuffer potential thread safety issue

\[fix]\[offload] Fix Offload readHandle cannot close multi times.

\[fix] \[broker] print non log when delete partitioned topic failed

([#22101)](https://github.com/apache/pulsar/pull/22101))) Revert "\[improve]\[admin] Expose the offload threshold in seconds to the amdin

\[fix]\[broker]\[branch-3.0] Return getOwnerAsync without waiting on source broker upon Assigning and Releasing and handle role change during role init

\[fix]\[broker]\[branch-3.0] Set ServiceUnitStateChannel topic compaction threshold explicitly, improve getOwnerAsync, and fix other bugs

\[improve]\[broker] Add an error log to troubleshoot the failure of starting broker registry.

\[fix]\[ml] Make mlOwnershipChecker asynchronous so that it doesn't block/deadlock threads

\[fix]\[test] fix test testSyncNormalPositionWhenTBRecover

\[fix]\[test] Fix test testAsyncFunctionMaxPending

\[fix]\[sec] Add a check for the input time value

\[fix] \[client] fix huge permits if acked a half batched message

\[fix] \[broker] Enabling batch causes negative unackedMessages due to ack and delivery concurrency

\[improve]\[broker] Cache the internal writer when sent to system topic.

\[improve] \[broker] Do not try to open ML when the topic meta does not exist and do not expect to create a new one. #21995

\[improve]\[admin] Expose the offload threshold in seconds to the amdin

\[fix]\[test] Fix test testTransactionBufferMetrics

\[improve]\[ci] Exclude jose4j to avoid CVE-2023-31582

\[fix] Bump org.apache.solr:solr-core from 8.11.1 to 8.11.3 in /pulsar-io/solr

e3f5115734 Fix byte-buddy version in presto LICENSE

\[improve]\[fn] Optimize Function Worker startup by lazy loading and direct zip/bytecode access

\[fix]\[broker] fix `Update contains no change` error when use `--update-auth-data` flag to update function/sink/source

\[fix] \[client] Do no retrying for error subscription not found when disabled allowAutoSubscriptionCreation

\[improve]\[proxy] When adding new brokers resolve the DNS name more quickly

### AoP

\[fix]\[test] Improve the declare exchange test

### KoP

Update LICENSE

### AWS Lambda Connector

Enable unit tests for weekly release

### pulsarctl

5cb0593 Disable bk unit test and fix it later ---

### StreamNative Pulsar Plugins

d76f2d3d \[fix]\[sec] Upgrade commons-compress to 1.26.0

### Function Mesh Worker Service

e8209687 Fix ci

474ff770 Deprecate classloader


# V3.0.3.1
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.3.1



## StreamNative Weekly Release Notes v3.0.3.1

#### General Changes

### Apache Pulsar

([#22023)](https://github.com/apache/pulsar/pull/22023))) Revert "\[fix]\[sec] Add a check for the input time value

### KoP

\[fix]\[transaction] Fix send messages with transaction in async way

### StreamNative Pulsar Plugins

Use an old version of the sn/charts

### Cloud Pulsar Plugins

ApiKeys: Avoid check JWT token expired time in authorization

\[SN-RBAC] added functions, sources, sinks interceptor path

Oauth2: Avoid check JWT token expired time in authorization

### Function Mesh Worker Service

d7b2f1da Avoid error in tune runner vm


# V3.0.3.2
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.3.2



## StreamNative Weekly Release Notes v3.0.3.2

#### General Changes

### MoP

Fix ClassCastException when scheduling to look up

### KoP

\[proxy] Fix duplicated sends when pending produce requests are ignored by network issue

### Function Mesh Worker Service

99d1a1b8 Cleanup disk

Validate functions\&connectors package url

### Google BigQuery Sink Connector

feat: Support protobuf native schema.


# V3.0.3.3
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.3.3



# StreamNative Weekly Release Notes v3.0.3.3

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.3.3](https://github.com/streamnative/pulsar/releases/tag/v3.0.3.3)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.3.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.3.3/images/sha256-ad3a4939b3e133380187233be5b2a4d91b660a45f8710ccaab36c5e7e8c12cdd)

## General Changes

### Apache Pulsar

\[improve]\[broker] Add fine-grain authorization to ns/topic management endpoints

e76ed4368a \[fix]\[ci]\[branch-3.0] Increase Maven's heap size from 1024m to 1500m as it is in master

6eb5068a89 \[fix]\[ci]\[branch-3.0] Increase Maven's heap size from 1024m to 1500m as it is in master

\[improve]\[broker] Add missing configuration keys for caching catch-up reads

\[improve]\[misc] Upgrade checkstyle to 10.14.2

### MoP

Fix proxy keepalive issue

### KoP

\[proxy] Fix duplicated sends when pending produce requests are ignored by network issue

### AMQP1\_0 Connector

Auth SN docker hub

### AWS SQS Connector

Auth SN docker hub

### pulsarctl

Auth SN docker hub

### StreamNative Pulsar Plugins

Auth SN dockerhub

### Function Mesh Worker Service

Check null value before use VpaSpec

Auth SN docker hub


# V3.0.3.4
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.3.4



# StreamNative Weekly Release Notes v3.0.3.4

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.3.4](https://github.com/streamnative/pulsar/releases/tag/v3.0.3.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.3.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.3.4/images/sha256-309f3e838a6dbe090bde10455d6bd07759a69b44d6d0af3c61b8d5a2eb4410bc)

## General Changes

### Apache Pulsar

\[fix]\[broker] Fix ResourceGroups loading

\[fix]\[broker] Fix ResourceGroup report local usage

\[fix] \[broker] fix mismatch between dispatcher.consumerList and dispatcher.consumerSet

\[fix] \[broker] Close dispatchers stuck due to mismatch between dispatcher.consumerList and dispatcher.consumerSet

\[fix] \[client] Unclear error message when creating a consumer with two same topics

### KoP

\[SNIP-122] Part 3: Support other admin protocols for dot-separated namespace prefix

### pulsarctl

Add docker hub login

### StreamNative Pulsar Plugins

Cherry pick license plugin feature to 3.0

### Function Mesh Worker Service

05df2d34 Add brokerAdditionalServlet

allow passing allowed runtimeFlags for java runtime


# V3.0.3.5
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.3.5



# StreamNative Weekly Release Notes v3.0.3.5

<Note title="Note">
  Please note this StreamNative Pulsar distribution will require a valid StreamNative subscription license key to run otherwise the image will fail to start.
</Note>

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.3.5](https://github.com/streamnative/pulsar/releases/tag/v3.0.3.5)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.3.5/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.3.5/images/sha256-16435693d857add05296f5507a72468a926fae909fe48272b265ecda7b194a9a)

## General Changes

### Apache Pulsar

\[improve]\[misc] Remove the call to sun InetAddressCachePolicy

\[fix]\[broker] Check cursor state before adding it to the `waitingCursors`

\[fix]\[client] Fix wrong results of hasMessageAvailable and readNext after seeking by timestamp

\[fix]\[broker] Avoid execute prepareInitPoliciesCacheAsync if namespace is deleted

\[fix]\[broker] Fix wrong double-checked locking for readOnActiveConsumerTask in dispatcher

\[fix]\[client] fix Reader.hasMessageAvailable might return true after seeking to latest

\[improve]\[client] Add backoff for `seek`

\[fix]\[misc] Make ConcurrentBitSet thread safe

\[fix]\[broker] Avoid expired unclosed ledgers when checking expired messages by ledger closure time

\[fix]\[test] Fix flaky RGUsageMTAggrWaitForAllMsgsTest

\[fix]\[client] Consumer lost message ack due to race condition in acknowledge with batch message

\[fix]\[broker] Fix OpReadEntry.skipCondition NPE issue

\[fix]\[test] Fix flaky ManagedLedgerErrorsTest.recoverAfterZnodeVersionError

\[fix] \[test] Fix flaky test ManagedLedgerTest.testGetNumberOfEntriesInStorage

\[fix]\[ml]Expose ledger timestamp

\[improve]\[misc] Include native epoll library for Netty for arm64

\[fix]\[client]Fixed getting an incorrect `maxMessageSize` value when accessing multiple clusters in the same process

\[improve]\[admin] Fix the `createMissingPartitions` doesn't response correctly

\[improve]\[misc] Upgrade Netty version to 4.1.105.Final

\[fix]\[sec] Go Functions security updates

\[fix]\[sec] Bump google.golang.org/grpc from 1.38.0 to 1.56.3 in /pulsar-function-go

\[fix]\[fn] enable Go function token auth and TLS

\[fix]\[sec] Upgrade prometheus client\_golang to v1.12.2 to fix CVE-2022-21698

### KoP

\[SNIP-122] Part 3: Support other admin protocols for dot-separated namespace prefix

### pulsarctl

Support no auth context

fix token

### StreamNative Pulsar Plugins

Update license message for branch-3.0


# V3.0.4.1
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.4.1



# StreamNative Weekly Release Notes v3.0.4.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.4.1](https://github.com/streamnative/pulsar/releases/tag/v3.0.4.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.4.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.4.1/images/sha256-de2f0beedde6364ca6243856a7be494318bbf236103ee2d88c1117808373b77d)

## General Changes

### Apache Pulsar

\[fix]\[broker] Skip topic.close during unloading if the topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to use ns.checkOwnershipPresentAsync for ExtensibleLoadBalancer

\[fix]\[build] Fix networkaddress.cache.negative.ttl config

\[improve]\[broker] Don't log brokerClientAuthenticationParameters and bookkeeperClientAuthenticationParameters by default

060bf61580 Bump version to next snapshot version

\[improve] \[broker] Avoid repeated Read-and-discard when using Key\_Shared mode

\[fix]\[broker] Fix issue of field 'topic' is not set when handle GetSchema request

8f17446355 \[improve]\[test]\[branch-3.0] Improve ManagedLedgerTest.testGetNumberOfEntriesInStorage

\[improve]\[misc] Pin Netty version in pulsar-io/alluxio

\[fix]\[build] Upgrade alluxio version to 2.9.3 to fix CVE-2023-38889

\[fix]\[test] Fix flaky test BrokerServiceAutoSubscriptionCreationTest

e3531e808c \[fix]\[test]\[branch-3.0] Fix broken ManagedLedgerTest.testGetNumberOfEntriesInStorage

### KoP

\[CI] Fix docker-compose command not found

### Cloud Storage Connector

Use the Apache images to run tests, in order to avoid permission issues.

### StreamNative Pulsar Plugins

\[test] Fix metadata integration test

### Google Pub / Sub Connector

Auth SN docker hub


# V3.0.4.3
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.4.3



# StreamNative Weekly Release Notes v3.0.4.3

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.4.3](https://github.com/streamnative/pulsar/releases/tag/v3.0.4.3)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.4.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.4.3/images/sha256-475fdfa944b9a76902f46940a8bf74a5cf0b40828b84323ddcf26c01cfb3cc71)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.0.4.3/images/sha256-fe72dc8961f76634e7949de6f9a7373d251463d12358e617f4f2bcf1ab1fa372)

## General Changes

### Apache Pulsar

\[improve]\[offload] Apply autoSkipNonRecoverableData configuration to tiered storage

\[fix]\[broker] Fix NPE causing dispatching to stop when using Key\_Shared mode and allowOutOfOrderDelivery=true

\[improve]\[build] Upgrade OWASP Dependency check version to 9.1.0

\[fix]\[broker] Fix a deadlock in SystemTopicBasedTopicPoliciesService during NamespaceEventsSystemTopicFactory init

\[improve]\[broker] Optimize gzip compression for /metrics endpoint by sharing/caching compressed result

\[fix]\[io] Kafka Source connector maybe stuck

\[fix]\[sec] Upgrade Bouncycastle to 1.78

\[fix]\[test] Flaky-test: testMessageExpiryWithTimestampNonRecoverableException and testIncorrectClientClock

\[fix]\[broker] Create new ledger after the current ledger is closed

\[fix]\[broker] Optimize /metrics, fix unbounded request queue issue and fix race conditions in metricsBufferResponse mode

\[improve]\[broker] Improve Gzip compression, allow excluding specific paths or disabling it

\[improve]\[test] Replace usage of curl in Java test and fix stream leaks

\[improve] \[broker] Servlet support response compression

\[fix] \[broker] Prevent long deduplication cursor backlog so that topic loading wouldn't timeout

\[fix]\[txn]Handle exceptions in the transaction pending ack init

\[improve]\[misc] Upgrade to Bookkeeper 4.16.5

### KoP

Add metrics documents for network in/out bytes

### AWS Lambda Connector

Upgrade commons-compress to fix CVE

### pulsarctl

fix source test typo

fix source test

### StreamNative Pulsar Plugins

Upgrade ZK, aws client and commons-configuration2

Upgrade x/net and protobuf to fix vulnerabilities

### Google BigQuery Sink Connector

Upgrade checkstyle version


# V3.0.4.4
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.4.4



# StreamNative Weekly Release Notes v3.0.4.4

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.4.4](https://github.com/streamnative/pulsar/releases/tag/v3.0.4.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.4.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.4.4/images/sha256-da823b4a9903ef71dc59778d87023e667d1ce2b7ad2805b4899f67050a6e7b31)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.0.4.4/images/sha256-1b5a76918cd2a36895881e4b73e9259c83341584780804484d09341dffc1e4c3)

## General Changes

### Apache Pulsar

\[fix]\[broker] Reader stuck after call hasMessageAvailable when enable replicateSubscriptionState

\[fix]\[test] Flaky-test: ManagedLedgerTest.testTimestampOnWorkingLedger

\[improve]\[broker] Propagate cause exception in TopicBusyException when applicable

\[improve]\[meta] Log a warning when ZK batch fails with connectionloss

\[fix]\[test] Clear fields in test cleanup to reduce memory consumption

\[fix]\[test] Fix resource leak in TransactionCoordinatorClientTest

\[fix]\[admin] Fix namespace admin api exception response

\[improve]\[sec] Align some namespace level policy authorisation check

\[fix]\[broker] Fix BufferOverflowException and EOFException bugs in /metrics gzip compression

\[fix]\[io] CompressionEnabled didn't work on elasticsearch sink

6e849fcb06 \[fix]\[test]\[branch-3.0] Fix test PersistentTopicsTest.testUpdatePartitionedTopic

d6791a8de2 Revert "\[fix]\[test]\[branch-3.0] Fix broken ManagedLedgerTest.testGetNumberOfEntriesInStorage"

\[fix]\[offload] Increase file upload limit from 2048MiB to 4096MiB for GCP/GCS offloading

\[fix]\[broker] upgrade jclouds 2.5.0 -> 2.6.0

\[fix]\[ml] Fix NPE of getValidPositionAfterSkippedEntries when recovering a terminated managed ledger

\[improve]\[broker] Support X-Forwarded-For and HA Proxy Protocol for resolving original client IP of http/https requests

\[fix]\[broker] Fix broken topic policy implementation compatibility with old pulsar version

\[fix]\[broker] Fix typos in Consumer class

\[improve]\[test] Move ShadowManagedLedgerImplTest to flaky tests

\[improve]\[broker] Repeat the handleMetadataChanges callback when configurationMetadataStore equals localMetadataStore

\[improve]\[broker] Add topic name to emitted error messages.

\[improve] Make the config `metricsBufferResponse` description more effective

\[fix]\[test] SchemaMap in AutoConsumeSchema has been reused

\[improve]\[broker] backlog quota exceed limit log replaced with `debug`

\[fix]\[broker] Fix message drop record in producer stat

\[fix]\[broker] Update topic partition failed when config `maxNumPartitionsPerPartitionedTopic<0`

\[improve]\[build] Upgrade Lombok to 1.18.32 for Java 22 support

### AoP

\[fix] Fix AoP can't work when enabling Pulsar transaction

### KoP

\[Bug fix] Fix the geo-replication message drop caused by the message metadata's producer name not being set

### AMQP1\_0 Connector

Fix integration test due to invalid package storage path

### StreamNative Pulsar Plugins

auditlog: perf: create AuditLogEvent instances only when there's a matching rule

Deferred generation id for AuditLogEvent

auditlog: replace rw lock & HashMaps with ConcurrentHashMap to reduce blocking code

auditlog: optimize uri matching by organizing condition and eliminating streams

auditlog: Cache regex compilation

### Google BigQuery Sink Connector

upgrade depend to fix cve

### Aws EventBridge Connector

Bump org.apache.avro:avro from 1.10.2 to 1.11.3


# V3.0.4.5
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.4.5



# StreamNative Weekly Release Notes v3.0.4.5

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.4.5](https://github.com/streamnative/pulsar/releases/tag/v3.0.4.5)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.4.5/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.4.5/images/sha256-d310e5a8f4cebfd12837ddacb4a9ebf19cf8d77700750a589a54ebb23b3e114e)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.0.4.5/images/sha256-97bb14789b7b667f1c6a4b579b8a970293eaff515d5b4f816e90060081fc8dbc)

## General Changes

### Apache Pulsar

\[improve]\[offload] Replace usage of shaded class in OffsetsCache

052525ffe6 \[fix]\[build]\[branch-3.0] Remove unused import added in cherry-picking

\[fix]\[offload] Fix OOM in tiered storage, caused by unbounded offsets cache

\[fix] \[broker] Fix nothing changed after removing dynamic configs

\[improve] Retry re-validating ResourceLock with backoff after errors

\[fix] \[test] Fix flaky test ReplicatorTest

\[fix]\[broker] One topic can be closed multiple times concurrently

\[fix] \[broker] Part-2: Replicator can not created successfully due to an orphan replicator in the previous topic owner

\[improve] \[broker] Create partitioned topics automatically when enable topic level replication

\[fix] \[broker] Part-1: Replicator can not created successfully due to an orphan replicator in the previous topic owner

\[fix] \[ml] Mark delete stuck due to switching cursor ledger fails

fd823f6cad \[cleanup] \[test] Cleanup unnecessary imports

\[fix] \[broker] Fix metrics pulsar\_topic\_load\_failed\_count is 0 when load non-persistent topic fails and fix the flaky test testBrokerStatsTopicLoadFailed

\[improve]\[broker] Add `topic_load_failed` metric

\[fix]\[test] Fix the flaky tests of ManagedLedgerImplUtilsTest

### AoP

bump Pulsar 3.0.4.5

### MoP

aa5a178 Fix compile issue.

### KoP

Remove spamming logs when loading groups from the offset topic

### Cloud Storage Connector

fix: Read JSON directly from the original data when formatType=json

### Function Mesh Worker Service

Support list functions/connectos across tenants and namespaces

Update doc to correct functionality on REST api


# V3.0.4.6
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.4.6



# StreamNative Weekly Release Notes v3.0.4.6

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.4.6](https://github.com/streamnative/pulsar/releases/tag/v3.0.4.6)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.4.6/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.4.6/images/sha256-bff488918ca7a4a298e5458dd8a08666ef784437f201502cb13915b4dbdf56c6)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.0.4.6/images/sha256-fd992de2631f80b8b95c265e40809cdc5c97dcf0ce3112aef0694a8dbc68ee63)

## General Changes

### Apache Pulsar

db40c8f31a \[improve]\[ci]\[branch-3.0] Upgrade actions in pulsar-ci and pulsar-ci-flaky, port owasp cache change

\[fix]\[test] Fix NPE in BookKeeperClusterTestCase tearDown

\[fix]\[broker] fix replicated subscriptions for transactional messages

\[fix] \[broker] rename to changeMaxReadPositionCount

\[fix]\[sec] Upgrade postgresql version to avoid CVE-2024-1597

\[fix]\[client] Fix ReaderBuilder doest not give illegalArgument on connection failure retry

\[fix]\[broker] Fix ProducerBusy issue due to incorrect userCreatedProducerCount on non-persistent topic

\[fix]\[broker] avoid offload system topic

\[improve]\[ws] Add memory limit configuration for Pulsar client used in Websocket proxy

\[fix]\[broker] Disable system topic message deduplication

\[fix] Fix Reader can be stuck from transaction aborted messages.

\[fix]\[test] Clear MockedPulsarServiceBaseTest fields to prevent test runtime memory leak

\[fix]\[storage] ReadonlyManagedLedger initialization does not fill in the properties

\[fix]\[sec] Upgrade elasticsearch-java version to avoid CVE-2023-4043

\[fix] \[client] Fix Consumer should return configured batch receive max messages

\[fix]\[sec] Upgrade aws-sdk.version to avoid CVE-2024-21634

\[fix]\[fn]make sure the classloader for ContextImpl is `functionClassLoader` in different runtimes

\[fix]\[broker] Avoid being stuck when closing the broker with extensible load manager

\[fix]\[io] Fix es index creation

\[improve] \[log] Print source client addr when enabled haProxyProtocolEnabled

\[fix]\[broker] usedLocallySinceLastReport should always be reset

\[fix] \[broker] Fix configurationMetadataSyncEventTopic is marked supporting dynamic setting, but not implemented

\[fix]\[broker] Fix typos lister -> listener

### AoP

Bump Pulsar 3.0.4.6

### KoP

Apply StreamNative copyright header

### pulsarctl

Add trivy scan workflow to avoid vulnerabilities

\[fix] Upgrade go version to 1.21 to fix CVE-2023-24538

### Snowflake Connector

\[branch-3.0] Fix incorrect license header in MessageIdUtils.java

\[branch-3.0] Fix `topic2table` not working and fix the doc

Fix messageId2Long cannot handle TopicMessageId

Add documentation for the schema conversion rule


# V3.0.5.1
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.5.1



# StreamNative Weekly Release Notes v3.0.5.1

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.5.1](https://github.com/streamnative/pulsar/releases/tag/v3.0.5.1)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.5.1/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.5.1/images/sha256-39b77fb1f891df35b586ef4e0ed2d11c418934e5090184d111dd519b579c1f83)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.0.5.1/images/sha256-695932ea8029a9a065afcc16016b4a11064fd8f0199f9db9726dc830aef75357)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.0.5.1/images/sha256-695932ea8029a9a065afcc16016b4a11064fd8f0199f9db9726dc830aef75357)

## General Changes

### Apache Pulsar

\[fix] \[ml] Add entry fail due to race condition about add entry failed/timeout and switch ledger

\[fix]\[ml]: subscription props could be lost in case of missing ledger during recovery

\[fix] \[broker] fix deadlock when disable topic level Geo-Replication

\[improve] \[test] Add a test to guarantee the TNX topics will not be replicated

\[improve]\[admin] Check if the topic existed before the permission operations

\[improve]\[broker] do not grant permission for each partition to reduce unnecessary zk metadata

\[improve]\[broker] checkTopicExists supports checking partitioned topic without index

\[feat]\[broker] Implementation of PIP-323: Complete Backlog Quota Telemetry

\[fix]\[admin] Fix can't delete tenant for v1

\[fix]\[broker] Make ExtensibleLoadManagerImpl.getOwnedServiceUnits async

\[fix]\[offload] Break the fillbuffer loop when met EOF

\[fix]\[schema] Error checking schema compatibility on a schema-less topic via REST API

\[improve] \[broker] \[break change] Do not create partitioned DLQ/Retry topic automatically

### KoP

Remove unnecessary debug log during the entry encode

### AWS SQS Connector

\[branch-3.0] Update pulsar version

Update license headers.

fix integrate test.

### AWS Lambda Connector

Update license headers to be proprietary.

### pulsarctl

Upgrade the dependency version to fix vulnerabilities

### StreamNative Pulsar Plugins

enable zookeeper in detector test

### Cloud Pulsar Plugins

\[improve]\[api-keys] Improve error logs when initialize failed

### Google Pub / Sub Connector

Update license header

### Google BigQuery Sink Connector

Update license headers

### Snowflake Connector

Update license header

### Aws EventBridge Connector

Update license header


# V3.0.5.2
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.5.2



# StreamNative Weekly Release Notes v3.0.5.2

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.5.2](https://github.com/streamnative/pulsar/releases/tag/v3.0.5.2)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.5.2/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.5.2/images/sha256-e3866b09294904007a26d578a125d9f59bfe1b361704fa717357fea696235d07)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.0.5.2/images/sha256-e622a253b12f8a1542a78842c6f500372fca3f129d743ac2287f903cea87b0ae)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.0.5.2/images/sha256-e622a253b12f8a1542a78842c6f500372fca3f129d743ac2287f903cea87b0ae)

## General Changes

### Apache Pulsar

\[improve]\[cli]\[branch-3.0] PIP-353: Improve transaction message visibility for peek-message

\[fix] \[broker] fix topic partitions was expanded even if disabled topic level replication

\[fix]\[admin] Clearly define REST API on Open API

\[fix]\[admin] Clearly define REST API on Open API for Topics

\[fix]\[admin] Clearly define REST API on Open API for Namesaces\@v2

\[fix]\[admin]\[part-1]Clearly define REST API on Open API

### KoP

Prevent the possible Netty TooLongFrameException with the default entry format

### Snowflake Connector

Add connection string identifier


# V3.0.5.3
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.5.3



# StreamNative Weekly Release Notes v3.0.5.3

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.5.3](https://github.com/streamnative/pulsar/releases/tag/v3.0.5.3)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.5.3/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.5.3/images/sha256-3bcf983c5fc5c96846cc13ed84683793553cc00a77de4900107ade8428b723f6)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.0.5.3/images/sha256-f402fd594623afd3b5928cd78dd7b87cc6adf6f093350c367461da6041cd9d05)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.0.5.3/images/sha256-f402fd594623afd3b5928cd78dd7b87cc6adf6f093350c367461da6041cd9d05)

## General Changes

### Apache Pulsar

\[improve] Upgrade Jetcd to 0.7.7 and VertX to 4.5.8

\[improve] \[client] improve the class GetTopicsResult

\[fix]\[broker] Fix cursor should use latest ledger config

\[cleanup]\[ml] ManagedCursor clean up.

\[improve]\[broker] Improve efficiency of checking message deletion

\[fix] Bump io.airlift:aircompressor from 0.20 to 0.27

\[fix]\[sec] Upgrade Bouncycastle libraries to address CVEs

0f08c6bb77 Bump version to 3.0.6-SNAPSHOT

\[improve]\[ml] RangeCache refactoring follow-up: use StampedLock instead of synchronized

\[improve]\[ml] RangeCache refactoring: test race conditions and prevent endless loops

\[fix]\[ml] Fix race conditions in RangeCache

\[improve]\[broker] Remove ClassLoaderSwitcher to avoid objects allocations and consistent the codestyle

\[improve]\[broker] Clear thread local BrokerEntryMetadata instance before reuse

\[fix]\[broker] EntryFilters fix NoClassDefFoundError due to closed classloader

\[improve]\[broker] avoid creating new objects when intercepting

### KoP

Support read or write producer state snapshot in metadata store

### pulsarctl

Support create token with headers

### StreamNative Pulsar Plugins

Update the metadata image


# V3.0.5.4
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.5.4



# StreamNative Weekly Release Notes v3.0.5.4

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.5.4](https://github.com/streamnative/pulsar/releases/tag/v3.0.5.4)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.5.4/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.5.4/images/sha256-a8022b1ae9d2c73bbdee57e7437a97f6504f2d7824c91fdb95f2ddaf1dcf9b38)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.0.5.4/images/sha256-c43bb34aa8774837c13d1b6a8c07153ec38a7424857afcb658af9d152a173f67)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.0.5.4/images/sha256-c43bb34aa8774837c13d1b6a8c07153ec38a7424857afcb658af9d152a173f67)

## General Changes

### Apache Pulsar

f089d4f5d5 fix: cannot find symbol from cherry-pick 73b50e

([#22867)](https://github.com/apache/pulsar/pull/22867))) \[fix] Revert "\[fix]\[cli] Fix the shell script parameter passthrough syntax

\[fix]\[broker] Fix topic status for oldestBacklogMessageAgeSeconds continuously increases even when there is no backlog.

\[fix]\[cli] Fix the pulsar-daemon parameter passthrough syntax

\[fix]\[broker]\[branch-3.0] The topic might reference a closed ledger

\[improve]\[misc] Upgrade to Netty 4.1.111.Final and switch to use grpc-netty-shaded

\[improve]\[broker] Include runtime dependencies in server distribution

\[improve]\[broker] Optimize PersistentTopic.getLastDispatchablePosition

\[fix]\[misc] Topic name from persistence name should decode local name

\[improve]\[broker] Follow up #4196 use `PulsarByteBufAllocator` handle OOM

\[improve] Upgrade IPAddress to 5.5.0

\[fix]\[cli] Fix Pulsar standalone "--wipe-data"

\[fix]\[cli] Fix Pulsar standalone shutdown - bkCluster wasn't closed

\[fix]\[cli] Fix the shell script parameter passthrough syntax

\[fix] Remove blocking calls from BookieRackAffinityMapping

\[improve]\[ci] Migrate from Gradle Enterprise to Develocity

\[fix]\[meta] Check if metadata store is closed in RocksdbMetadataStore

\[improve]\[build] Support git worktree working directory while building docker images

### AoP

\[fix] Release `EntryImpl` while reading exchange topic

### KoP

Register the callback for PartitionLog's init future only for the first time it's created

### Function Mesh Worker Service

Use stg oauth2 parameters

### Lakehouse Connector

Update the snappy download link in the Dockerfile

Install the snappy lib in the alpine image

Make seprate workflow for the release

Add dockerfile for the pulsar-io-lakehouse


# V3.0.5.5
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.5.5



# StreamNative Weekly Release Notes v3.0.5.5

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.5.5](https://github.com/streamnative/pulsar/releases/tag/v3.0.5.5)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.5.5/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.5.5/images/sha256-49328747090775b9f57311fce4130f98afaf78e66ec49211f0050a133ad23f1c)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.0.5.5/images/sha256-c8794707299f644b83b6eb6fe6f4d8e0e583d710eb206f31db67458625eac76f)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.0.5.5/images/sha256-c8794707299f644b83b6eb6fe6f4d8e0e583d710eb206f31db67458625eac76f)

## General Changes

### Apache Pulsar

\[fix]\[broker] Can't connecte to non-persist topic when enable broker client tls

\[fix]\[broker] Fix broker OOM when upload a large package.

\[improve]\[broker] Improve exception for topic does not have schema to check

\[feat]\[broker]\[branch-3.0] PIP-321 Introduce allowed-cluster at the namespace level

\[improve] \[broker] PIP-356 Support Geo-Replication starts at earliest position

\[fix]\[broker] Support lookup options for extensible load manager

\[fix]\[broker] Check the broker is available for the SLA monitor bundle when the ExtensibleLoadManager is enabled

\[fix]\[broker] Ensure that PulsarService is ready for serving incoming requests

\[fix]\[broker] Update init and shutdown time and other minor logic (ExtensibleLoadManagerImpl only)

\[fix]\[broker] Asynchronously return brokerRegistry.lookupAsync when checking if broker is active(ExtensibleLoadManagerImpl only)

\[fix]\[broker] Fix NPE after publishing a tombstone to the service unit channel

\[fix]\[broker] Immediately tombstone Deleted and Free state bundles

\[fix]\[broker] Fix Replicated Topic unload bug when ExtensibleLoadManager is enabled

\[improve]\[broker]Ensure namespace deletion doesn't fail

\[fix]\[broker] Fix updatePartitionedTopic when replication at ns level and topic policy is set

\[improve]\[fn] Make producer cache bounded and expiring in Functions/Connectors

\[fix]\[client] Fix orphan consumer when reconnection and closing are concurrency executing

\[fix]\[ci] Fix jacoco code coverage report aggregation

\[improve]\[misc] Replace rename-netty-native-libs.sh script with renaming with maven-shade-plugin

\[fix]\[ci] Replace removed macos-11 with macos-latest in GitHub Actions

([#22908)](https://github.com/apache/pulsar/pull/22908))) Revert "\[improve]\[broker] Optimize `ConcurrentOpenLongPairRangeSet` by RoaringBitmap

\[improve]\[misc]\[branch-3.2] Upgrade to Bookkeeper 4.16.6

\[improve] \[broker] make system topic distribute evenly.

\[fix]\[misc] Rename netty native libraries in pulsar-client-admin-shaded

\[cleanup]\[misc] Remove classifier from netty-transport-native-unix-common dependency

\[improve]\[broker] Optimize `ConcurrentOpenLongPairRangeSet` by RoaringBitmap

\[fix]\[broker] Check the markDeletePosition and calculate the backlog

\[fix]\[fn] Support compression type and crypto config for all producers in Functions and Connectors

\[fix] \[broker] broker log a full thread dump when a deadlock is detected in healthcheck every time

\[fix] \[client] Fix resource leak in Pulsar Client since HttpLookupService doesn't get closed

\[fix]\[test] Fix TableViewBuilderImplTest NPE and infinite loop

\[fix]\[fn] Enable optimized Netty direct byte buffer support for Pulsar Function runtimes

\[improve] Refactored BK ClientFactory to return futures

\[fix] \[broker] Messages lost on the remote cluster when using topic level replication

\[fix]\[test] Fix thread leaks in Managed Ledger tests and remove duplicate shutdown code

\[fix]\[client] fix producer/consumer perform lookup for migrated topic

\[fix] \[broker] response not-found error if topic does not exist when calling getPartitionedTopicMetadata

\[improve] \[client] PIP-344 support feature flag supportsGetPartitionedMetadataWithoutAutoCreation

\[fix] \[client] PIP-344 Do not create partitioned metadata when calling pulsarClient.getPartitionsForTopic(topicName)

### AoP

\[branch-3.0] Bump Pulsar 3.0.5.5

### MoP

cf6e090 Upgrade version to 3.0.5.5

### KoP

Fix the TopicExistsInfo object not recycled

### Cloud Storage Connector

\[fix]: fix Parquet/Avro format with separated key value avro-avro messages

### AWS SQS Connector

Support load config from secrets

make source queue size configurable

77e3b1b fix maunl workflow

75ba9d4 feat: Support cutomize trigger a release

### pulsarctl

Fix json marshal error for Secrets and UserConfigs when creating/updating functions

### StreamNative Pulsar Plugins

\[branch-3.0] Bump Pulsar 3.0.5.5

c619b912 Fix pulsar-broker-auth-multiple-key-token version

Support multiple private keys token AuthenticationProvider

### Cloud Pulsar Plugins

\[branch-3.0] Compatible changes for Auth0 Actions migration

\[branch-3.0] Bump Pulsar 3.0.5.5

### Function Mesh Worker Service

bump sn-operator to v0.5.0-rc.15

add kafka connect apis

Bump function mesh to 0.21.0

### Snowflake Connector

Improve json schema conversion


# V3.0.5.6
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.5.6



# StreamNative Weekly Release Notes v3.0.5.6

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.5.6](https://github.com/streamnative/pulsar/releases/tag/v3.0.5.6)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.5.6/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.5.6/images/sha256-03700c235dc9aacc19a9e4f71e67d5ce159a12a9e7ac07af726cbc33da881eb3)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.0.5.6/images/sha256-41cfec350c22d3c7bdd23b0fbe5dd3abefe681639333750bc7a694124c8af0d4)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.0.5.6/images/sha256-41cfec350c22d3c7bdd23b0fbe5dd3abefe681639333750bc7a694124c8af0d4)

## General Changes

### Apache Pulsar

\[improve]\[build] Upgrade dependency-check-maven-plugin to 10.0.2

\[fix]\[misc] Remove RoaringBitmap dependency from pulsar-common

\[fix]\[broker] PulsarStandalone started with error if --stream-storage-port is not 4181

\[improve]\[broker] Use RoaringBitmap in tracking individual acks to reduce memory usage

\[fix]\[broker] Fix MessageDeduplication replay timeout cause topic loading stuck

\[fix]\[ci] Fix OWASP Dependency Check download by using NVD API key

\[fix] Make operations on `individualDeletedMessages` in lock scope

### KoP

Fix producer state snapshot not taken during shutdown

### StreamNative Pulsar Plugins

\[branch-3.0] Migrate ops related classes from Pulsar

Add Pulsar OIDC plugin

\[improve] \[log] Change log level of consumer not found to WARN

detector: use new detector for 3.0.x

### Snowflake Connector

\[feat] Support metadata field mapping


# V3.0.5.7
Source: https://docs.streamnative.io/release-notes/pulsar/v3.0/v3.0.5.7



# StreamNative Weekly Release Notes v3.0.5.7

## Download

### Distributions

* [https://github.com/streamnative/pulsar/releases/tag/v3.0.5.7](https://github.com/streamnative/pulsar/releases/tag/v3.0.5.7)

### Packages

* [Maven Central](https://search.maven.org/artifact/io.streamnative/pulsar/3.0.5.7/pom)

### Images

* [sn-platform](https://hub.docker.com/layers/streamnative/sn-platform/3.0.5.7/images/sha256-ed81a599a6e340a806da3e0d4f0b3f737e9924755004fbfc3895aff01a43a49c)

* [sn-platform-slim](https://hub.docker.com/layers/streamnative/sn-platform-slim/3.0.5.7/images/sha256-f12cc5a9bc26bb49886639db83b037561c5d05ea225821d1b96b5c473e507e7b)

* [private-cloud](https://hub.docker.com/layers/streamnative/private-cloud/3.0.5.7/images/sha256-f12cc5a9bc26bb49886639db83b037561c5d05ea225821d1b96b5c473e507e7b)

## General Changes

### Apache Pulsar

18970684b1 fix code style

\[improve] \[broker] Improve CPU resources usege of TopicName Cache

\[improve] \[broker] high CPU usage caused by list topics under namespace

\[improve]\[broker]\[branch-3.0] PIP-364: Introduce a new load balance algorithm AvgShedder

fdd9747968 fix code style introduce by #22983

\[fix]\[broker] Replication stuck when partitions count between two clusters is not the same

\[fix]\[broker] Fix stuck when enable topic level replication and build remote admin fails

\[fix]\[admin] Fix half deletion when attempt to topic with a incorrect API

\[fix]\[broker] Fix geo-replication admin client url

\[fix]\[broker]Fix lookupService.getTopicsUnderNamespace can not work with a quote pattern

\[fix]\[client] Fix pattern consumer create crash if a part of partitions of a topic have been deleted

### KoP

Fix producer state snapshot not taken during shutdown

### pulsarctl

6f25051 Fix TestDeleteNonExistPartitionedTopic

### StreamNative Pulsar Plugins

558db4b0 Remove oidc test

### Function Mesh Worker Service

fix docker build

\[branch3.0] add missing piece

allow passing javaopts to kafka connect

support jwt token fallback for oauth2 handler

add integration tests for kafka connect


# Build Consumer
Source: https://docs.streamnative.io/clients/kafka-clients/.net/tutorial/kafka-dotnet-build-consumer



Next, create the .NET consumer application by pasting the following code into a file `consumer/consumer.cs`.

```csharp theme={null}
using Confluent.Kafka;
using System;
using System.Threading;

class Consumer {

    static void Main(string[] args)
    {
        var config = new ConsumerConfig
        {
            // User-specific properties that you must set
            BootstrapServers = "<BOOTSTRAP SERVERS>",
            SaslUsername     = "unused",
            SaslPassword     = "token:<API KEY>"

            // Fixed properties
            SecurityProtocol = SecurityProtocol.SaslSsl,
            SaslMechanism    = SaslMechanism.Plain,
            GroupId          = "kafka-dotnet-getting-started",
            AutoOffsetReset  = AutoOffsetReset.Earliest
        };

        const string topic = "purchases";

        CancellationTokenSource cts = new CancellationTokenSource();
        Console.CancelKeyPress += (_, e) => {
            e.Cancel = true; // prevent the process from terminating.
            cts.Cancel();
        };

        using (var consumer = new ConsumerBuilder<string, string>(config).Build())
        {
            consumer.Subscribe(topic);
            try {
                while (true) {
                    var cr = consumer.Consume(cts.Token);
                    Console.WriteLine($"Consumed event from topic {topic}: key = {cr.Message.Key,-10} value = {cr.Message.Value}");
                }
            }
            catch (OperationCanceledException) {
                // Ctrl-C was pressed.
            }
            finally{
                consumer.Close();
            }
        }
    }
}
```

Fill in the appropriate `<BOOTSTRAP SERVERS>` endpoint and `<API KEY>` in the `BootstrapServers` and `SaslPassword` properties where the client configuration `config` object is created.

You can test the syntax before proceding by running the following command:

```bash theme={null}
cd ../consumer
dotnet build consumer.csproj
cd ..
```


# Build Producer
Source: https://docs.streamnative.io/clients/kafka-clients/.net/tutorial/kafka-dotnet-build-producer



Let's create the .NET producer application by pasting the following code into a file `producer/producer.cs`.

```csharp theme={null}
using Confluent.Kafka;
using System;

class Producer {
    static void Main(string[] args)
    {
        const string topic = "purchases";

        string[] users = { "eabara", "jsmith", "sgarcia", "jbernard", "htanaka", "awalther" };
        string[] items = { "book", "alarm clock", "t-shirts", "gift card", "batteries" };

        var config = new ProducerConfig
        {
            // User-specific properties that you must set
            BootstrapServers = "<BOOTSTRAP SERVERS>",
            SaslUsername     = "unused",
            SaslPassword     = "token:<API KEY>",

            // Fixed properties
            SecurityProtocol = SecurityProtocol.SaslSsl,
            SaslMechanism    = SaslMechanism.Plain,
            Acks             = Acks.All
        };

        using (var producer = new ProducerBuilder<string, string>(config).Build())
        {
            var numProduced = 0;
            Random rnd = new Random();
            const int numMessages = 10;
            for (int i = 0; i < numMessages; ++i)
            {
                var user = users[rnd.Next(users.Length)];
                var item = items[rnd.Next(items.Length)];

                producer.Produce(topic, new Message<string, string> { Key = user, Value = item },
                    (deliveryReport) =>
                    {
                        if (deliveryReport.Error.Code != ErrorCode.NoError) {
                            Console.WriteLine($"Failed to deliver message: {deliveryReport.Error.Reason}");
                        }
                        else {
                            Console.WriteLine($"Produced event to topic {topic}: key = {user,-10} value = {item}");
                            numProduced += 1;
                        }
                    });
            }

            producer.Flush(TimeSpan.FromSeconds(10));
            Console.WriteLine($"{numProduced} messages were produced to topic {topic}");
        }
    }
}
```

Fill in the appropriate `<BOOTSTRAP SERVERS>` endpoint and `<API KEY>` in the `BootstrapServers` and `SaslPassword` properties where the client configuration `config` object is created.

You can test the syntax before proceding by running the following command:

```bash theme={null}
cd producer
dotnet build producer.csproj
```


# Cluster Setup
Source: https://docs.streamnative.io/clients/kafka-clients/.net/tutorial/kafka-dotnet-cluster-setup



We are going to need a StreamNative Cloud cluster for our client application to operate with. This guide can help you create your first StreamNative Cloud cluster.

## Create a StreamNative Cloud cluster

From within the StreamNative Cloud console, creating a new cluster is just a few clicks.

<Tabs>
  <Tab title="Cloud Console">
    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Organizations** to list your created organizations.

    2. Click the name of your first organization. In the figure below, the organization name is `Demos`.

       <img alt="screenshot of organization section" />

    3. On the **Select an Instance** card of the **Organization Dashboard** page, click the `+` button.

    4. On the **Choose the deployment type for your instance**, click **Deploy Serverless** to start the instance creation process.

    5. On the **Instance Configuration** page, enter a name for your instance, and select a Cloud Provider. The instance name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

    6. Click **Cluster Location** to start the cluster creation process.

    7. On the **Cluster Location** page, enter a name for your cluster, select the cluster location, and then click **Finish**. The cluster name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

    The cluster page appears, displaying the cluster creation process. Depending on the chosen cloud provider and other settings, it may take a few seconds to several minutes to provision the cluster. Once the cluster has been provisioned, the **Cluster Dashboard** page will be displayed.

    Now you can get started configuring apps and data on your new cluster.
  </Tab>
</Tabs>

## Get the cluster service URLs

Next, you'll need to get your StreamNative Cloud cluster's Service URLs to configure your client applications. These URLs allow your applications to connect to and interact with your cluster.

Your cluster provides different Service URLs depending on which protocol you plan to use:

For Pulsar Protocol:

* **Broker Service URL**: Used to configure Pulsar clients to connect to the brokers for producing and consuming messages
* **HTTP Service URL**: Used for administrative operations via the Pulsar admin API

For Kafka Protocol:

* **Kafka Service URL**: Used as bootstrap servers in Kafka client configurations
* **Schema Registry URL**: Used to configure Kafka serializers and deserializers that work with schemas

You'll need these URLs in the upcoming steps when setting up your producer and consumer clients.

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

## Create a service account and API key

Next, to connect your client application to your cluster to produce and consume messages, you need to create a [Service Account](/cloud/security/authentication/service-accounts/manage-service-accounts) and choose an authentication mechanism: either [API Key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview) or [OAuth](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview).

API Key authentication is quicker to implement since you only need to create an API key in StreamNative Cloud. This guide uses API Key authentication.

<Note title="Note">
  Currently, you can't edit a service account. If you need a service account to have Super Admin access, make sure to enable it when creating the service account. By default, service accounts do not have Super Admin enabled.
</Note>

<Tabs>
  <Tab title="Cloud Console">
    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Accounts & Accesses**.

    2. On the left navigation pane, click **Service Accounts**.

    3. On the **Service Account** page, click **+ New Service Account**.

    4. On the **Create Service Account** dialog, enter a name for the service account, and then click **Confirm**.

    5. On the **Service Account** page, in the row of the service account you just created, click the **...** icon, and select **Create API Key** in the dropdown menu.

    6. On the **New API Key** dialog:

       * Enter a name for the API key
       * Set the expiration date
       * Select the instance you created in Step 2
       * Write a description for the API key
       * Click **Confirm**

    <Note title="Note">
      An API key and associated secret apply to the active StreamNative instance. If you add a new instance, you must create a new API key for producers and consumers on the new Pulsar instance. For more information, see [Use API Keys to Authenticate to StreamNative Cloud](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview).
    </Note>

    7. After the API key is created, you can see the API key shown in the **New API Key** dialog. Click the **Copy and close** button to copy the API key to your clipboard. **Please note that you cannot retrieve the API key later after closing the dialog.**
  </Tab>
</Tabs>

Note the API key as we will use them when configuring the producer and consumer clients in upcoming steps.


# Consume Messages
Source: https://docs.streamnative.io/clients/kafka-clients/.net/tutorial/kafka-dotnet-consume-messages



From another terminal, run the following command to run the consumer application which will read the events from the `purchases` topic and write the information to the terminal.

```bash theme={null}
cd consumer
dotnet run
```

The consumer application will start and print any events it has not yet consumed and then wait for more events to arrive. On startup of the consumer, you should see output resembling this:

```bash theme={null}
Consumed event from topic purchases: key = jsmith     value = gift card
Consumed event from topic purchases: key = awalther   value = batteries
Consumed event from topic purchases: key = awalther   value = gift card
Consumed event from topic purchases: key = awalther   value = book
Consumed event from topic purchases: key = htanaka    value = book
Consumed event from topic purchases: key = sgarcia    value = gift card
Consumed event from topic purchases: key = sgarcia    value = t-shirts
Consumed event from topic purchases: key = sgarcia    value = gift card
Consumed event from topic purchases: key = sgarcia    value = book
Consumed event from topic purchases: key = sgarcia    value = gift card
```

Rerun the producer to see more events, or feel free to modify the code as necessary to create more or different events.

Once you are done, enter `Ctrl-C` to terminate the consumer application.


# Create Project
Source: https://docs.streamnative.io/clients/kafka-clients/.net/tutorial/kafka-dotnet-create-project



Create a new directory anywhere you’d like for this project:

```bash theme={null}
mkdir kafka-dotnet-getting-started
cd kafka-dotnet-getting-started
mkdir producer
mkdir consumer
```

Next we’ll create two different C# project files, one for the producer and one for the consumer. The project files specify the output type of project artifact which is an executable for both the producer and consumer. It also specifies the required dependencies that the .NET platform needs for the project.

Copy the following into a project file named `producer.csproj` in the `producer` subdirectory:

```xml theme={null}
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net8.0</TargetFramework>
    <StartupObject>Producer</StartupObject>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Confluent.Kafka" Version="2.3" />
  </ItemGroup>

</Project>
```

Copy the following into a project file named `consumer.csproj` in the `consumer` subdirectory:

```xml theme={null}
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net8.0</TargetFramework>
    <StartupObject>Consumer</StartupObject>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Confluent.Kafka" Version="2.3" />
  </ItemGroup>

</Project>
```


# Create Topic
Source: https://docs.streamnative.io/clients/kafka-clients/.net/tutorial/kafka-dotnet-create-topic



In a StreamNative Cluster, topics are grouped into [tenants](/cloud/manage-data-streams/tenant) and [namespaces](/cloud/manage-data-streams/namespace). In this guide, we will use the existing `public/default` namespace.

## Authorize the Service Account

<Note title="Note">
  If you create a super-user service account, you can skip this step.
</Note>

Before configuring the producer and consumer clients, you need to authorize the service account to grant the necessary permissions for the service account to interact with your StreamNative Cloud cluster.

To grant the service account permissions on the namespace level, follow these steps:

<Tabs>
  <Tab title="Cloud Console">
    1. Navigate to the **Namespace Dashboard** page by [switching to the namespace workspace](/cloud/get-started/cloud-console#switch-a-namespace).

    2. On the **Namespace Dashboard** page, click **Configuration** on the left navigation pane.

    3. On the **Namespace configuration** page, click **ADD ROLE**, and select the service account that you want to authorize.

    4. On the drop-down menu below the service account, select the proper permissions to assign to the service account. There are six permissions in total:

    * `consume`: allow the service account to consume messages.
    * `produce`: allow the service account to publish messages.
    * `functions`: allow the service account to submit and manage functions.
    * `sinks`: allow the service account to create and manage sink connectors.
    * `sources`: allow the service account to create and manage source connectors.
    * `packages`: allow the service account to upload and manage pulsar packages. If you want to submit a customized function/connector, then you will need to upload the function/connector’s JAR/NAR/Python file first, which requires the `packages` permission.

      <img alt="Authorize Service Account" />
  </Tab>
</Tabs>

In this guide, we will authorize the service account with the `produce` and `consume` permissions for the `public/default` namespace.

## Create a Topic

Navigate to the **Namespace Dashboard** page of the `public/default` namespace and follow the instructions below to create a topic, `purchases`, which you will use to produce and consume events.

<Tabs>
  <Tab title="StreamNative Console">
    To create a topic, follow these steps.

    1. Navigate to the desired **Namespace Dashboard** page by [switching to the namespace workspace](/cloud/get-started/cloud-console#switch-a-namespace).

    2. On the left navigation pane, under **Resources**, click **Topics**.

    3. Click **New Topic**.

       <img alt="screenshot of new topic dialog box" />

    4. Configure the topic, as outlined in the following table.

       | Item       | Description                                                                                                                                                                                                                                                                    |
       | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
       | Persistent | Configure the type of the topic. <br />- Persistent: messages in a persistent topic are durably persisted on the storage disk. <br />- Non-persistent: messages in a non-persistent topic are not persisted on the storage disk. <br /> By default, it is set to *Persistent*. |
       | Topic Name | Enter a name for the topic. It is a string of up to 40 characters, supporting lowercase letters (a-z), numeric characters (0-9), and the special character hyphen (-).                                                                                                         |
       | Partitions | (Optional) Configure the number of partitions for a partitioned topic. You can have from 1 up to 100 partitions per topic. It's recommended to have at least 1 partition.                                                                                                      |

    5. Click **Confirm**.
  </Tab>
</Tabs>


# Introduction
Source: https://docs.streamnative.io/clients/kafka-clients/.net/tutorial/kafka-dotnet-introduction



In this tutorial, you will build C# client applications which produce and consume messages from an Apache Kafka® cluster.

As you learn how to run your first Kafka application, we recommend using [StreamNative Cloud](https://streamnative.io/deployment) so you don't have to run your own Kafka cluster and can focus on client development.

If you do not already have an account, you can follow the steps below to sign up for StreamNative Cloud. New signups receive \$200 to spend within StreamNative Cloud. No credit card is required for the first 30 days or until your credits run out.

## Sign up for StreamNative Cloud

<Note title="Note">
  If you have an email account configured for using Single Sign-On (SSO) with StreamNative Cloud, use that email address and password when signing up.
</Note>

To sign up, navigate to the [StreamNative Cloud Console login page](https://console.streamnative.cloud/?defaultMethod=signup). Follow the prompts to create an account.

After you click **Finish**, you might have to wait briefly for your first organization to be created.


# Prerequisites
Source: https://docs.streamnative.io/clients/kafka-clients/.net/tutorial/kafka-dotnet-prerequisites



Using Windows? You'll need to download [Windows Subsystem for Linux](https://learn.microsoft.com/en-us/windows/wsl/install).

This guide assumes that you already have [.NET Core](https://dotnet.microsoft.com/en-us/download) (>= 8.0) installed.


# Produce Messages
Source: https://docs.streamnative.io/clients/kafka-clients/.net/tutorial/kafka-dotnet-produce-messages



The `dotnet` command is used to build and run the .NET project.

In order to run the producer, use the `dotnet run` command from the `producer` directory:

```bash theme={null}
cd producer
dotnet run
```

You should see output resembling this:

```bash theme={null}
Produced event to topic purchases: key = sgarcia    value = alarm clock
Produced event to topic purchases: key = jsmith     value = alarm clock
Produced event to topic purchases: key = sgarcia    value = book
Produced event to topic purchases: key = htanaka    value = batteries
Produced event to topic purchases: key = htanaka    value = book
Produced event to topic purchases: key = eabara     value = batteries
Produced event to topic purchases: key = jbernard   value = batteries
Produced event to topic purchases: key = eabara     value = t-shirts
Produced event to topic purchases: key = jbernard   value = t-shirts
Produced event to topic purchases: key = eabara     value = t-shirts
10 messages were produced to topic purchases
```


# What's Next
Source: https://docs.streamnative.io/clients/kafka-clients/.net/tutorial/kafka-dotnet-whats-next



* For the C# client API, checkout the [.NET documentation](https://docs.confluent.io/platform/current/clients/confluent-kafka-dotnet/_site/api/Confluent.Kafka.html)


# Build Consumer
Source: https://docs.streamnative.io/clients/kafka-clients/c-and-c++/tutorial/kafka-c-build-consumer



Next, create the consumer application by pasting the following C code into a file named `consumer.c`.

```c theme={null}
#include <glib.h>
#include <librdkafka/rdkafka.h>

#include "common.c"

static volatile sig_atomic_t run = 1;

/**
 * @brief Signal termination of program
 */
static void stop(int sig) {
    run = 0;
}

int main (int argc, char **argv) {
    rd_kafka_t *consumer;
    rd_kafka_conf_t *conf;
    rd_kafka_resp_err_t err;
    char errstr[512];

    // Create client configuration
    conf = rd_kafka_conf_new();

    // User-specific properties that you must set
    set_config(conf, "bootstrap.servers", "<BOOTSTRAP SERVERS>");
    set_config(conf, "sasl.username",     "unused");
    set_config(conf, "sasl.password",     "token:<API KEY>");

    // Fixed properties
    set_config(conf, "security.protocol", "SASL_SSL");
    set_config(conf, "sasl.mechanisms",   "PLAIN");
    set_config(conf, "group.id",          "kafka-c-getting-started");
    set_config(conf, "auto.offset.reset", "earliest");

    // Create the Consumer instance.
    consumer = rd_kafka_new(RD_KAFKA_CONSUMER, conf, errstr, sizeof(errstr));
    if (!consumer) {
        g_error("Failed to create new consumer: %s", errstr);
        return 1;
    }
    rd_kafka_poll_set_consumer(consumer);

    // Configuration object is now owned, and freed, by the rd_kafka_t instance.
    conf = NULL;

    // Convert the list of topics to a format suitable for librdkafka.
    const char *topic = "purchases";
    rd_kafka_topic_partition_list_t *subscription = rd_kafka_topic_partition_list_new(1);
    rd_kafka_topic_partition_list_add(subscription, topic, RD_KAFKA_PARTITION_UA);

    // Subscribe to the list of topics.
    err = rd_kafka_subscribe(consumer, subscription);
    if (err) {
        g_error("Failed to subscribe to %d topics: %s", subscription->cnt, rd_kafka_err2str(err));
        rd_kafka_topic_partition_list_destroy(subscription);
        rd_kafka_destroy(consumer);
        return 1;
    }

    rd_kafka_topic_partition_list_destroy(subscription);

    // Install a signal handler for clean shutdown.
    signal(SIGINT, stop);

    // Start polling for messages.
    while (run) {
        rd_kafka_message_t *consumer_message;

        consumer_message = rd_kafka_consumer_poll(consumer, 500);
        if (!consumer_message) {
            g_message("Waiting...");
            continue;
        }

        if (consumer_message->err) {
            if (consumer_message->err == RD_KAFKA_RESP_ERR__PARTITION_EOF) {
                /* We can ignore this error - it just means we've read
                 * everything and are waiting for more data.
                 */
            } else {
                g_message("Consumer error: %s", rd_kafka_message_errstr(consumer_message));
                return 1;
            }
        } else {
            g_message("Consumed event from topic %s: key = %.*s value = %s",
                      rd_kafka_topic_name(consumer_message->rkt),
                      (int)consumer_message->key_len,
                      (char *)consumer_message->key,
                      (char *)consumer_message->payload
                      );
        }

        // Free the message when we're done.
        rd_kafka_message_destroy(consumer_message);
    }

    // Close the consumer: commit final offsets and leave the group.
    g_message( "Closing consumer");
    rd_kafka_consumer_close(consumer);

    // Destroy the consumer.
    rd_kafka_destroy(consumer);

    return 0;
}
```

Fill in the appropriate `<BOOTSTRAP SERVERS>` endpoint and `<API KEY>` in the `bootstrap.servers` and `sasl.password` properties where the client configuration `conf` object is created.


# Build Producer
Source: https://docs.streamnative.io/clients/kafka-clients/c-and-c++/tutorial/kafka-c-build-producer



Let's create the producer application by first adding a utility method for setting configuration in a file named `common.c`:

```c theme={null}
#include <glib.h>

/* Wrapper to set config values and error out if needed.
 */
static void set_config(rd_kafka_conf_t *conf, char *key, char *value) {
    char errstr[512];
    rd_kafka_conf_res_t res;

    res = rd_kafka_conf_set(conf, key, value, errstr, sizeof(errstr));
    if (res != RD_KAFKA_CONF_OK) {
        g_error("Unable to set config: %s", errstr);
        exit(1);
    }
}
```

Next, paste the following C code into a file named `producer.c`:

```c theme={null}
#include <glib.h>
#include <librdkafka/rdkafka.h>

#include "common.c"

#define ARR_SIZE(arr) ( sizeof((arr)) / sizeof((arr[0])) )


/* Optional per-message delivery callback (triggered by poll() or flush())
 * when a message has been successfully delivered or permanently
 * failed delivery (after retries).
 */
static void dr_msg_cb (rd_kafka_t *kafka_handle,
                       const rd_kafka_message_t *rkmessage,
                       void *opaque) {
    if (rkmessage->err) {
        g_error("Message delivery failed: %s", rd_kafka_err2str(rkmessage->err));
    }
}

int main (int argc, char **argv) {
    rd_kafka_t *producer;
    rd_kafka_conf_t *conf;
    char errstr[512];

    // Create client configuration
    conf = rd_kafka_conf_new();

    // User-specific properties that you must set
    set_config(conf, "bootstrap.servers", "<BOOTSTRAP SERVERS>");
    set_config(conf, "sasl.username",     "unused");
    set_config(conf, "sasl.password",     "token:<API KEY>");

    // Fixed properties
    set_config(conf, "security.protocol", "SASL_SSL");
    set_config(conf, "sasl.mechanisms",   "PLAIN");
    set_config(conf, "acks",              "all");

    // Install a delivery-error callback.
    rd_kafka_conf_set_dr_msg_cb(conf, dr_msg_cb);

    // Create the Producer instance.
    producer = rd_kafka_new(RD_KAFKA_PRODUCER, conf, errstr, sizeof(errstr));
    if (!producer) {
        g_error("Failed to create new producer: %s", errstr);
        return 1;
    }

    // Configuration object is now owned, and freed, by the rd_kafka_t instance.
    conf = NULL;

    // Produce data by selecting random values from these lists.
    int message_count = 10;
    const char *topic = "purchases";
    const char *user_ids[6] = {"eabara", "jsmith", "sgarcia", "jbernard", "htanaka", "awalther"};
    const char *products[5] = {"book", "alarm clock", "t-shirts", "gift card", "batteries"};

    for (int i = 0; i < message_count; i++) {
        const char *key =  user_ids[random() % ARR_SIZE(user_ids)];
        const char *value =  products[random() % ARR_SIZE(products)];
        size_t key_len = strlen(key);
        size_t value_len = strlen(value);

        rd_kafka_resp_err_t err;

        err = rd_kafka_producev(producer,
                                RD_KAFKA_V_TOPIC(topic),
                                RD_KAFKA_V_MSGFLAGS(RD_KAFKA_MSG_F_COPY),
                                RD_KAFKA_V_KEY((void*)key, key_len),
                                RD_KAFKA_V_VALUE((void*)value, value_len),
                                RD_KAFKA_V_OPAQUE(NULL),
                                RD_KAFKA_V_END);

        if (err) {
            g_error("Failed to produce to topic %s: %s", topic, rd_kafka_err2str(err));
            return 1;
        } else {
            g_message("Produced event to topic %s: key = %12s value = %12s", topic, key, value);
        }

        rd_kafka_poll(producer, 0);
    }

    // Block until the messages are all sent.
    g_message("Flushing final messages..");
    rd_kafka_flush(producer, 10 * 1000);

    if (rd_kafka_outq_len(producer) > 0) {
        g_error("%d message(s) were not delivered", rd_kafka_outq_len(producer));
    }

    g_message("%d events were produced to topic %s.", message_count, topic);

    rd_kafka_destroy(producer);

    return 0;
}
```

Fill in the appropriate `<BOOTSTRAP SERVERS>` endpoint and `<API KEY>` in the `bootstrap.servers` and `sasl.password` properties where the client configuration `conf` object is created.


# Cluster Setup
Source: https://docs.streamnative.io/clients/kafka-clients/c-and-c++/tutorial/kafka-c-cluster-setup



We are going to need a StreamNative Cloud cluster for our client application to operate with. This guide can help you create your first StreamNative Cloud cluster.

## Create a StreamNative Cloud cluster

From within the StreamNative Cloud console, creating a new cluster is just a few clicks.

<Tabs>
  <Tab title="Cloud Console">
    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Organizations** to list your created organizations.

    2. Click the name of your first organization. In the figure below, the organization name is `Demos`.

       <img alt="screenshot of organization section" />

    3. On the **Select an Instance** card of the **Organization Dashboard** page, click the `+` button.

    4. On the **Choose the deployment type for your instance**, click **Deploy Serverless** to start the instance creation process.

    5. On the **Instance Configuration** page, enter a name for your instance, and select a Cloud Provider. The instance name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

    6. Click **Cluster Location** to start the cluster creation process.

    7. On the **Cluster Location** page, enter a name for your cluster, select the cluster location, and then click **Finish**. The cluster name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

    The cluster page appears, displaying the cluster creation process. Depending on the chosen cloud provider and other settings, it may take a few seconds to several minutes to provision the cluster. Once the cluster has been provisioned, the **Cluster Dashboard** page will be displayed.

    Now you can get started configuring apps and data on your new cluster.
  </Tab>
</Tabs>

## Get the cluster service URLs

Next, you'll need to get your StreamNative Cloud cluster's Service URLs to configure your client applications. These URLs allow your applications to connect to and interact with your cluster.

Your cluster provides different Service URLs depending on which protocol you plan to use:

For Pulsar Protocol:

* **Broker Service URL**: Used to configure Pulsar clients to connect to the brokers for producing and consuming messages
* **HTTP Service URL**: Used for administrative operations via the Pulsar admin API

For Kafka Protocol:

* **Kafka Service URL**: Used as bootstrap servers in Kafka client configurations
* **Schema Registry URL**: Used to configure Kafka serializers and deserializers that work with schemas

You'll need these URLs in the upcoming steps when setting up your producer and consumer clients.

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

## Create a service account and API key

Next, to connect your client application to your cluster to produce and consume messages, you need to create a [Service Account](/cloud/security/authentication/service-accounts/manage-service-accounts) and choose an authentication mechanism: either [API Key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview) or [OAuth](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview).

API Key authentication is quicker to implement since you only need to create an API key in StreamNative Cloud. This guide uses API Key authentication.

<Note title="Note">
  Currently, you can't edit a service account. If you need a service account to have Super Admin access, make sure to enable it when creating the service account. By default, service accounts do not have Super Admin enabled.
</Note>

<Tabs>
  <Tab title="Cloud Console">
    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Accounts & Accesses**.

    2. On the left navigation pane, click **Service Accounts**.

    3. On the **Service Account** page, click **+ New Service Account**.

    4. On the **Create Service Account** dialog, enter a name for the service account, and then click **Confirm**.

    5. On the **Service Account** page, in the row of the service account you just created, click the **...** icon, and select **Create API Key** in the dropdown menu.

    6. On the **New API Key** dialog:

       * Enter a name for the API key
       * Set the expiration date
       * Select the instance you created in Step 2
       * Write a description for the API key
       * Click **Confirm**

    <Note title="Note">
      An API key and associated secret apply to the active StreamNative instance. If you add a new instance, you must create a new API key for producers and consumers on the new Pulsar instance. For more information, see [Use API Keys to Authenticate to StreamNative Cloud](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview).
    </Note>

    7. After the API key is created, you can see the API key shown in the **New API Key** dialog. Click the **Copy and close** button to copy the API key to your clipboard. **Please note that you cannot retrieve the API key later after closing the dialog.**
  </Tab>
</Tabs>

Note the API key as we will use them when configuring the producer and consumer clients in upcoming steps.


# Consume Messages
Source: https://docs.streamnative.io/clients/kafka-clients/c-and-c++/tutorial/kafka-c-consume-messages



Make the consumer executable and run it:

```bash theme={null}
make consumer
./consumer
```

You should see output resembling this:

```bash theme={null}
** Message: 20:22:48.849: Consumed event from topic purchases: key = eabara value = batteries
** Message: 20:22:48.849: Consumed event from topic purchases: key = jbernard value = t-shirts
** Message: 20:22:48.849: Consumed event from topic purchases: key = eabara value = gift card
** Message: 20:22:48.849: Consumed event from topic purchases: key = jbernard value = batteries
** Message: 20:22:48.849: Consumed event from topic purchases: key = eabara value = alarm clock
** Message: 20:22:48.849: Consumed event from topic purchases: key = eabara value = t-shirts
** Message: 20:22:48.849: Consumed event from topic purchases: key = awalther value = alarm clock
** Message: 20:22:48.849: Consumed event from topic purchases: key = htanaka value = gift card
** Message: 20:22:48.849: Consumed event from topic purchases: key = awalther value = gift card
** Message: 20:22:48.849: Consumed event from topic purchases: key = sgarcia value = alarm clock
** Message: 20:22:48.849: Consumed event from topic purchases: key = jsmith value = alarm clock
** Message: 20:22:48.849: Consumed event from topic purchases: key = sgarcia value = book
** Message: 20:22:48.849: Consumed event from topic purchases: key = jsmith value = alarm clock
** Message: 20:22:48.849: Consumed event from topic purchases: key = sgarcia value = t-shirts
** Message: 20:22:48.849: Consumed event from topic purchases: key = sgarcia value = batteries
```

Rerun the producer to see more events, or feel free to modify the code as necessary to create more or different events.

Once you are done, enter `Ctrl-C` to terminate the consumer application.


# Create Project
Source: https://docs.streamnative.io/clients/kafka-clients/c-and-c++/tutorial/kafka-c-create-project



Create a new directory anywhere you’d like for this project:

```bash theme={null}
mkdir kafka-c-getting-started && cd kafka-c-getting-started
```

Create the following `Makefile` for the project:

```Makefile theme={null}
ALL: producer consumer

CFLAGS=-Wall $(shell pkg-config --cflags glib-2.0 rdkafka)
LDLIBS=$(shell pkg-config --libs glib-2.0 rdkafka)
```


# Create Topic
Source: https://docs.streamnative.io/clients/kafka-clients/c-and-c++/tutorial/kafka-c-create-topic



In a StreamNative Cluster, topics are grouped into [tenants](/cloud/manage-data-streams/tenant) and [namespaces](/cloud/manage-data-streams/namespace). In this guide, we will use the existing `public/default` namespace.

## Authorize the Service Account

<Note title="Note">
  If you create a super-user service account, you can skip this step.
</Note>

Before configuring the producer and consumer clients, you need to authorize the service account to grant the necessary permissions for the service account to interact with your StreamNative Cloud cluster.

To grant the service account permissions on the namespace level, follow these steps:

<Tabs>
  <Tab title="Cloud Console">
    1. Navigate to the **Namespace Dashboard** page by [switching to the namespace workspace](/cloud/get-started/cloud-console#switch-a-namespace).

    2. On the **Namespace Dashboard** page, click **Configuration** on the left navigation pane.

    3. On the **Namespace configuration** page, click **ADD ROLE**, and select the service account that you want to authorize.

    4. On the drop-down menu below the service account, select the proper permissions to assign to the service account. There are six permissions in total:

    * `consume`: allow the service account to consume messages.
    * `produce`: allow the service account to publish messages.
    * `functions`: allow the service account to submit and manage functions.
    * `sinks`: allow the service account to create and manage sink connectors.
    * `sources`: allow the service account to create and manage source connectors.
    * `packages`: allow the service account to upload and manage pulsar packages. If you want to submit a customized function/connector, then you will need to upload the function/connector’s JAR/NAR/Python file first, which requires the `packages` permission.

      <img alt="Authorize Service Account" />
  </Tab>
</Tabs>

In this guide, we will authorize the service account with the `produce` and `consume` permissions for the `public/default` namespace.

## Create a Topic

Navigate to the **Namespace Dashboard** page of the `public/default` namespace and follow the instructions below to create a topic, `purchases`, which you will use to produce and consume events.

<Tabs>
  <Tab title="StreamNative Console">
    To create a topic, follow these steps.

    1. Navigate to the desired **Namespace Dashboard** page by [switching to the namespace workspace](/cloud/get-started/cloud-console#switch-a-namespace).

    2. On the left navigation pane, under **Resources**, click **Topics**.

    3. Click **New Topic**.

       <img alt="screenshot of new topic dialog box" />

    4. Configure the topic, as outlined in the following table.

       | Item       | Description                                                                                                                                                                                                                                                                    |
       | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
       | Persistent | Configure the type of the topic. <br />- Persistent: messages in a persistent topic are durably persisted on the storage disk. <br />- Non-persistent: messages in a non-persistent topic are not persisted on the storage disk. <br /> By default, it is set to *Persistent*. |
       | Topic Name | Enter a name for the topic. It is a string of up to 40 characters, supporting lowercase letters (a-z), numeric characters (0-9), and the special character hyphen (-).                                                                                                         |
       | Partitions | (Optional) Configure the number of partitions for a partitioned topic. You can have from 1 up to 100 partitions per topic. It's recommended to have at least 1 partition.                                                                                                      |

    5. Click **Confirm**.
  </Tab>
</Tabs>


# Introduction
Source: https://docs.streamnative.io/clients/kafka-clients/c-and-c++/tutorial/kafka-c-introduction



In this tutorial, you will build C client applications which produce and consume messages from an Apache Kafka® cluster.

As you learn how to run your first Kafka application, we recommend using [StreamNative Cloud](https://streamnative.io/deployment) so you don't have to run your own Kafka cluster and can focus on client development.

If you do not already have an account, you can follow the steps below to sign up for StreamNative Cloud. New signups receive \$200 to spend within StreamNative Cloud. No credit card is required for the first 30 days or until your credits run out.

## Sign up for StreamNative Cloud

<Note title="Note">
  If you have an email account configured for using Single Sign-On (SSO) with StreamNative Cloud, use that email address and password when signing up.
</Note>

To sign up, navigate to the [StreamNative Cloud Console login page](https://console.streamnative.cloud/?defaultMethod=signup). Follow the prompts to create an account.

After you click **Finish**, you might have to wait briefly for your first organization to be created.


# Prerequisites
Source: https://docs.streamnative.io/clients/kafka-clients/c-and-c++/tutorial/kafka-c-prerequisites



Using Windows? You'll need to download [Windows Subsystem for Linux](https://learn.microsoft.com/en-us/windows/wsl/install).

This guide assumes that you already have a C compiler installed. The code in this guide has been tested with GCC and Clang/LLVM.

You’ll also need to install [librdkafka](https://github.com/confluentinc/librdkafka?tab=readme-ov-file#installation), [pkg-config](https://www.freedesktop.org/wiki/Software/pkg-config/) and [glibc](https://www.gnu.org/software/libc/). These libraries are widely available - search your package manager for `librdkafka`, `pkg-config` and `glib`.


# Produce Messages
Source: https://docs.streamnative.io/clients/kafka-clients/c-and-c++/tutorial/kafka-c-produce-messages



If you are using a Mac, please export the following environment variable:

```bash theme={null}
export LDFLAGS="-L/opt/homebrew/opt/openssl@3/lib"
export CPPFLAGS="-I/opt/homebrew/opt/openssl@3/include"
```

Build and make the producer executable by running the following command:

```bash theme={null}
make producer
./producer
```

You should see output resembling this:

```bash theme={null}
** Message: 20:20:47.435: Produced event to topic purchases: key =       jsmith value =  alarm clock
** Message: 20:20:47.439: Produced event to topic purchases: key =     jbernard value =         book
** Message: 20:20:47.439: Produced event to topic purchases: key =     awalther value =         book
** Message: 20:20:47.439: Produced event to topic purchases: key =      htanaka value =     t-shirts
** Message: 20:20:47.439: Produced event to topic purchases: key =     jbernard value =  alarm clock
** Message: 20:20:47.439: Produced event to topic purchases: key =      sgarcia value =     t-shirts
** Message: 20:20:47.439: Produced event to topic purchases: key =      sgarcia value =    batteries
** Message: 20:20:47.439: Produced event to topic purchases: key =     awalther value =  alarm clock
** Message: 20:20:47.439: Produced event to topic purchases: key =       eabara value =  alarm clock
** Message: 20:20:47.439: Produced event to topic purchases: key =      htanaka value =  alarm clock
** Message: 20:20:47.439: Flushing final messages..
** Message: 20:20:49.032: 10 events were produced to topic purchases.
```


# What's Next
Source: https://docs.streamnative.io/clients/kafka-clients/c-and-c++/tutorial/kafka-c-whats-next



* For the librdkafka client API, check out the [librdkafka documentation](https://github.com/confluentinc/librdkafka?tab=readme-ov-file#documentation).


# Create Project
Source: https://docs.streamnative.io/clients/kafka-clients/go/tutorial/kafka-go-create-project



Create a new directory anywhere you’d like for this project:

```bash theme={null}
mkdir kafka-go-getting-started && cd kafka-go-getting-started
```

Initialize the Go module and download the Confluent Go Kafka dependency:

```bash theme={null}
go mod init kafka-go-getting-started
go get github.com/confluentinc/confluent-kafka-go/v2/kafka
```


# Introduction
Source: https://docs.streamnative.io/clients/kafka-clients/go/tutorial/kafka-go-introduction



In this tutorial, you will build Go client applications which produce and consume messages from an Apache Kafka® cluster.

As you learn how to run your first Kafka application, we recommend using [StreamNative Cloud](https://streamnative.io/deployment) so you don't have to run your own Kafka cluster and can focus on client development.

If you do not already have an account, you can follow the steps below to sign up for StreamNative Cloud. New signups receive \$200 to spend within StreamNative Cloud. No credit card is required for the first 30 days or until your credits run out.

## Sign up for StreamNative Cloud

<Note title="Note">
  If you have an email account configured for using Single Sign-On (SSO) with StreamNative Cloud, use that email address and password when signing up.
</Note>

To sign up, navigate to the [StreamNative Cloud Console login page](https://console.streamnative.cloud/?defaultMethod=signup). Follow the prompts to create an account.

After you click **Finish**, you might have to wait briefly for your first organization to be created.


# Prerequisites
Source: https://docs.streamnative.io/clients/kafka-clients/go/tutorial/kafka-go-prerequisites



Using Windows? You'll need to download [Windows Subsystem for Linux](https://learn.microsoft.com/en-us/windows/wsl/install).

This guide assumes that you already have the [Go language tools (version 1.18 or later)](https://go.dev/doc/install) installed.


# Kafka Java Client Guide
Source: https://docs.streamnative.io/clients/kafka-clients/java/kafka-java-client-guide



You can use Kafka Java client to produce and consume messages from a StreamNative Cloud cluster. An overview of the Kafka producers and consumers for the Java client is provided below.

A producer sends messages to topics in a StreamNative Cloud cluster. Key components of a Java producer are listed below:

* **ProducerRecord**: Represents a message to be sent to a topic. It requires a topic name to send the message, and optionally, you can also specify a key and a partition number.
* **KafkaProducer**: Responsible for sending messages to their respective topics.
* **Serializer**: Converts user objects to bytes to be sent to the StreamNative Cloud cluster. Kafka provides serializers for common data types, and you can also write your own serializers.

A consumer reads messages from topics in a StreamNative Cloud cluster. Key components of a Java consumer are listed below:

* **ConsumerRecord**: Represents a message read from StreamNative Cloud.
* **KafkaConsumer**: Responsible for reading messages from the StreamNative Cloud cluster.
* **Deserializer**: Converts bytes received from the StreamNative Cloud cluster to user objects. Kafka provides deserializers for common data types, and you can also write your own deserializers.
* **ConsumerGroup**: A group of consumers that work together to consume messages from a topic.

For a step-by-step guide on building a Java client application using Kafka Protocol, see [Getting Started with Kafka Protocol and Java](/clients/kafka-clients/java/tutorial/kafka-java-introduction).

## Client installation

To use the Kafka Java client, you can add the following maven dependency to your `pom.xml` file:

```xml theme={null}
<dependency>
	<groupId>org.apache.kafka</groupId>
	<artifactId>kafka-clients</artifactId>
	<version>3.9.0</version>
</dependency>
```

## Authentication

StreamNative Cloud supports using [SASL/PLAIN](https://kafka.apache.org/documentation/#security_sasl_plain) authentication to connect Kafka clients to StreamNative Cloud. You can specify the SASL mechanism in the properties file used for initializing the Kafka producer or consumer. An example of the properties file is provided below:

```properties theme={null}
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="unused" password="token:<API KEY>";
```

Replace `<API KEY>` with your StreamNative Cloud API key. See [API Keys](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview) for more information.

## Kafka Producer

### Initialization

The Java producer is constructed with a standard `Properties` file. The following example shows how to initialize a producer:

```java theme={null}
Properties props = new Properties();
props.put("client.id", InetAddress.getLocalHost().getHostName());
props.put("bootstrap.servers", "<BOOTSTRAP_SERVERS>");
props.put("acks", "all");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

Producer<String, String> producer = new KafkaProducer<>(props);
```

Replace `<BOOTSTRAP_SERVERS>` with the bootstrap servers for your StreamNative Cloud cluster. You can find the bootstrap servers from **Cluster Details** page in **Cluster Dashboard**.

Configuration errors will result in a raised `KafkaException` from the constructor of `KafkaProducer`.

### Asynchronous send

The Java producer supports asynchronous send of messages to StreamNative Cloud via the `send()` API. The `send()` API returns a future which can be polled to get the result of the send operation.

```java theme={null}
final ProducerRecord<K, V> record = new ProducerRecord<>(topic, key, value);
Future<RecordMetadata> future = producer.send(record);
```

The producer example shows how to invoke some code after the write operation has completed you can also provide a callback. In Java, this is done by implementing the `Callback` interface.

```java theme={null}
final ProducerRecord<K, V> record = new ProducerRecord<>(topic, key, value);
producer.send(record, new Callback() {
  public void onCompletion(RecordMetadata metadata, Exception e) {
    if (e != null)
      log.debug("Send failed for record {}", record, e);
  }
});
```

In the Java implementation you should avoid doing any expensive work in this callback since it is executed in the producer’s IO thread.

### Synchronous send

```java theme={null}
Future<RecordMetadata> future = producer.send(record);
RecordMetadata metadata = future.get();
```

## Kafka Consumer

The Java consumer is constructed with a standard `Properties` file. The following example shows how to initialize a consumer:

```java theme={null}
Properties config = new Properties();
config.put("client.id", InetAddress.getLocalHost().getHostName());
config.put("group.id", "foo");
config.put("bootstrap.servers", "<BOOTSTRAP_SERVERS>");
KafkaConsumer<K, V> consumer = new KafkaConsumer<>(config);
```

Replace `<BOOTSTRAP_SERVERS>` with the bootstrap servers for your StreamNative Cloud cluster. You can find the bootstrap servers from **Cluster Details** page in **Cluster Dashboard**.

Configuration errors will result in a raised `KafkaException` from the constructor of `KafkaConsumer`.

### Basic usage

The Java client is designed around an event loop which is driven by the `poll()` API. This design is motivated by the UNIX `select` and `poll` system calls. A basic consumption loop with the Java API usually takes the following form:

```java theme={null}
while (running) {
  ConsumerRecords<K, V> records = consumer.poll(Long.MAX_VALUE);
  for (ConsumerRecord<K, V> record : records) {
    // application-specific processing
    System.out.println("Received message: " + record.value());
  }
  consumer.commitSync();
}
```

There is no background thread in the Java consumer. The API depends on calls to `poll()` to drive all of its IO including:

* Joining the consumer group and handling partition rebalances.
* Sending periodic heartbeats if part of an active generation.
* Sending periodic offset commits (if autocommit is enabled).
* Sending and receiving fetch requests for assigned partitions.

Due to this single-threaded model, no heartbeats can be sent while the application is handling the records returned from a call to `poll()`. This means that the consumer will fall out of the consumer group if either the event loop terminates or if a delay in record processing causes the session timeout to expire before the next iteration of the loop. This is actually by design. One of the problems that the Java client attempts to solve is ensuring the liveness of consumers in the group. As long as the consumer is assigned partitions, no other members in the group can consume from the same partitions, so it is important to ensure that it is actually making progress and has not become a zombie.

This feature protects your application from a large class of failures, but the downside is that it puts the burden on you to tune the session timeout so that the consumer does not exceed it in its normal record processing. The `max.poll.records` configuration option places an upper bound on the number of records returned from each call. You should use both `poll()` and `max.poll.records` with a fairly high session timeout (e.g. 30 to 60 seconds), and keeping the number of records processed on each iteration bounded so that worst-case behavior is predictable.

If you fail to tune these settings appropriately, the consequence is typically a `CommitFailedException` raised from the call to commit offsets for the processed records. If you are using the automatic commit policy, then you might not even notice when this happens since the consumer silently ignores commit failures internally (unless it’s occurring often enough to impact lag metrics). You can catch this exception and either ignore it or perform any needed rollback logic.

```java theme={null}
try {
  consumer.commitSync();
} catch (CommitFailedException e) {
  // application-specific rollback logic of processed records
}
```

### Synchronous commit

The simplest and most reliable way to manually commit offsets is using a synchronous commit with `commitSync()`. As its name suggests, this method blocks until the commit has completed successfully.

```java theme={null}
private void doCommitSync() {
  try {
    consumer.commitSync();
  } catch (WakeupException e) {
    // we're shutting down, but finish the commit first and then
    // rethrow the exception so that the main loop can exit
    doCommitSync();
    throw e;
  } catch (CommitFailedException e) {
    // the commit failed with an unrecoverable error. if there is any
    // internal state which depended on the commit, you can clean it
    // up here. otherwise it's reasonable to ignore the error and go on
    log.debug("Commit failed", e);
  }
}

public void run() {
  try {
    consumer.subscribe(topics);

    while (true) {
      ConsumerRecords<K, V> records = consumer.poll(Long.MAX_VALUE);
      records.forEach(record -> process(record));
      doCommitSync();
    }
  } catch (WakeupException e) {
    // ignore, we're closing
  } catch (Exception e) {
    log.error("Unexpected error", e);
  } finally {
    consumer.close();
    shutdownLatch.countDown();
  }
}
```

In this example, a try/catch block is added around the call to `commitSync()`. The `CommitFailedException` is thrown when the commit cannot be completed because the group has been rebalanced. This is the main thing to be careful of when using the Java client. Since all network IO (including heartbeating) and message processing is done in the foreground, it is possible for the session timeout to expire while a batch of messages is being processed. To handle this, you have two choices.

First you can adjust the `session.timeout.ms` setting to ensure that the handler has enough time to finish processing messages. You can then tune `max.partition.fetch.bytes` to limit the amount of data returned in a single batch, though you will have to consider how many partitions are in the subscribed topics.

The second option is to do message processing in a separate thread, but you will have to manage flow control to ensure that the threads can keep up. For example, just pushing messages into a blocking queue would probably not be sufficient unless the rate of processing can keep up with the rate of delivery (in which case you might not need a separate thread). It may even exacerbate the problem if the poll loop is stuck blocking on a call to offer() while the background thread is handling an even larger batch of messages. The Java API offers a pause() method to help in these situations.

For now, you should set `session.timeout.ms` large enough that commit failures from rebalances are rare. As mentioned above, the only drawback to this is a longer delay before partitions can be re-assigned in the event of a hard failure (where the consumer cannot be cleanly shut down with close()). This should be rare in practice.

You should be careful in this example since the `wakeup()` might be triggered while the commit is pending. The recursive call is safe since the wakeup will only be triggered once.

### Delivery guarantees

In the previous example, you get “at least once” delivery since the commit follows the message processing. By changing the order, however, you can get “at most once” delivery. But you must be a little careful with the commit failure, so you should change `doCommitSync` to return whether or not the commit succeeded. There’s also no longer any need to catch the `WakeupException` in the synchronous commit.

```java theme={null}
private boolean doCommitSync() {
  try {
    consumer.commitSync();
    return true;
  } catch (CommitFailedException e) {
    // the commit failed with an unrecoverable error. if there is any
    // internal state which depended on the commit, you can clean it
    // up here. otherwise it's reasonable to ignore the error and go on
    log.debug("Commit failed", e);
    return false;
  }
}

public void run() {
  try {
    consumer.subscribe(topics);

    while (true) {
      ConsumerRecords<K, V> records = consumer.poll(Long.MAX_VALUE);
      if (doCommitSync())
        records.forEach(record -> process(record));
    }
  } catch (WakeupException e) {
    // ignore, we're closing
  } catch (Exception e) {
    log.error("Unexpected error", e);
  } finally {
    consumer.close();
    shutdownLatch.countDown();
  }
}
```

Correct offset management is crucial because it affects the delivery guarantees of your application.

### Asynchronous commit

```java theme={null}
public void run() {
  try {
    consumer.subscribe(topics);

    while (true) {
      ConsumerRecords<K, V> records = consumer.poll(Long.MAX_VALUE);
      records.forEach(record -> process(record));
      consumer.commitAsync();
    }
  } catch (WakeupException e) {
    // ignore, we're closing
  } catch (Exception e) {
    log.error("Unexpected error", e);
  } finally {
    consumer.close();
    shutdownLatch.countDown();
  }
}
```

The API gives you a callback which is invoked when the commit either succeeds or fails:

```java theme={null}
consumer.commitAsync(new OffsetCommitCallback() {
  public void onComplete(Map<TopicPartition, OffsetAndMetadata> offsets, Exception exception) {
    if (exception != null)
      log.debug("Commit failed for offsets {}", offsets, exception);
  }
});
```

In the example below, synchronous commits are incorporated on rebalances and on close. For this, the `subscribe()` method has a variant which accepts a `ConsumerRebalanceListener`, which has two methods to hook into rebalance behavior.

```java theme={null}
private void doCommitSync() {
  try {
    consumer.commitSync();
  } catch (WakeupException e) {
    // we're shutting down, but finish the commit first and then
    // rethrow the exception so that the main loop can exit
    doCommitSync();
    throw e;
  } catch (CommitFailedException e) {
    // the commit failed with an unrecoverable error. if there is any
    // internal state which depended on the commit, you can clean it
    // up here. otherwise it's reasonable to ignore the error and go on
    log.debug("Commit failed", e);
  }
}

public void run() {
  try {
    consumer.subscribe(topics, new ConsumerRebalanceListener() {
      @Override
      public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
        doCommitSync();
      }

      @Override
      public void onPartitionsAssigned(Collection<TopicPartition> partitions) {}
    });

    while (true) {
      ConsumerRecords<K, V> records = consumer.poll(Long.MAX_VALUE);
      records.forEach(record -> process(record));
      consumer.commitAsync();
    }
  } catch (WakeupException e) {
    // ignore, we're closing
  } catch (Exception e) {
    log.error("Unexpected error", e);
  } finally {
    try {
      doCommitSync();
    } finally {
      consumer.close();
      shutdownLatch.countDown();
    }
  }
}
```

## API documentation

Click [here](https://kafka.apache.org/39/javadoc/index.html) to view the Java Client API documentation.

## References

* [Free Kafka Training](https://courses.streamnative.io/courses/getting-started-kafka-one-streamnative-platform/)


# Build Consumer
Source: https://docs.streamnative.io/clients/kafka-clients/java/tutorial/kafka-java-build-consumer



Next, create the Java consumer application by pasting the following code into a file `src/main/java/io/streamnative/developer/ConsumerExample.java`.

```java theme={null}
package io.streamnative.developer;

import org.apache.kafka.clients.consumer.Consumer;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;

import java.time.Duration;
import java.util.Arrays;
import java.util.Properties;

import static org.apache.kafka.clients.CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG;
import static org.apache.kafka.clients.CommonClientConfigs.SECURITY_PROTOCOL_CONFIG;
import static org.apache.kafka.clients.consumer.ConsumerConfig.*;
import static org.apache.kafka.common.config.SaslConfigs.SASL_JAAS_CONFIG;
import static org.apache.kafka.common.config.SaslConfigs.SASL_MECHANISM;

public class ConsumerExample {

    public static void main(final String[] args) {

        final Properties props = new Properties() {{
            // User-specific properties that you must set
            put(BOOTSTRAP_SERVERS_CONFIG, "<BOOTSTRAP SERVERS>");
            put(SASL_JAAS_CONFIG,         "org.apache.kafka.common.security.plain.PlainLoginModule required username='unused' password='token:<API KEY>';");

            // Fixed properties
            put(KEY_DESERIALIZER_CLASS_CONFIG,   StringDeserializer.class.getCanonicalName());
            put(VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getCanonicalName());
            put(GROUP_ID_CONFIG,                 "kafka-java-getting-started");
            put(AUTO_OFFSET_RESET_CONFIG,        "earliest");
            put(SECURITY_PROTOCOL_CONFIG,        "SASL_SSL");
            put(SASL_MECHANISM,                  "PLAIN");
        }};

        final String topic = "purchases";

        try (final Consumer<String, String> consumer = new KafkaConsumer<>(props)) {
            consumer.subscribe(Arrays.asList(topic));
            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
                for (ConsumerRecord<String, String> record : records) {
                    String key = record.key();
                    String value = record.value();
                    System.out.println(
                            String.format("Consumed event from topic %s: key = %-10s value = %s", topic, key, value));
                }
            }
        }
    }

}
```

Fill in the appropriate `<BOOTSTRAP SERVERS>` endpoint and `<API KEY>` in the `BOOTSTRAP_SERVERS_CONFIG` and `SASL_JAAS_CONFIG` properties where the client configuration `props` object is created.

Once again, you can compile the code before preceding by with:

```bash theme={null}
gradle build
```

And you should see the following output:

```
BUILD SUCCESSFUL in 1s
```


# Build Producer
Source: https://docs.streamnative.io/clients/kafka-clients/java/tutorial/kafka-java-build-producer



Create a directory for the Java files in this project:

```bash theme={null}
mkdir -p src/main/java/io/streamnative/developer
```

Let's create the Java producer application by pasting the following code into a file `src/main/java/io/streamnative/developer/ProducerExample.java`.

```java theme={null}
package io.streamnative.developer;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;

import java.util.Properties;
import java.util.Random;

import static org.apache.kafka.clients.CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG;
import static org.apache.kafka.clients.CommonClientConfigs.SECURITY_PROTOCOL_CONFIG;
import static org.apache.kafka.clients.producer.ProducerConfig.*;
import static org.apache.kafka.common.config.SaslConfigs.*;

public class ProducerExample {

    public static void main(final String[] args) {
        final Properties props = new Properties() {{
            // User-specific properties that you must set
            put(BOOTSTRAP_SERVERS_CONFIG, "<BOOTSTRAP SERVERS>");
            put(SASL_JAAS_CONFIG,         "org.apache.kafka.common.security.plain.PlainLoginModule required username='unused' password='token:<API KEY>';");

            // Fixed properties
            put(KEY_SERIALIZER_CLASS_CONFIG,   StringSerializer.class.getCanonicalName());
            put(VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getCanonicalName());
            put(ACKS_CONFIG,                   "all");
            put(SECURITY_PROTOCOL_CONFIG,      "SASL_SSL");
            put(SASL_MECHANISM,                "PLAIN");
        }};

        final String topic = "purchases";

        String[] users = {"eabara", "jsmith", "sgarcia", "jbernard", "htanaka", "awalther"};
        String[] items = {"book", "alarm clock", "t-shirts", "gift card", "batteries"};
        try (final Producer<String, String> producer = new KafkaProducer<>(props)) {
            final Random rnd = new Random();
            final int numMessages = 10;
            for (int i = 0; i < numMessages; i++) {
                String user = users[rnd.nextInt(users.length)];
                String item = items[rnd.nextInt(items.length)];

                producer.send(
                        new ProducerRecord<>(topic, user, item),
                        (event, ex) -> {
                            if (ex != null)
                                ex.printStackTrace();
                            else
                                System.out.printf("Produced event to topic %s: key = %-10s value = %s%n", topic, user, item);
                        });
            }
            System.out.printf("%s events were produced to topic %s%n", numMessages, topic);
        }

    }
}
```

Fill in the appropriate `<BOOTSTRAP SERVERS>` endpoint and `<API KEY>` in the `BOOTSTRAP_SERVERS_CONFIG` and `SASL_JAAS_CONFIG` properties where the client configuration `props` object is created.

You can test the syntax before proceding by running the following command:

```bash theme={null}
gradle build
```

And you should see the following output:

```bash theme={null}
BUILD SUCCESSFUL in 1s
```


# Cluster Setup
Source: https://docs.streamnative.io/clients/kafka-clients/java/tutorial/kafka-java-cluster-setup



We are going to need a StreamNative Cloud cluster for our client application to operate with. This guide can help you create your first StreamNative Cloud cluster.

## Create a StreamNative Cloud cluster

From within the StreamNative Cloud console, creating a new cluster is just a few clicks.

<Tabs>
  <Tab title="Cloud Console">
    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Organizations** to list your created organizations.

    2. Click the name of your first organization. In the figure below, the organization name is `Demos`.

       <img alt="screenshot of organization section" />

    3. On the **Select an Instance** card of the **Organization Dashboard** page, click the `+` button.

    4. On the **Choose the deployment type for your instance**, click **Deploy Serverless** to start the instance creation process.

    5. On the **Instance Configuration** page, enter a name for your instance, and select a Cloud Provider. The instance name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

    6. Click **Cluster Location** to start the cluster creation process.

    7. On the **Cluster Location** page, enter a name for your cluster, select the cluster location, and then click **Finish**. The cluster name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

    The cluster page appears, displaying the cluster creation process. Depending on the chosen cloud provider and other settings, it may take a few seconds to several minutes to provision the cluster. Once the cluster has been provisioned, the **Cluster Dashboard** page will be displayed.

    Now you can get started configuring apps and data on your new cluster.
  </Tab>
</Tabs>

## Get the cluster service URLs

Next, you'll need to get your StreamNative Cloud cluster's Service URLs to configure your client applications. These URLs allow your applications to connect to and interact with your cluster.

Your cluster provides different Service URLs depending on which protocol you plan to use:

For Pulsar Protocol:

* **Broker Service URL**: Used to configure Pulsar clients to connect to the brokers for producing and consuming messages
* **HTTP Service URL**: Used for administrative operations via the Pulsar admin API

For Kafka Protocol:

* **Kafka Service URL**: Used as bootstrap servers in Kafka client configurations
* **Schema Registry URL**: Used to configure Kafka serializers and deserializers that work with schemas

You'll need these URLs in the upcoming steps when setting up your producer and consumer clients.

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

## Create a service account and API key

Next, to connect your client application to your cluster to produce and consume messages, you need to create a [Service Account](/cloud/security/authentication/service-accounts/manage-service-accounts) and choose an authentication mechanism: either [API Key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview) or [OAuth](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview).

API Key authentication is quicker to implement since you only need to create an API key in StreamNative Cloud. This guide uses API Key authentication.

<Note title="Note">
  Currently, you can't edit a service account. If you need a service account to have Super Admin access, make sure to enable it when creating the service account. By default, service accounts do not have Super Admin enabled.
</Note>

<Tabs>
  <Tab title="Cloud Console">
    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Accounts & Accesses**.

    2. On the left navigation pane, click **Service Accounts**.

    3. On the **Service Account** page, click **+ New Service Account**.

    4. On the **Create Service Account** dialog, enter a name for the service account, and then click **Confirm**.

    5. On the **Service Account** page, in the row of the service account you just created, click the **...** icon, and select **Create API Key** in the dropdown menu.

    6. On the **New API Key** dialog:

       * Enter a name for the API key
       * Set the expiration date
       * Select the instance you created in Step 2
       * Write a description for the API key
       * Click **Confirm**

    <Note title="Note">
      An API key and associated secret apply to the active StreamNative instance. If you add a new instance, you must create a new API key for producers and consumers on the new Pulsar instance. For more information, see [Use API Keys to Authenticate to StreamNative Cloud](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview).
    </Note>

    7. After the API key is created, you can see the API key shown in the **New API Key** dialog. Click the **Copy and close** button to copy the API key to your clipboard. **Please note that you cannot retrieve the API key later after closing the dialog.**
  </Tab>
</Tabs>

Note the API key as we will use them when configuring the producer and consumer clients in upcoming steps.


# Consume Messages
Source: https://docs.streamnative.io/clients/kafka-clients/java/tutorial/kafka-java-consume-messages



From another terminal, run the following command to run the consumer application which will read the events from the `purchases` topic and write the information to the terminal.

```bash theme={null}
java -cp build/libs/kafka-java-getting-started-0.0.1.jar io.streamnative.developer.ConsumerExample
```

The consumer application will start and print any events it has not yet consumed and then wait for more events to arrive. On startup of the consumer, you should see output resembling this:

```bash theme={null}
Consumed event from topic purchases: key = awalther   value = book
Consumed event from topic purchases: key = htanaka    value = book
Consumed event from topic purchases: key = sgarcia    value = gift card
Consumed event from topic purchases: key = sgarcia    value = t-shirts
Consumed event from topic purchases: key = sgarcia    value = gift card
Consumed event from topic purchases: key = sgarcia    value = book
Consumed event from topic purchases: key = sgarcia    value = gift card
Consumed event from topic purchases: key = sgarcia    value = t-shirts
Consumed event from topic purchases: key = sgarcia    value = batteries
Consumed event from topic purchases: key = htanaka    value = batteries
Consumed event from topic purchases: key = htanaka    value = book
Consumed event from topic purchases: key = awalther   value = book
Consumed event from topic purchases: key = htanaka    value = t-shirts
Consumed event from topic purchases: key = awalther   value = alarm clock
Consumed event from topic purchases: key = htanaka    value = alarm clock
Consumed event from topic purchases: key = sgarcia    value = gift card
Consumed event from topic purchases: key = sgarcia    value = t-shirts
Consumed event from topic purchases: key = sgarcia    value = book
```

Rerun the producer to see more events, or feel free to modify the code as necessary to create more or different events.

Once you are done, enter `Ctrl-C` to terminate the consumer application.


# Create Project
Source: https://docs.streamnative.io/clients/kafka-clients/java/tutorial/kafka-java-create-project



Create a new directory anywhere you’d like for this project:

```bash theme={null}
mkdir kafka-java-getting-started && cd kafka-java-getting-started
```

Create the following Gradle build file for the project, named `build.gradle`:

```gradle theme={null}
buildscript {
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath "gradle.plugin.com.github.jengelman.gradle.plugins:shadow:7.0.0"
    }
}

plugins {
    id "java"
    id "idea"
    id "eclipse"
}

sourceCompatibility = "1.11"
targetCompatibility = "1.11"
version = "0.0.1"

repositories {
    mavenCentral()
}

apply plugin: "com.github.johnrengelman.shadow"

dependencies {
    implementation group: 'org.slf4j', name: 'slf4j-nop', version: '2.0.3'
    implementation group: 'org.apache.kafka', name: 'kafka-clients', version: '3.6.0'
}

jar {
    manifest {
        attributes(
                "Class-Path": configurations.compileClasspath.collect { it.getName() }.join(" "),
                "Main-Class": "io.streamnative.developer.ProducerExample"
        )
    }
}

shadowJar {
    archiveBaseName = "kafka-java-getting-started"
    archiveClassifier = ''
}
```


# Create Topic
Source: https://docs.streamnative.io/clients/kafka-clients/java/tutorial/kafka-java-create-topic



In a StreamNative Cluster, topics are grouped into [tenants](/cloud/manage-data-streams/tenant) and [namespaces](/cloud/manage-data-streams/namespace). In this guide, we will use the existing `public/default` namespace.

## Authorize the Service Account

<Note title="Note">
  If you create a super-user service account, you can skip this step.
</Note>

Before configuring the producer and consumer clients, you need to authorize the service account to grant the necessary permissions for the service account to interact with your StreamNative Cloud cluster.

To grant the service account permissions on the namespace level, follow these steps:

<Tabs>
  <Tab title="Cloud Console">
    1. Navigate to the **Namespace Dashboard** page by [switching to the namespace workspace](/cloud/get-started/cloud-console#switch-a-namespace).

    2. On the **Namespace Dashboard** page, click **Configuration** on the left navigation pane.

    3. On the **Namespace configuration** page, click **ADD ROLE**, and select the service account that you want to authorize.

    4. On the drop-down menu below the service account, select the proper permissions to assign to the service account. There are six permissions in total:

    * `consume`: allow the service account to consume messages.
    * `produce`: allow the service account to publish messages.
    * `functions`: allow the service account to submit and manage functions.
    * `sinks`: allow the service account to create and manage sink connectors.
    * `sources`: allow the service account to create and manage source connectors.
    * `packages`: allow the service account to upload and manage pulsar packages. If you want to submit a customized function/connector, then you will need to upload the function/connector’s JAR/NAR/Python file first, which requires the `packages` permission.

      <img alt="Authorize Service Account" />
  </Tab>
</Tabs>

In this guide, we will authorize the service account with the `produce` and `consume` permissions for the `public/default` namespace.

## Create a Topic

Navigate to the **Namespace Dashboard** page of the `public/default` namespace and follow the instructions below to create a topic, `purchases`, which you will use to produce and consume events.

<Tabs>
  <Tab title="StreamNative Console">
    To create a topic, follow these steps.

    1. Navigate to the desired **Namespace Dashboard** page by [switching to the namespace workspace](/cloud/get-started/cloud-console#switch-a-namespace).

    2. On the left navigation pane, under **Resources**, click **Topics**.

    3. Click **New Topic**.

       <img alt="screenshot of new topic dialog box" />

    4. Configure the topic, as outlined in the following table.

       | Item       | Description                                                                                                                                                                                                                                                                    |
       | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
       | Persistent | Configure the type of the topic. <br />- Persistent: messages in a persistent topic are durably persisted on the storage disk. <br />- Non-persistent: messages in a non-persistent topic are not persisted on the storage disk. <br /> By default, it is set to *Persistent*. |
       | Topic Name | Enter a name for the topic. It is a string of up to 40 characters, supporting lowercase letters (a-z), numeric characters (0-9), and the special character hyphen (-).                                                                                                         |
       | Partitions | (Optional) Configure the number of partitions for a partitioned topic. You can have from 1 up to 100 partitions per topic. It's recommended to have at least 1 partition.                                                                                                      |

    5. Click **Confirm**.
  </Tab>
</Tabs>


# Introduction
Source: https://docs.streamnative.io/clients/kafka-clients/java/tutorial/kafka-java-introduction



In this tutorial, you will build Java client applications which produce and consume messages from an Apache Kafka® cluster.

As you learn how to run your first Kafka application, we recommend using [StreamNative Cloud](https://streamnative.io/deployment) so you don't have to run your own Kafka cluster and can focus on client development.

If you do not already have an account, you can follow the steps below to sign up for StreamNative Cloud. New signups receive \$200 to spend within StreamNative Cloud. No credit card is required for the first 30 days or until your credits run out.

## Sign up for StreamNative Cloud

<Note title="Note">
  If you have an email account configured for using Single Sign-On (SSO) with StreamNative Cloud, use that email address and password when signing up.
</Note>

To sign up, navigate to the [StreamNative Cloud Console login page](https://console.streamnative.cloud/?defaultMethod=signup). Follow the prompts to create an account.

After you click **Finish**, you might have to wait briefly for your first organization to be created.


# Prerequisites
Source: https://docs.streamnative.io/clients/kafka-clients/java/tutorial/kafka-java-prerequisites



Using Windows? You'll need to download [Windows Subsystem for Linux](https://learn.microsoft.com/en-us/windows/wsl/install).

This guide assumes that you already have:

* [Gradle](https://gradle.org/install/) installed.
* [Java 11](https://openjdk.java.net/install/) or later installed. Verify that `java -version` outputs a version number like `11.0.20` and ensure that the `JAVA_HOME` environment variable is set.


# Produce Messages
Source: https://docs.streamnative.io/clients/kafka-clients/java/tutorial/kafka-java-produce-messages



To build a JAR that we can run from the command line, first run:

```bash theme={null}
gradle shadowJar
```

And you should see the following output:

```bash theme={null}
BUILD SUCCESSFUL in 1s
```

Run the following command to execute the producer application, which will produce some random events to the `purchases` topic.

```bash theme={null}
java -cp build/libs/kafka-java-getting-started-0.0.1.jar io.streamnative.developer.ProducerExample
```

You should see output resembling this:

```bash theme={null}
10 events were produced to topic purchases
Produced event to topic purchases: key = awalther   value = t-shirts
Produced event to topic purchases: key = jbernard   value = alarm clock
Produced event to topic purchases: key = eabara     value = book
Produced event to topic purchases: key = htanaka    value = book
Produced event to topic purchases: key = jbernard   value = batteries
Produced event to topic purchases: key = awalther   value = book
Produced event to topic purchases: key = htanaka    value = book
Produced event to topic purchases: key = sgarcia    value = gift card
Produced event to topic purchases: key = sgarcia    value = t-shirts
Produced event to topic purchases: key = sgarcia    value = book
```


# What's Next
Source: https://docs.streamnative.io/clients/kafka-clients/java/tutorial/kafka-java-whats-next



* For the Java client API, checkout the [Java documentation](https://kafka.apache.org/36/javadoc/index.html)


# Connect to StreamNative Cloud Using Kafka Clients
Source: https://docs.streamnative.io/clients/kafka-clients/kafka-clients-overview



StreamNative Cloud supports Kafka clients and tools, allowing you to develop applications using your preferred programming language, IDE, and test framework through the Kafka Protocol. The following sections provide working examples that demonstrate how to read from, process, and write data to StreamNative Cloud using Kafka clients.

## Kafka Clients

* [Java](/clients/kafka-clients/java/tutorial/kafka-java-introduction)
* [Python](/clients/kafka-clients/python/tutorial/kafka-python-introduction)
* [Go](/clients/kafka-clients/go/tutorial/kafka-go-introduction)
* [.NET](/clients/kafka-clients/.net/tutorial/kafka-dotnet-introduction)
* [Node.js](/clients/kafka-clients/node.js/tutorial/kafka-js-introduction)
* [C/C++](/clients/kafka-clients/c-and-c++/tutorial/kafka-c-introduction)
* [Spring Boot](/clients/kafka-clients/spring-boot/tutorial/kafka-spring-boot-introduction)


# Interoperability between Apache Kafka and Apache Pulsar
Source: https://docs.streamnative.io/cloud/build/kafka-clients/advanced-features/interoperability-between-kafka-and-pulsar



## Produce and consume messages between Kafka and Pulsar

StreamNative supports three data entry formats, `kafka` and `pulsar`. Each format has distinct characteristics:

* `kafka` format: This provides the best performance; however, a Pulsar consumer cannot consume it unless a payload processor is employed.
* `pulsar` format: This is the default data entry format on StreamNative cloud which supporting interoperability between Kafka and Pulsar clients, including Kafka client to Kafka client, and Pulsar client to Kafka client interactions, and vice versa. This means data between Apache Kafka and Apache Pulsar are interoperable. It is suitable for most scenarios where performance is not a critical consideration.
* `pulsar_non_batched` format: It is similar to pulsar entry format, the difference is this entry format will encode the Kafka batch messages to non-batched messages for Pulsar client to consume messages with a key-shared subscription, for Ursa, the behavior will be same as pulsar format.

StreamNative Cloud also supports specifying the entry format on a per-topic basis. The entry format can be set through topic properties by using `bin/pulsar-admin topics update-properties`. The configuration key is `kafkaEntryFormat`, and the possible values are `kafka` or `pulsar`. The default value is `pulsar` if not specified.

### Interoperability between Pulsar and Kafka clients

With the `kafka` entry format, Kafka producers can produce and consume messages directly and freely. However, Pulsar producers **SHOULD NOT** produce messages in these topics because they are unable to encode messages into a format consumable by Kafka clients.

However, since version `2.9` of the Pulsar client, we introduced a message payload processor for Pulsar consumers. This means that messages produced from Kafka producers can now be consumed and decoded by Pulsar consumers.

For the `pulsar` format, it allows messages to be freely produced and consumed between Kafka and Pulsar clients. The message format conversion is automatically handled by the broker, enabling more flexible use of either Kafka clients or Pulsar clients.

### Details for the message format conversion with `pulsar` format

The conversion is mostly intuitive except for some edge cases. Take the following simple case for example,

```java theme={null}
// The type of `kafkaProducer` is `KafkaProducer<String, String>`
kafkaProducer.send(new ProducerRecord<>(topic, "value")).get();
// The type of `pulsarConsumer` is `Consumer<byte[]>`
final var msg = pulsarConsumer.receive();
final var value = new String(msg.getValue(), StandardCharsets.UTF_8); // value will be "value"
```

The value received by a Pulsar consumer is guaranteed to be the same with the original value sent by a Kafka producer.

However, a Kafka message has some extra metadata like:

* key: the key used for routing the message to a specific partition
* headers: a list of headers, each header is a key-value pair

A Pulsar message has similar fields:

* key: the same as Kafka's key
* ordering key: the key used for `Key_Shared` subscriptions
* properties: a list of properties, each property is a key-value pair

It should be noted that the types of key and header value are both `byte[]` in Kafka, while in Pulsar, the types of key and property value are both `String`. For keys, each key will be converted to a base64-encoded string as Pulsar's key. See the following example:

```java theme={null}
kafkaProducer.send(new ProducerRecord<>(topic, 0, "key", "value")).get();
final var msg = pulsarConsumer.receive();
final String key = msg.getKey(); // "a2V5"
final byte[] keyBytes = msg.getKeyBytes(); // [107, 101, 121] (the byte array of "key")
final boolean hasBase64EncodedKey = msg.hasBase64EncodedKey(); // true
final byte[] orderingKey = msg.getOrderingKey(); // [107, 101, 121]
```

**You should use `getKeyBytes()` or `getOrderingKey()` to retrieve the original keys of Kafka messages**. The anti-intuitive behavior is that the `getKey()` method will return the base64-encoded string. This behavior is made because the byte array could vary after the `bytes -> UTF-8 string -> bytes` conversion, for example:

```java theme={null}
final var keyBytes = new byte[]{ 0x11, 0x22, 0x33, 0x44, 0x55, 0x66, 0x77, (byte) 0x88 };
// converted: [0x11, 0x22, 0x33, 0x44, 0x55, 0x66, 0x77, 0xef, 0xbf, 0xbd]
final var converted = new String(keyBytes, StandardCharsets.UTF_8).getBytes(StandardCharsets.UTF_8);
```

Things get much more complicated with the conversion on headers.

1. For a header key, there could be multiple values in a Kafka message but there is only a single property value for a given property key in a Pulsar message.
2. There is no way to get the bytes of a Pulsar property value.

For the 1st issue, the latest value will be retained.

For the 2nd issue, the conversion will be performed with the following approach:

1. Convert the bytes directly as a UTF-8 string
2. If the bytes is not a valid UTF-8 string's bytes, convert it to a base64 encoded string.

For example,

```java theme={null}
final var headers = new ArrayList<Header>();
headers.add(new RecordHeader("header-key", "header-value".getBytes(StandardCharsets.UTF_8)));
headers.add(new RecordHeader("header-key", "header-value-2".getBytes(StandardCharsets.UTF_8)));
headers.add(new RecordHeader("header-key-2", new byte[]{ (byte) 0x88 }));
kafkaProducer.send(new ProducerRecord<>(topic, 0, "key", "value", headers)).get();
final var msg = pulsarConsumer.receive();
final var value1 = msg.getProperty("header-key"); // "header-value-2"
final var value2 = msg.getProperty("header-key-2"); // "iA=="
```

There is another corner case that an extra property whose key is `__ksn_internal_header_format` will be received by the Pulsar consumer if there is a header value that is a base64-encoded string's bytes.

```java theme={null}
final var headers = new ArrayList<Header>();
headers.add(new RecordHeader("header-key", "a2v5".getBytes(StandardCharsets.UTF_8)));
kafkaProducer.send(new ProducerRecord<>(topic, 0, "key", "value", headers)).get();
final var msg = pulsarConsumer.receive();
final var value1 = msg.getProperty("header-key"); // "a2V5"
final var value2 = msg.getProperty("__ksn_internal_header_format"); // a non-null JSON value
```

### Step to Produce msg with Kafka and Consume msg with Pulsar client

**Requirements**

Pulsar Client Version ≥ 2.9
Pulsar Admin Version ≥ 2.10

**Step**

1. Create topic with kafka format via Pulsar admin CLI

<Note title="Note">
  The kafka format can be following the same step, and using `kafkaEntryFormat=kafka` property.
</Note>

```shell theme={null}
bin/pulsar-admin \
  --admin-url <Pulsar Service Http Url> \
  --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
  --auth-params token:<API Key> \
  topics create persistent://public/default/topic-with-kafka-format --metadata kafkaEntryFormat=kafka

# Get properties to check if the entry format properties setting successfully
bin/pulsar-admin \
  --admin-url <Pulsar Service Http Url> \
  --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
  --auth-params token:<API Key> \
  topics get-properties persistent://public/default/topic-with-kafka-format
```

2. Install client libraries

   ```xml theme={null}
   <!-- Use to decode the Kafka format message -->
   <dependency>
       <groupId>io.streamnative.pulsar.handlers</groupId>
       <artifactId>kafka-payload-processor</artifactId>
       <version>3.1.0.4</version>
   </dependency>

   <dependency>
       <groupId>org.apache.kafka</groupId>
       <artifactId>kafka-clients</artifactId>
       <version>3.4.0</version>
   </dependency>

   <dependency>
     <groupId>org.apache.pulsar</groupId>
     <artifactId>pulsar-client</artifactId>
     <version>3.1.0</version>
   </dependency>
   ```

3. Kafka Producer and Pulsar Consumer with Kafka format

   ```java theme={null}
   import io.streamnative.pulsar.handlers.kop.KafkaPayloadProcessor;
   import org.apache.kafka.clients.CommonClientConfigs;
   import org.apache.kafka.clients.producer.KafkaProducer;
   import org.apache.kafka.clients.producer.ProducerConfig;
   import org.apache.kafka.clients.producer.ProducerRecord;
   import org.apache.kafka.clients.producer.RecordMetadata;
   import org.apache.kafka.common.serialization.StringSerializer;
   import org.apache.pulsar.client.api.AuthenticationFactory;
   import org.apache.pulsar.client.api.Consumer;
   import org.apache.pulsar.client.api.Message;
   import org.apache.pulsar.client.api.PulsarClient;
   import org.apache.pulsar.client.api.SubscriptionInitialPosition;

   import java.io.IOException;
   import java.util.Properties;
   import java.util.concurrent.ExecutionException;
   import java.util.concurrent.Future;

   public class KafkaFormatKafkaProducePulsarConsume {

       public static void main(String[] args) throws ExecutionException, InterruptedException, IOException {
           // replace these configs with your cluster
           final String kafkaServerUrl = "<Kafka Service Url>";
           final String pulsarServerUrl = "<Pulsar Service Http Url>";
           final String jwtToken = "<API Key>";
           final String token = "token:" + jwtToken;

           final String topicName = "topic-with-kafka-format";
           final String namespace = "public/default";

           final Properties props = new Properties();
           props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaServerUrl);
           props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
           props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
           // 1. Create a producer with token authentication, which is equivalent to SASL/PLAIN mechanism in Kafka
           props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_SSL");
           props.put("sasl.mechanism", "PLAIN");
           props.put("sasl.jaas.config", String.format(
                   "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"%s\" password=\"%s\";",
                   namespace, token));

           // 2. Create a producer
           final KafkaProducer<String, String> producer = new KafkaProducer<>(props);

           // 3. Produce messages with Kafka producer
           for (int i = 0; i < 5; i++) {
               String value = "hello world";
               final Future<RecordMetadata> recordMetadataFuture = producer.send(new ProducerRecord<>(topicName, value));
               final RecordMetadata recordMetadata = recordMetadataFuture.get();
               System.out.println("Send " + value + " to " + recordMetadata);
           }
           producer.close();

           // 4. Consume messages with Pulsar consumer
           PulsarClient client = PulsarClient.builder()
                   .serviceUrl(pulsarServerUrl)
                   .authentication(AuthenticationFactory.token(jwtToken))
                   .build();

           Consumer<byte[]> consumer = client.newConsumer()
                   .topic(topicName)
                   .subscriptionName("test")
                   .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
                   // Set the Kafka payload processor to decode the kafka format message
                   .messagePayloadProcessor(new KafkaPayloadProcessor())
                   .subscribe();

           for (int i = 0; i < 5; i++) {
               Message<byte[]> msg = consumer.receive();
               consumer.acknowledge(msg);
               System.out.println("Receive message " + new String(msg.getData()));
           }

           consumer.close();
           client.close();
       }
   }
   ```

### Step to Produce msg with Pulsar and Consume msg with Kafka client

1. Create topic with pulsar format via Pulsar admin CLI

   ```shell theme={null}
   bin/pulsar-admin \
     --admin-url <Pulsar Service Http Url> \
     --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
     --auth-params token:<API Key> \
     topics create-partitioned-topic persistent://public/default/topic-with-pulsar-format -p 3 --metadata kafkaEntryFormat=pulsar

   bin/pulsar-admin \
     --admin-url <Pulsar Service Http Url> \
     --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
     --auth-params token:<API Key> \
     topics get-properties persistent://public/default/topic-with-pulsar-format
   ```

2. Install client libraries

   ```xml theme={null}
   <dependency>
       <groupId>org.apache.kafka</groupId>
       <artifactId>kafka-clients</artifactId>
       <version>3.4.0</version>
   </dependency>

   <dependency>
     <groupId>org.apache.pulsar</groupId>
     <artifactId>pulsar-client</artifactId>
     <version>3.1.0</version>
   </dependency>
   ```

3. Pulsar Producer and Kafka Consumer with Pulsar format

   ```java theme={null}
   import org.apache.kafka.clients.CommonClientConfigs;
   import org.apache.kafka.clients.consumer.ConsumerConfig;
   import org.apache.kafka.clients.consumer.ConsumerRecords;
   import org.apache.kafka.clients.consumer.KafkaConsumer;
   import org.apache.kafka.common.serialization.StringDeserializer;
   import org.apache.pulsar.client.api.AuthenticationFactory;
   import org.apache.pulsar.client.api.MessageId;
   import org.apache.pulsar.client.api.Producer;
   import org.apache.pulsar.client.api.PulsarClient;

   import java.io.IOException;
   import java.time.Duration;
   import java.util.Collections;
   import java.util.Properties;
   import java.util.concurrent.ExecutionException;

   public class PulsarFormatPulsarProduceKafkaConsume {

       public static void main(String[] args) throws ExecutionException, InterruptedException, IOException {
           // Replace these configs with your cluster
           final String kafkaServerUrl = "<Kafka Service Url>";
           final String pulsarServerUrl = "<Pulsar Service Http Url>";
           final String jwtToken = "<API Key>";
           final String token = "token:" + jwtToken;

           final String topicName = "topic-with-pulsar-format";
           final String namespace = "public/default";

           // 1. Create a Pulsar producer
           PulsarClient client = PulsarClient.builder()
                   .serviceUrl(pulsarServerUrl)
                   .authentication(AuthenticationFactory.token(jwtToken))
                   .build();

           Producer<byte[]> producer = client.newProducer()
                   .topic(topicName)
                   .create();

           // 2. Produce messages with Pulsar producer
           for (int i = 0; i < 5; i++) {
               String message = "my-message-" + i;
               MessageId msgId = producer.send(message.getBytes());
               System.out.println("Publish " + "my-message-" + i
                       + " and message ID " + msgId);
           }
           producer.close();
           client.close();

           // 3. Consume messages with Pulsar consumer
           final Properties props = new Properties();
           props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaServerUrl);
           props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
           props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
           props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
           props.put(ConsumerConfig.GROUP_ID_CONFIG, "test");

           // 4. Create a producer with token authentication, which is equivalent to SASL/PLAIN mechanism in Kafka
           props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_SSL");
           props.put("sasl.mechanism", "PLAIN");
           props.put("sasl.jaas.config", String.format(
                   "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"%s\" password=\"%s\";",
                   namespace, token));

           final KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
           consumer.subscribe(Collections.singleton(topicName));

           // 5. Consume messages with Kafka consumer
           boolean running = true;
           while (running) {
               System.out.println("running");
               final ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1));
               if (!records.isEmpty()) {
                   records.forEach(record -> System.out.println("Receive record: " + record.value() + " from "
                           + record.topic() + "-" + record.partition() + "@" + record.offset()));
                   running = false;
               }
           }
           consumer.close();
       }
   }
   ```

### Step to Produce msg with Kafka client and Consume msg with Pulsar key-shared subscription

1. Create topic with pulsar format via Pulsar admin CLI

   ```shell theme={null}
   bin/pulsar-admin \
     --admin-url <Pulsar Service Http Url> \
     --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
     --auth-params token:<API Key> \
     topics create-partitioned-topic persistent://public/default/topic-with-pulsar-non-batched-format -p 3 --metadata kafkaEntryFormat=pulsar_non_batched

   bin/pulsar-admin \
     --admin-url <Pulsar Service Http Url> \
     --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
     --auth-params token:<API Key> \
     topics get-properties persistent://public/default/topic-with-pulsar-non-batched-format
   ```

2. Install client libraries

   ```xml theme={null}
   <dependency>
       <groupId>org.apache.kafka</groupId>
       <artifactId>kafka-clients</artifactId>
       <version>3.4.0</version>
   </dependency>

   <dependency>
     <groupId>org.apache.pulsar</groupId>
     <artifactId>pulsar-client</artifactId>
     <version>3.1.0</version>
   </dependency>
   ```

3. Kafka Producer and Pulsar Consumer with `pulsar_non_batched` format

   ```java theme={null}
   import org.apache.kafka.clients.CommonClientConfigs;
   import org.apache.kafka.clients.producer.KafkaProducer;
   import org.apache.kafka.clients.producer.ProducerConfig;
   import org.apache.kafka.clients.producer.ProducerRecord;
   import org.apache.kafka.common.serialization.StringSerializer;
   import org.apache.pulsar.client.api.Consumer;
   import org.apache.pulsar.client.api.Message;
   import org.apache.pulsar.client.api.PulsarClient;
   import org.apache.pulsar.client.api.SubscriptionType;
   import org.apache.pulsar.client.api.AuthenticationFactory;

   import java.nio.charset.StandardCharsets;
   import java.util.*;
   import java.util.concurrent.*;
   import java.util.concurrent.atomic.AtomicInteger;

   public class PulsarFormatKafkaProducePulsarKeySharedSubscriptionConsume {

       private static final int NUM_CONSUMERS = 3;
       private static final int NUM_MESSAGES = 100;
       private static final int NUM_KEYS = 10;

       // Replace these configs with your actual cluster settings
       private static final String pulsarServiceUrl = "<Pulsar Service Http Url>";
       private static final String kafkaServiceUrl = "<Kafka Service Url>";
       private static final String jwtToken = "<API Key>";
       private static final String namespace = "public/default";

       public static void main(String[] args) throws Exception {
           final String topic = "topic-with-pulsar-non-batched-format";
           final String subscription = "test-key-shared-subscription-group";

           PulsarClient client = PulsarClient.builder()
               .serviceUrl(pulsarServiceUrl)
               .authentication(AuthenticationFactory.token(jwtToken))
               .build();

           // Create kafka producer
           Properties props = new Properties();
           props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaServiceUrl);
           props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
           props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);

           // Add authentication if needed
           props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_SSL");
           props.put("sasl.mechanism", "PLAIN");
           props.put("sasl.jaas.config", String.format(
               "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"%s\" password=\"%s\";",
               namespace, "token:" + jwtToken));

           final KafkaProducer<String, String> kafkaProducer = new KafkaProducer<>(props);

           // Create multiple pulsar consumers with Key_Shared subscription
           List<Consumer<byte[]>> consumers = new ArrayList<>();
           for (int i = 0; i < NUM_CONSUMERS; i++) {
               Consumer<byte[]> consumer = client.newConsumer()
                   .topic(topic)
                   .subscriptionName(subscription)
                   .subscriptionType(SubscriptionType.Key_Shared)
                   .subscribe();
               consumers.add(consumer);
           }

           // Maps to track message distribution
           Map<String, Integer> keyToConsumerMap = new ConcurrentHashMap<>();
           Map<Integer, Set<String>> consumerToKeysMap = new HashMap<>();
           for (int i = 0; i < NUM_CONSUMERS; i++) {
               consumerToKeysMap.put(i, new HashSet<>());
           }

           // Atomic counter to track processed messages
           AtomicInteger processedMessages = new AtomicInteger(0);

           // Create consumer futures
           List<CompletableFuture<Void>> futures = new ArrayList<>();
           for (int i = 0; i < NUM_CONSUMERS; i++) {
               final int consumerId = i;
               futures.add(CompletableFuture.runAsync(() -> {
                   try {
                       Consumer<byte[]> consumer = consumers.get(consumerId);
                       while (processedMessages.get() < NUM_MESSAGES) {
                           Message<byte[]> msg = consumer.receive(1, TimeUnit.SECONDS);
                           if (msg != null) {
                               String key = new String(msg.getKeyBytes(), StandardCharsets.UTF_8);

                               // Record which consumer got which key
                               keyToConsumerMap.put(key, consumerId);
                               consumerToKeysMap.get(consumerId).add(key);

                               consumer.acknowledge(msg);
                               processedMessages.incrementAndGet();
                               System.out.println("Consumer " + consumerId + " received message with key: " + key);
                           }
                       }
                   } catch (Exception e) {
                       throw new CompletionException(e);
                   }
               }));
           }

           // Create combined future for all consumers
           CompletableFuture<Void> allFutures = CompletableFuture.allOf(futures.toArray(new CompletableFuture[0]));

           // Send messages with different keys
           System.out.println("Sending first batch of messages");
           for (int i = 0; i < NUM_MESSAGES / 2; i++) {
               String key = "key-" + (i % NUM_KEYS);
               String value = "value-" + i;
               kafkaProducer.send(new ProducerRecord<>(topic, key, value));
           }
           kafkaProducer.flush();

           System.out.println("Sending second single batch of messages");
           for (int i = NUM_MESSAGES / 2; i < NUM_MESSAGES; i++) {
               String key = "key-" + (i % NUM_KEYS);
               String value = "value-" + i;
               kafkaProducer.send(new ProducerRecord<>(topic, key, value));
               kafkaProducer.flush();
           }

           // Wait for all messages to be consumed (with timeout)
           try {
               System.out.println("Waiting for all messages to be consumed...");
               allFutures.get(30, TimeUnit.SECONDS);
           } catch (TimeoutException e) {
               // Cancel all futures if timeout occurs
               futures.forEach(f -> f.cancel(true));
               System.out.println("Not all messages were consumed within timeout");
               throw new RuntimeException("Not all messages were consumed within timeout", e);
           }

           // Close resources
           System.out.println("Closing consumers and producer");
           for (Consumer<byte[]> consumer : consumers) {
               consumer.close();
           }
           client.close();
           kafkaProducer.close();

           // Verification: Check that each key went to exactly one consumer
           Map<String, List<Integer>> keyViolations = new HashMap<>();
           for (String key : keyToConsumerMap.keySet()) {
               for (int i = 0; i < NUM_MESSAGES; i++) {
                   String messageKey = "key-" + (i % NUM_KEYS);
                   if (messageKey.equals(key)
                       && !Objects.equals(keyToConsumerMap.get(key), keyToConsumerMap.get(messageKey))) {
                       keyViolations.computeIfAbsent(key, k -> new ArrayList<>()).add(i);
                   }
               }
           }

           if (keyViolations.isEmpty()) {
               System.out.println("Key consistency verification passed");
           } else {
               System.out.println("Key consistency violated: " + keyViolations);
               throw new RuntimeException("Key consistency violated: " + keyViolations);
           }

           // Verify keys are distributed among consumers
           boolean allConsumersReceived = consumerToKeysMap.values().stream()
               .noneMatch(Set::isEmpty);

           if (allConsumersReceived) {
               System.out.println("All consumers received messages");
           } else {
               System.out.println("Some consumers didn't receive any messages");
               throw new RuntimeException("Some consumers didn't receive any messages");
           }

           // Print key distribution
           System.out.println("Key distribution among consumers:");
           for (int i = 0; i < NUM_CONSUMERS; i++) {
               System.out.println("Consumer " + i + " received keys: " + consumerToKeysMap.get(i));
           }
       }
   }
   ```

## Interoperability between Kafka and Pulsar Transactions

Currently, it is not possible to interoperate Kafka and Pulsar transactions together. You must choose one or the other because they use different mechanisms to store transactional states.

## Interoperability between Kafka and Pulsar Schema

StreamNative Cloud supports both the Kafka and Pulsar schema registries as central repositories to store registered schema information, which enables producers and consumers to coordinate the schema of a topic's messages through brokers. However, Kafka schemas and Pulsar schemas cannot be used simultaneously due to their differing API definitions and schema storage locations.

We also plan to achieve a unified schema registry, which will support both Kafka and Pulsar schemas. This will allow for the exchangeable production and consumption of messages with schema using both Pulsar and Kafka clients:

* Pulsar consumers will be able to consume messages with schema produced by Kafka producers.
* Kafka consumers will be able to consume messages with schema produced by Pulsar producers.

## Use Pulsar admin to get KSN producer and consumer stats

After the pulsar version `3.3.5.1` or `4.0.1.1`, we can use the `pulsar-admin` CLI to get the KSN's topic producer and consumer stats.

<Note title="Note">
  1. Ursa Engine currently only supports namespace level stats.
  2. If you're creating a Pulsar subscription on this topic, do not use `__ksn_internal_subscription` as the subscription name.
</Note>

The KSN broker will register an internal producer and consumer for each topic. The producer's name is `{clusterName}-{generatorInstanceId}-{counter}`, and the consumer's name is `__KSN__internal_consumer_{remoteAddress}`. The subscription name is `__ksn_internal_subscription`.

The internal producer and consumer will not send or consume messages, nor will they acknowledge messages or affect message retention.

You can use the following command to get the stats:

```shell theme={null}
# Get the topic stats, some of the fields are removed for better readability
$ pulsar-admin topics partitioned-stats test-topic
{
  "msgRateIn" : 5000.014579625847,
  "msgThroughputIn" : 618419.0865918549,
  "msgRateOut" : 5000.265921846748,
  "msgThroughputOut" : 576849.8534719701,
  "bytesInCounter" : 228942495,
  "msgInCounter" : 1957421,
  "bytesOutCounter" : 220383984,
  "msgOutCounter" : 1957222,
  "averageMsgSize" : 123.68345666666666,
  "storageSize" : 229986335,
  "backlogSize" : 149996147,
  "publishers" : [ {
    "msgRateIn" : 5000.014579625847,
    "msgThroughputIn" : 618419.0865918549,
    "averageMsgSize" : 123.68345666666667,
    "producerId" : 0,
  } ],
  "subscriptions" : {
    "__ksn_internal_subscription" : {
      "msgRateOut" : 5000.265921846748,
      "msgThroughputOut" : 576849.8534719701,
      "bytesOutCounter" : 220383984,
      "msgOutCounter" : 1957222,
      "consumers" : [ {
        "msgRateOut" : 5000.265921846748,
        "msgThroughputOut" : 576849.8534719701,
        "bytesOutCounter" : 220383984,
        "msgOutCounter" : 1957222,
      } ],
    }
  },
  "metadata" : {
    "partitions" : 1,
    "deleted" : false,
    "properties" : {
      "kafkaTopicUUID" : "6374b5c6-d932-43ad-8f70-7960d3236bba"
    }
  },
  "partitions" : { }
}
```

You can also see the producer and consumer stats in the StreamNative Cloud console.

<img alt="Producer and Consumer Stats" />


# Kafka Compacted Topic
Source: https://docs.streamnative.io/cloud/build/kafka-clients/advanced-features/kafka-compacted-topic



<Warning title="Warning">
  If you are using a [Ursa-Engine](/cloud/overview/data-streaming-engine) powered cluster, please note that transactions and topic compaction are not supported in Ursa Engine.
</Warning>

StreamNative and Apache Kafka both support topic compaction, a key-based data retention mechanism. Unlike the segment-based data retention mechanism, which removes old segments based on time or size, the key-based mechanism retains the latest value for a given key. However, there are subtle differences in how data compaction is handled between StreamNative and Apache Kafka because the topic compaction mechanism on StreamNative Cloud is based on Apache Pulsar.

* Apache Kafka supports a `delete+compact` retention policy, which can remove the record of an old key even if it is the latest value for that key and applies the segment-based data cleanup policy (1 day by default). However, StreamNative cannot remove compacted keys by retention time or size unless a tombstone (null value) message is written into the topic.
* The topic config options `max.compaction.lag.ms` and `min.compaction.lag.ms` are not supported.
* Topic compaction for Kafka currently cannot work with transactions, but support for this feature is planned for the future.
* In Kafka, a tombstone (a key with a null value) is retained for a period set by `delete.retention.ms`. In contrast, StreamNative/Pulsar removes the tombstone immediately.
* StreamNative supports manually triggering compaction, whereas Kafka does not.

## Use Compacted Topic

### Create Compacted Topic

To create a compact topic, you can follow the [CLI Tools tutorial](/cloud/build/kafka-clients/quick-starts/cloud-connect-kafka-cli) and use the following command line to create the compact topic:

```shell theme={null}
./bin/kafka-topics.sh --create --bootstrap-server <YOUR-BOOTSTRAP-SERVER-ADDRESS> --replication-factor 1 --partitions 1 --topic my_compact_topic --config "cleanup.policy=compact"
```

### Configure Compaction Policy

You can change the `compaction-threshold` policy to control how often compression is triggered (default 100MB) it specifies how large the topic backlog can grow before compaction is triggered, or you can manually trigger compaction using the Pulsar administrative API. For more information, see [Topic Compaction Cookbook](https://pulsar.apache.org/docs/3.2.x/cookbooks-compaction/#configure-compaction-to-run-automatically).


# Multi-Tenancy
Source: https://docs.streamnative.io/cloud/build/kafka-clients/advanced-features/kafka-multi-tenancy



## Topic Naming Rule

Pulsar supports [multi-tenancy](https://pulsar.apache.org/docs/next/concepts-multi-tenancy/). The client side should specifies a long topic name URL like `"persistent://tenant/ns/topic"`. StreamNative leverages Pulsar's multi-tenancy feature and keeps the compatibility with the short topic names in Kafka.

For Kafka client users, they should follow the topic naming rule in this section. Otherwise, unexpected behaviors might happen.

You should specific topic names like:

* `topic`: it refers the topic `persistent://public/default/topic` in the `public/default` namespace.
* `tenant.ns.topic`: it refers the topic `persistent://tenant/ns/topic` in the `tenant/ns` namespace.
* `tenant.ns.xxx.yyy`: it refers the topic `persistent://tenant/ns/xxx.yyy` in the `tenant/ns` namespace.

In short, use a short topic name if you don't use the multi-tenancy feature. Otherwise, add the namespace prefix for topics in non-default namespaces. Besides, Kafka clients can only access namespaces whose name doe not contain any dot character.

<Note title="Note">
  In early versions of StreamNative Cloud clusters, the topic naming style is similar to Pulsar, i.e. you need to specify `tenant/ns/topic` rather than `tenant.ns.topic` to access topic `topic` in namespace `tenant/ns`.
</Note>

## Listing topics

By default, when listing topics via Kafka clients or Kafka CLI, only topics in the default namespace will be listed. This behavior is intended to avoid accessing other namespaces that you might not have permission to access.

If you want to list topics in some other namespaces (e.g. `tenant1/ns1` and `tenant2/ns2`), you can use Pulsar admin CLI to [update the configuration dynamically](https://pulsar.apache.org/docs/next/admin-api-brokers/#update-broker-conf-dynamically):

```bash theme={null}
./bin/pulsar-admin brokers update-dynamic-config --config kopAllowedNamespaces --value "tenant1/ns1,tenant2/ns2"
```

For the permission to list topics, you need to grant produce or consume permissions on all namespaces in the `kopAllowedNamespaces` config to the role you have. See [Kafka ACLs on StreamNative Cloud](/cloud/security/access/access-control-lists/kafka-acls) for more details.

Take the command above for example and assume the role is `user`, you need to run the following commands to grant the permissions:

```bash theme={null}
./bin/pulsar-admin namespaces grant-permission --actions produce --role user tenant1/ns1
./bin/pulsar-admin namespaces grant-permission --actions produce --role user tenant2/ns2
```

You can also grant the permissions simply in the cloud console UI.

<Note title="Note">
  It's highly recommended to add the namespace to the `kopAllowedNamespaces` config when you created a topic used by Kafka clients in a new namespace. For now, you need to execute the `pulsar-admin brokers` command manually to update this config. In future, this step will be done automatically in the cloud console.
</Note>


# Kafka Transactions
Source: https://docs.streamnative.io/cloud/build/kafka-clients/advanced-features/kafka-transaction



<Warning title="Warning">
  If you are using a [Ursa-Engine](/cloud/overview/data-streaming-engine) powered cluster, please note that transactions and topic compaction are not supported in Ursa Engine.
</Warning>

StreamNative supports Apache Kafka® compatible transaction semantics and APIs. For example, you can fetch messages starting from the last consumed offset and process them transactionally one by one, updating the last consumed offset and generating events as you go.

If a producer sends multiple messages to the same or different partitions and a network connection or broker failure occurs, it can be guaranteed that all messages are either completely written to the partition or not at all. This is crucial for applications that require strict guarantees, such as financial services transactions.

Transactions ensure exactly-once semantics (EOS) and atomicity.

EOS helps developers avoid the anomalies of at-most-once processing (possible event loss) and at-least-once processing (possible event duplication). Combined with idempotent producers, StreamNative supports EOS, ensuring that events are neither lost nor duplicated. Atomicity commits a set of messages across partitions as a unit: either all messages are committed, or none are. Data encapsulated and transmitted in a single operation can only succeed or fail globally, ensuring consistent transaction outcomes.

## Connect to your cluster and send transaction messages

This section describes how to connect to your cluster and send transaction messages.

### Before you begin

<Note title="Note">
  * Before using an API key, verify that the service account is authorized to access the resources, such as tenants, namespaces, and topics.
  * The password for different utilities as `kcat` will be equal to `token:<API KEY>`.
</Note>

You can follow the instructions to [create an API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#using-api-keys-to-connect-to-your-cluster) for the service account you choose to use.

### Steps

1. Add Maven dependencies.

   ```xml theme={null}
   <dependency>
       <groupId>org.apache.kafka</groupId>
       <artifactId>kafka-clients</artifactId>
       <version>3.4.0</version>
   </dependency>
   ```

2. Open a terminal and run a Kafka consumer to receive a message from the `test-transaction-topic` topic. In this case, the isolation level to `read_committed`, which means only 5 committed messages will be consumed, if you want to consume both committed and uncommitted messages, set it to `read_uncommitted`.

   ```java theme={null}
   package org.example;

   import java.time.Duration;
   import java.util.Collections;
   import java.util.Properties;
   import org.apache.kafka.clients.CommonClientConfigs;
   import org.apache.kafka.clients.consumer.ConsumerConfig;
   import org.apache.kafka.clients.consumer.ConsumerRecords;
   import org.apache.kafka.clients.consumer.KafkaConsumer;
   import org.apache.kafka.clients.producer.ProducerConfig;
   import org.apache.kafka.common.config.SaslConfigs;
   import org.apache.kafka.common.serialization.StringDeserializer;
   import org.apache.kafka.common.serialization.StringSerializer;

   public class SNCloudReadCommittedConsumer {
       public static void main(String[] args) {

           // Replace these configs for your cluster
           String serverUrl = "SERVER-URL";
           String jwtToken = "YOUR-API-KEY";
           String token = "token:" + jwtToken;

           final String topicName = "test-transaction-topic";
           String namespace = "public/default";

           final Properties props = new Properties();
           props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, serverUrl);
           props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
           props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
           props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
           props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
           props.put(ConsumerConfig.GROUP_ID_CONFIG, "hello-world");
           props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
           props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_SSL");
           props.put(SaslConfigs.SASL_MECHANISM, "PLAIN");
           props.put(SaslConfigs.SASL_JAAS_CONFIG, String.format(
                   "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"%s\" password=\"%s\";",
                   namespace, token));

           // Set the isolation level to read_committed, which means only committed messages will be consumed
           // If you want to consume both committed and uncommitted messages, set it to read_uncommitted
           props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");

           // Create a consumer
           final KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
           consumer.subscribe(Collections.singleton(topicName));
           System.out.println("running");
           while (true) {
               final ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1));
               if (!records.isEmpty()) {
                   records.forEach(record -> System.out.println("Receive record: " + record.value() + " from "
                        + record.topic() + "-" + record.partition() + "@" + record.offset()));
               }
           }
       }
   }
   ```

   * `serverUrl`: the Kafka service URL of your StreamNative cluster.
   * `jwtToken`: an API key of your service account.

3. Open another terminal and run a Kafka producer to send 5 messages to the `test-transaction-topic` topic and commit it, then send 5 messages and abort it.

   ```java theme={null}
   package org.example;

   import java.util.Properties;
   import java.util.concurrent.ExecutionException;
   import java.util.concurrent.Future;
   import org.apache.kafka.clients.CommonClientConfigs;
   import org.apache.kafka.clients.producer.KafkaProducer;
   import org.apache.kafka.clients.producer.ProducerConfig;
   import org.apache.kafka.clients.producer.ProducerRecord;
   import org.apache.kafka.clients.producer.RecordMetadata;
   import org.apache.kafka.common.config.SaslConfigs;
   import org.apache.kafka.common.serialization.StringSerializer;

   public class SNCloudTransactionProducer {
       public static void main(String[] args) throws ExecutionException, InterruptedException {
           // 1. Replace these configs for your cluster
           String serverUrl = "SERVER-URL";
           String jwtToken = "YOUR-API-KEY";
           String token = "token:" + jwtToken;
           final String topicName = "test-transaction-topic";
           String namespace = "public/default";

           // 2. Create a producer with token authentication, which is equivalent to SASL/PLAIN mechanism in Kafka
           final Properties props = new Properties();
           props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, serverUrl);
           props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
           props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
           props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_SSL");
           props.put(SaslConfigs.SASL_MECHANISM, "PLAIN");
           props.put(SaslConfigs.SASL_JAAS_CONFIG, String.format(
                   "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"%s\" password=\"%s\";",
                   namespace, token));

           // 3. Set the transactional.id property
           props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "my-transactional-id");

           // 4. Create a producer and start a transaction
           final KafkaProducer<String, String> producer = new KafkaProducer<>(props);
           producer.initTransactions();
           producer.beginTransaction();

           // 5. Produce 5 messages and commit it
           for (int i = 0; i < 5; i++) {
               String value = "Commit message " + i;
               final Future<RecordMetadata> recordMetadataFuture = producer.send(new ProducerRecord<>(topicName, value));
               final RecordMetadata recordMetadata = recordMetadataFuture.get();
               System.out.println("Send " + value + " to " + recordMetadata);
           }

           producer.commitTransaction();

           // 6. Produce 5 messages and abort it
           producer.beginTransaction();
           for (int i = 0; i < 5; i++) {
               String value = "Abort message " + i;
               final Future<RecordMetadata> recordMetadataFuture = producer.send(new ProducerRecord<>(topicName, value));
               final RecordMetadata recordMetadata = recordMetadataFuture.get();
               System.out.println("Send " + value + " to " + recordMetadata);
           }

           producer.close();
       }
   }
   ```

   * `serverUrl`: the Kafka service URL of your StreamNative cluster.
   * `jwtToken`: an API key of your service account.


# Kafka Compatibility Overview
Source: https://docs.streamnative.io/cloud/build/kafka-clients/compatibility/kafka-compatibility



<Warning title="Warning">
  If you are using a [Ursa-Engine](/cloud/overview/data-streaming-engine) powered cluster, please note that transactions and topic compaction are not supported in Ursa Engine.
</Warning>

StreamNative Cloud is compatible with Apache Kafka versions 0.9 and later, with specific exceptions noted on this page.

## TLS SNI extension requirements

StreamNative Cloud uses TLS Service Name Indication (SNI) to route the Kafka or Pulsar requests to the correct brokers. In order to connect to a StreamNative Cloud cluster, the client libraries you are using must include a SNI extension in the TLS handshake. Most of the Kafka client libraries do include the SNI extension in the TLS handshake.

For Kafka protocol connections (which use port `9093` on StreamNative Cloud), the SNI extension must be set to the DNS hostname of the cluster endpoint or one of the brokers.

1. For initial connections to the cluster, they include the configured bootstrap endpoint as the SNI extension.

2. For the subsequent connections to individual brokers which are made after bootstrapping, they include that specific broker's endpoint as the SNI extension.

These broker endpoints are automatically discovered as part of Kafka client bootstrapping process and do not need to be configured.

For REST/HTTPS connections (which use port `443`), the SNI extension must be set to the DNS hostname of the cluster endpoint.

Any forward proxies deployed in your environment that are in the path of the network traffic to StreamNative Cloud must be configured to forward the SNI extension unchanged.

For testing conformance with TLS SNI requirements, see [Test Connectivity](/cloud/networking/networking-testing).

See [TLS Extensions: Server Name Indication](https://datatracker.ietf.org/doc/html/rfc6066#section-3) and [TLS Protocol Version 1.3: Server Name Indication](https://datatracker.ietf.org/doc/html/rfc8446#section-4.2) for more details.

## Kafka client compatibility

Clients developed for Kafka versions 0.9 or later are compatible with StreamNative Cloud. Modern clients will automatically negotiate protocol versions or utilize an earlier protocol version that StreamNative Cloud accepts.

<Note title="Note">
  We recommend always using the latest supported version of a client.
</Note>

The following clients have been validated with StreamNative Cloud:

|          |                                                                                                                                                               |
| -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Language | Client                                                                                                                                                        |
| Java     | [Apache Kafka Java Client](https://kafka.apache.org/)                                                                                                         |
| C/C++    | [librdkafka](https://github.com/confluentinc/librdkafka)                                                                                                      |
| Go       | [confluent-kafka-go](https://github.com/confluentinc/confluent-kafka-go) [franz-go](https://github.com/twmb/franz-go) [sarama](https://github.com/IBM/sarama) |
| Python   | [confluent-kafka-python](https://github.com/confluentinc/confluent-kafka-python)                                                                              |
| Node.js  | [KafkaJS](https://github.com/tulios/kafkajs)                                                                                                                  |
| .NET     | [confluent-kafka-dotnet](https://github.com/confluentinc/confluent-kafka-dotnet)                                                                              |
| Rust     | [rust-rdkafka](https://github.com/fede1024/rust-rdkafka)                                                                                                      |

Other clients that use the Kafka protocol but have not been validated by StreamNative may still be compatible with StreamNative Cloud, subject to the limitations described below. This is particularly true for clients based on [librdkafka](https://github.com/confluentinc/librdkafka).

If you encounter any compatibility issues with a client not listed above, please reach out to the StreamNative team through the StreamNative Cloud console chat or [Support Portal](https://support.streamnative.io/) for assistance.

<Note title="Note">
  Specifically, librdkafka 2.5.0 and 2.5.3, and all other client SDKs based on these two versions are not supported (e.g. confluent-kafka-python 2.5.0 and 2.5.3). It's caused by a backward compatibility regression in librdkafka, which is fixed in 2.6.0 (see [librdkafka #4871](https://github.com/confluentinc/librdkafka/pull/4871)).
</Note>

## Unsupported Kafka features

StreamNative Cloud does not currently support the following Apache Kafka features:

* Managing SASL users with Kafka Admin APIs: Use Pulsar Admin instead.
* Managing Kafka ACLs with Kafka Admin APIs: Use [Pulsar ACLs](id:access-control) instead. For more details, see [Kafka ACLs](/cloud/security/access/access-control-lists/kafka-acls).
* Limited support for Topic Configuration: Since StreamNative Cloud is based on Apache Pulsar, the storage layer differs from Apache Kafka. The topic configuration only supports `cleanup.policy` to set the compact or delete retention policy.

If you encounter any issues while working with a Kafka tool, you can file a support ticket through the [Support Portal](https://support.streamnative.io/).


# Kafka Protocol and Features
Source: https://docs.streamnative.io/cloud/build/kafka-clients/compatibility/kafka-protocol-and-features



## Supported Kafka Requests

The current implementation of the Apache Kafka protocol in StreamNative Cloud supports all the functionalities in Classic Engine and limited functionalities in the Ursa Engine. Specifically, the following Apache Kafka requests are currently supported:

| API Key | Message                 | Classic Engine | Ursa Engine |
| ------- | ----------------------- | -------------- | ----------- |
| 0       | Produce                 | Y              | Y           |
| 1       | Fetch                   | Y              | Y           |
| 2       | ListOffsets             | Y              | Y           |
| 3       | Metadata                | Y              | Y           |
| 8       | OffsetCommit            | Y              | Y           |
| 9       | OffsetFetch             | Y              | Y           |
| 10      | FindCoordinator         | Y              | Y           |
| 11      | JoinGroup               | Y              | Y           |
| 12      | Heartbeat               | Y              | Y           |
| 13      | LeaveGroup              | Y              | Y           |
| 14      | SyncGroup               | Y              | Y           |
| 15      | DescribeGroups          | Y              | Y           |
| 16      | ListGroups              | Y              | Y           |
| 18      | ApiVersions             | Y              | Y           |
| 19      | CreateTopics            | Y              | Y           |
| 20      | DeleteTopics            | Y              | Y           |
| 21      | DeleteRecords           | Y              | Y           |
| 22      | InitProducerId          | Y              | Y (\*)      |
| 24      | AddPartitionsToTxn      | Y              | N           |
| 25      | AddOffsetsToTxn         | Y              | N           |
| 26      | EndTxn                  | Y              | N           |
| 28      | TxnOffsetCommit         | Y              | Y           |
| 32      | AlterConfigs            | Y              | Y           |
| 32      | DescribeConfigs         | Y              | Y           |
| 37      | CreatePartitions        | Y              | Y           |
| 44      | IncrementalAlterConfigs | Y              | Y           |
| 47      | OffsetDelete            | Y              | Y           |
| 51      | DeleteGroups            | Y              | Y           |
| 60      | DescribeCluster         | Y              | Y           |
| 61      | DescribeProducers       | Y              | N           |
| 65      | DescribeTransactions    | Y              | N           |
| 66      | ListTransactions        | Y              | N           |

**Notes**

* `InitProducerId` in Ursa Engine is partially supported when the `transactional.id` is **NOT** specified.

We're actively working on adding support for more Kafka features and more Kafka requests to both Classic Engine and Ursa Engine. Please [contact us](https://streamnative.io/contact) for specific feature requests or if you notice any discrepancies.

## Unsupported Kafka Requests

Because the Kafka protocol implementation in StreamNative Cloud is based on Pulsar for Classic Engine and Ursa for Ursa Engine, the following Kafka requests are irrelevant and not supports. For example, requests like:

### Kafka metadata or storage related

These requests are irrelevant to StreamNative Cloud since data storage and replication are handled differently - either in BookKeeper or in the underlying object storage. Therefore, these requests are unlikely to be supported.

| API Key | Request Name                |
| ------- | --------------------------- |
| 4       | LeaderAndIsr                |
| 5       | StopReplica                 |
| 6       | UpdateMetadata              |
| 7       | ControlledShutdown          |
| 23      | OffsetForLeaderEpoch        |
| 34      | AlterReplicaLogDirs         |
| 35      | DescribeLogDirs             |
| 43      | ElectLeaders                |
| 45      | AlterPartitionReassignments |
| 46      | ListPartitionReassignments  |
| 55      | DescribeQuorum              |
| 56      | AlterPartition              |
| 57      | UpdateFeatures              |
| 58      | Envelope                    |
| 64      | UnregisterBroker            |
| 67      | AllocateProducerIds         |
| 80      | AddRaftVoter                |
| 81      | RemoveRaftVoter             |

### Security related

These security and ACL-related requests are not supported because StreamNative Cloud provides more comprehensive security mechanisms, including OAuth2 and Role-Based Access Control (RBAC). For information about using Kafka ACLs in StreamNative Cloud, see [Manage Kafka ACLs](/cloud/security/access/access-control-lists/kafka-acls) to learn how to map Kafka ACLs to Pulsar ACLs.

| API Key | Request Name                 |
| ------- | ---------------------------- |
| 29      | DescribeAcls                 |
| 30      | CreateAcls                   |
| 31      | DeleteAcls                   |
| 38      | CreateDelegationToken        |
| 39      | RenewDelegationToken         |
| 40      | ExpireDelegationToken        |
| 41      | DescribeDelegationToken      |
| 50      | DescribeUserScramCredentials |
| 51      | AlterUserScramCredentials    |

### Quotas related

Kafka's quota feature is not supported. Please use Pulsar's quota management instead.

| API Key | Request Name         |
| ------- | -------------------- |
| 48      | DescribeClientQuotas |
| 49      | AlterClientQuotas    |

### API from latest KIPs

The API requests introduced from the latest KIPs are not yet supported. We are actively evaluating the feasibility of supporting them. If you have any specific feature requests, please [contact us](https://streamnative.io/contact).

| API Key | Request Name               | KIP                                                                                                                                                                                    |
| ------- | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 68      | ConsumerGroupHeartbeat     | [KIP-848: The Next Generation of the Consumer Rebalance Protocol](https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol) |
| 69      | ConsumerGroupDescribe      | [KIP-848: The Next Generation of the Consumer Rebalance Protocol](https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol) |
| 71      | GetTelemetrySubscriptions  | [KIP-714: Client metrics and observability](https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability)                                             |
| 72      | PushTelemetry              | [KIP-714: Client metrics and observability](https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability)                                             |
| 74      | ListClientMetricsResources | [KIP-1000: List Client Metrics Configuration Resources](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1000%3A+List+Client+Metrics+Configuration+Resources)                     |
| 75      | DescribeTopicPartitions    | [KIP-966: Eligible Leader Replicas](https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas)                                                             |

## Supported topic configs

StreamNative Cloud supports the following topic configurations:

* `cleanup.policy`: This configuration can be set to either `compact` or `compact,delete`. The policy controls topic compaction and is only effective when `compact` is included in the setting. Due to StreamNative Cloud's storage architecture, which preserves both raw and compacted segments rather than replacing individual log segments with compacted ones, we currently only support the values `compact` and `compact,delete`.

## Supported Schema Registry APIs

StreamNative's Kafka Schema Registry is compatible with [Confluent Schema Registry](https://docs.confluent.io/platform/current/schema-registry/develop/api.html).

Following are the supported APIs:

### Schemas

```bash theme={null}
GET /schemas/ids/{int: id}
GET /schemas/ids/{int: id}/subjects
GET /schemas/ids/{int: id}/schema
GET /schemas/ids/{int: id}/versions
GET /schemas/types
```

### Subjects

```bash theme={null}
GET /subjects
GET /subjects/(string: subject)/versions
GET /subjects/(string: subject)/versions/(versionId: version)
GET /subjects/(string: subject)/versions/(versionId: version)/schema
GET /subjects/(string: subject)/versions/(string: versionId)/referencedby
POST /subjects/(string: subject)
POST /subjects/(string: subject)/versions?normalize=(boolean: normalize)
DELETE /subjects/(string: subject)?permanent=(boolean: permanent)
DELETE /subjects/(string: subject)/versions/(string: versionId)
```

### Mode

```bash theme={null}
GET /mode
```

### Compatibility

```bash theme={null}
GET /compatibility/subjects/${subject}/versions/latest
```

### Config

```bash theme={null}
GET /config
GET /config/${subject}
PUT /config/${subject}
```


# Configuring Kafka Clients
Source: https://docs.streamnative.io/cloud/build/kafka-clients/config/config-kafka-client



You can use Kafka clients to produce and consume messages to and from a StreamNative Cloud cluster.

Before you start using Kafka clients, you need to configure them properly. This section provides the necessary configurations and general guidelines for Kafka clients.

## General Recommendations

Kafka client configurations provide flexibility and control over various aspects of the client's behavior, performance, security, reliability, and more. Properly configuring these settings helps optimize the client's interactions with the StreamNative Cloud cluster and ensures efficient message processing. The following are two specific areas where ensuring correct settings positively impacts the workload:

* **Performance**: Client configurations can be adjusted to optimize the client's performance. Adjusting properties that control batching, compression, linger, and prefetch can significantly impact client throughput, latency, and resource utilization.

* **Robustness**: Kafka clients need to handle errors with retries or fail gracefully until a solution can be implemented to resolve the issue. Ensuring the configuration is correct can enhance application resilience and ensure reliability for mission-critical workloads.

## Configuration Overview

Client configuration settings can be grouped into the following categories:

* **Connection and network**: A Kafka client must establish a connection with StreamNative Cloud clusters to produce and consume messages. This category includes settings for bootstrap servers, connection timeout, and network buffer sizes. Optimizing these settings can ensure reliable and efficient communication between your clients and StreamNative Cloud.

* **Authentication and security**: Kafka supports various security mechanisms, such as TLS encryption, SASL authentication, and authorization using ACLs. This category includes security-related settings, such as SSL certificates, authentication protocols, and user credentials. Properly configuring security settings ensure the confidentiality, integrity, and authenticity of the communication between clients and StreamNative Cloud.

* **Message delivery and processing**: Kafka clients can process messages in various ways, such as consuming messages from specific topics, committing message offsets, or specifying how to handle message errors. This category includes settings for message delivery guarantees, acknowledgment mechanisms, and error handling strategies. Properly configuring these settings can ensure consistent and reliable message delivery, optimize processing performance, and handle errors effectively.

## Connection and Network settings

### Bootstrap Servers

The bootstrap servers you need to configure for Kafka clients is the **Kafka Service URL** of your StreamNative Cloud cluster.

You can obtain the bootstrap servers URL in two ways:

* **Cloud Console**: Navigate to the **Cluster Details** page in the **Cluster Dashboard**, locate the **Kafka Service URL** and copy it.
* **snctl CLI**: Run the following command, replacing `<cluster-name>` with your StreamNative Cloud cluster name:
  ```bash theme={null}
  echo "$(snctl get pulsarclusters <cluster-name> -o jsonpath='{.spec.serviceEndpoints[0].dnsName}'):9093"
  ```

For more information about service URLs, see [Cluster Service URLs](/cloud/clusters/manage-clusters/cluster#cluster-service-urls).

### Tune DNS resolution

Please consider the following JVM settings when using Java clients to connect to StreamNative Cloud:

* JVM properties
  * `networkaddress.cache.ttl`: Set it to `30` seconds.
  * `networkaddress.cache.negative.ttl`: Set it to `0` seconds.
* Kafka Producer and Consumer settings
  * `consumer.client.dns.lookup`: Set it to `use_all_dns_ips`.
  * `producer.client.dns.lookup`: Set it to `use_all_dns_ips`.

### Eliminate Cross-AZ Networking Traffic

<Note title="Note">
  This feature is only available in **Ursa Engine** clusters running version `4.0.0.7` or later. **Classic Engine** clusters use cross-AZ replication for data durability and availability and cannot take advantage of this optimization.
</Note>

Ursa Engine leverages object storage to store data, eliminating the need for cross-AZ replication. This architecture provides two key benefits:

1. You can produce and consume messages across different availability zones without incurring additional inter-AZ networking costs
2. Kafka clients can connect exclusively to brokers within their same availability zone, reducing network latency and costs

#### Configure Availability Zone Affinity

<Note title="Note">
  For cluster versions 4.0.0.7 or older, you must set the `client.id` to match the availability zone ID to enable zone-aware routing.
</Note>

To enable zone-aware routing and optimize your network costs:

1. Ensure at least one broker is deployed in the same availability zone as your Kafka clients
2. Specify your availability **zone ID** in your client ID by appending `zone_id=<zone-id>` to the client ID. The client ID must follow this format: `zone_id=<zone-id>,key1=value1,key2=value2`

For example, if your application runs in availability zone `us-west-1a` and the zone ID is `usw-az1`, set your client ID to `zone_id=usw-az1,other=value`. This ensures your client connects to brokers in the same zone.

<Note title="Note">
  The `zone_id` in the client ID must exactly match the availability zone ID for zone-aware routing to work correctly.
</Note>

To find the availability zone ID where your application runs, refer to:

* [Availability Zone IDs for your AWS resources](https://docs.aws.amazon.com/ram/latest/userguide/working-with-az-ids.html)

## Authentication and Security settings

### Authentication

StreamNative Cloud uses [SASL/PLAIN](https://kafka.apache.org/documentation/#security_sasl_plain) authentication for Kafka client connections. To authenticate your Kafka clients, you'll need to:

1. Create a [**Service Account**](/cloud/security/authentication/service-accounts/service-accounts) and generate an API key. For details, see [API Keys](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview).

2. Configure the following authentication settings when initializing your Kafka producer or consumer:

* `sasl.mechanism=PLAIN` - Specifies SASL/PLAIN as the authentication mechanism
* `security.protocol=SASL_SSL` - Enables SASL authentication over SSL/TLS
* `sasl.username` - Can be set to any value as it is not used
* `sasl.password=token:<API KEY>` - Must be set to `token:` followed by your generated API key

#### Java Client Settings

An example of the properties file for Java based applications and clients is provided below:

```properties theme={null}
sasl.mechanism=PLAIN
security.protocol=SASL_SSL
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="unused" password="token:<API KEY>";
```

Please replace `<API KEY>` with the API key you generated.

#### librdkafka Settings

An example of the properties file for [librdkafka](https://github.com/edenhill/librdkafka) based applications and clients is provided below:

```properties theme={null}
sasl.mechanism=PLAIN
security.protocol=SASL_SSL
sasl.username=unused
sasl.password=token:<API KEY>
```

Please replace `<API KEY>` with the API key you generated.

## Common client settings

The following table provides several common client settings for **Producers** and **Consumers** that you can review for potential modification.

| Configuration property                   | Java default       | librdkafka default                 | Notes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| ---------------------------------------- | ------------------ | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `client.id`                              | empty string       | `rdkafka`                          | You should set the `client.id` to something meaningful in your application, especially if you are running multiple clinets or want to easily trace logs or activities to specific client instances. This setting is also important for zone-aware routing, as it helps StreamNative Cloud route the traffic to the correct availability zone to eliminate cross-AZ networking traffic. See [Eliminate Cross-AZ Networking Traffic](#eliminate-cross-az-networking-traffic) for more information. |
| `connections.max.idle.ms`                | 540000 ms (9 mins) | See librdkafka `socket.timeout.ms` | You can change this when an intermediate load balancer disconnects idle connections after inactivity. For example, AWS 350 seconds, Azure 4 minutes, Google Cloud 10 minutes.                                                                                                                                                                                                                                                                                                                    |
| `socket.connection.setup.timeout.max.ms` | 30000 ms (30 secs) | not available                      | librdkafka doesn't have exponential backoff for this timeout, so you can increase `socket.connection.setup.timeout.ms` to avoid connection failures.                                                                                                                                                                                                                                                                                                                                             |
| `socket.connection.setup.timeout.ms`     | 10000 ms (10 secs) | 30000 ms (30 secs)                 | librdkafka doesn't have exponential backoff for this timeout, so you can increase this value to avoid connection failures.                                                                                                                                                                                                                                                                                                                                                                       |
| `metadata.max.age.ms`                    | 300000 ms (5 mins) | 900000 ms (15 mins)                | librdkafka has the `topic.metadata.refresh.interval.ms` setting that defaults to 300000 ms (5 mins).                                                                                                                                                                                                                                                                                                                                                                                             |
| `reconnect.backoff.max.ms`               | 1000 ms (1 second) | 10000 ms (10 seconds)              |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `reconnect.backoff.ms`                   | 50 ms              | 100 ms                             |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `max.in.flight.requests.per.connection`  | 5                  | 1000000                            | librdkafka produces to a single partition per batch, setting it to 5 limits producing to 5 partitions per broker                                                                                                                                                                                                                                                                                                                                                                                 |


# Configuring Kafka Consumer
Source: https://docs.streamnative.io/cloud/build/kafka-clients/config/config-kafka-consumer



An Apache Kafka consumer is a client application that subscribes to one or more topics and reads & processes messages from them. This section describes the configuration options available for Kafka consumers.

## Consumer Configuration

This following sections describe the key configuration settings for Kafka consumers and explain how they affect consumer behavior.

For common client settings, such as networking and authentication settings, see [Configuring Kafka Clients](/cloud/build/kafka-clients/config/config-kafka-client).

### Group Configuration

#### `group.id`

A unique identifier for the consumer group. While optional, you should always configure a group ID unless you are using the simple assignment API and don't need to store offsets in Kafka.

#### `session.timeout.ms`

Controls how long a consumer can go without sending heartbeats to the coordinator before being considered failed. The default is `10` seconds for C/C++ and Java clients. You can increase this value to avoid excessive rebalancing due to poor network connectivity or long GC pauses. However, using a larger timeout means it will take longer for the coordinator to detect crashed consumers and reassign their partitions to other group members. For normal shutdowns, the consumer explicitly leaves the group, triggering an immediate rebalance.

#### `heartbeat.interval.ms`

Controls how frequently the consumer sends heartbeats to the coordinator. These heartbeats also help detect when rebalancing is needed, so a lower interval generally enables faster rebalancing. The default is `3` seconds. For larger consumer groups, consider increasing this value to reduce coordinator load.

#### `max.poll.interval.ms`

Specifies the maximum allowed time between calls to the consumer's poll method (or `Consume` method in .NET) before the consumer is considered failed. The default is `300` seconds and can be increased if your application needs more time to process messages. For Java consumers, you can also adjust `max.poll.records` to control how many records are processed in each poll iteration.

### Offset Management Configuration

There are two main settings that affect how offsets are managed: whether automatic offset committing is enabled and the offset reset policy.

#### `enable.auto.commit`

Controls whether the consumer automatically commits offsets periodically (default is `true`). When enabled, offsets are committed at the interval specified by `auto.commit.interval.ms`, which defaults to `5` seconds. When disabled, you must manually commit offsets using `commitSync()` or `commitAsync()`.

#### `auto.offset.reset`

Determines how the consumer behaves when it needs to read from a position with no committed offset, or when the committed offset is invalid (out of range). This can occur when:

* The consumer group is first created
* The committed offset has been deleted due to retention policies
* The consumer requests an offset that does not exist

Valid values are:

* `latest` (default): Start reading from the newest available messages
* `earliest`: Start reading from the beginning of the topic
* `none`: Throw an exception if no previous offset is found

### Partition Assignment Configuration

#### `partition.assignment.strategy`

Controls how partitions are distributed among consumer instances when using group management. All consumers in the same group must use the same strategy.

This setting accepts a comma-separated list of fully qualified class names that implement the `PartitionAssignor` interface. Multiple strategies can be specified to support transitioning between strategies while maintaining compatibility with consumers using the previous strategy.

The following strategies are available:

* **Range Assignment (Default)**

  * Class: `org.apache.kafka.clients.consumer.RangeAssignor`
  * Behavior: Distributes partitions of each topic evenly across consumers in a group after sorting both partitions and consumers
  * Best for: Cases where partition count exceeds consumer count
  * Limitation: May result in uneven distribution if partition count is not divisible by consumer count

* **Round Robin Assignment**

  * Class: `org.apache.kafka.clients.consumer.RoundRobinAssignor`
  * Behavior: Distributes partitions one by one across consumers in a round-robin fashion
  * Best for: Scenarios requiring even distribution regardless of partition count
  * Limitation: May trigger more frequent rebalances compared to Range Assignment

* **Sticky Assignment**

  * Class: `org.apache.kafka.clients.consumer.StickyAssignor`
  * Behavior: Maintains stable partition assignments across rebalances while ensuring balanced distribution
  * Best for: Applications sensitive to partition reassignment overhead
  * Limitation: May not achieve optimal balance when cluster topology changes frequently

* **Cooperative Sticky Assignment**

  * Class: `org.apache.kafka.clients.consumer.CooperativeStickyAssignor`
  * Behavior: Enables incremental rebalancing while maintaining sticky assignments
  * Best for: Large consumer groups where minimizing rebalance impact is critical
  * Limitation: Requires all consumers to support cooperative rebalancing protocol

## Message Handling

The Java consumer performs all I/O and processing in the foreground thread, while librdkafka-based clients (C/C++, Python, Go, and C#) use a background thread. This architectural difference has several important implications:

1. **Thread Safety**: In librdkafka-based clients, polling is thread-safe and can be used from multiple threads. This allows you to parallelize message handling across multiple threads, as poll operations retrieve messages from a queue that's filled by the background thread.

2. **Background Processing**: With librdkafka-based clients, heartbeats and rebalancing occur in the background thread. This provides two key effects:
   * Advantage: Message handling won't cause the consumer to miss a rebalance
   * Disadvantage: If your message processor fails, the background thread continues sending heartbeats, causing the consumer to retain its partitions and potentially accumulate read lag until the process is terminated

Despite these architectural differences, the clients' approaches are conceptually similar. For example, you can implement a similar pattern in the Java client by introducing a queue between the poll loop and message processors. In this setup, the poll loop would populate the queue, and processors would consume messages from it, effectively creating a background processing model.


# Configuring Kafka Producer
Source: https://docs.streamnative.io/cloud/build/kafka-clients/config/config-kafka-producer



An Apache Kafka producer is a client application that publishes (sends) messages to StreamNative Cloud. This section gives an overview of the Kafka producer and an introduction to the configuration settings for Producers.

## Producer Configuration

The following sections describe the key configuration settings for Kafka producers and explain how they affect producer behavior.

For common client settings, such as networking and authentication settings, see [Configuring Kafka Clients](/cloud/build/kafka-clients/config/config-kafka-client).

### Message Durability

#### `acks`

Controls the durability of messages written to StreamNative Cloud. This setting has three possible values:

* `all` (default): Provides the strongest durability guarantee. The broker will wait for the message to be successfully persisted to the storage layer (either BookKeeper or object storage) before sending an acknowledgment response.

* `1`: Requires acknowledgement only from the owner broker. Provides moderate durability with better performance than `all`.

* `0`: Provides no durability guarantees but maximum throughput. The broker does not send any response, so you cannot verify message delivery or determine message offsets.

For C/C++, Python, Go and .NET clients, this is configured per-topic. To apply globally, use:

* C/C++: `default_topic_conf` sub-configuration
* Python, Go, .NET: `default.topic.config` sub-configuration

### Message Ordering

By default, messages are written to the broker in the same order that they are received by the producer client. However, certain configuration settings can affect this ordering.

#### `retries`

This setting controls message retry attempts when a send fails. When set to a value greater than `0`, retries are enabled (the default is `0`, which disables retries). With retries enabled, message re-ordering can occur if a retry succeeds after subsequent messages have already been written successfully.

#### `max.in.flight.requests.per.connection`

To maintain strict message ordering while using retries, set this value to `1`. This ensures only one request can be sent to the broker at a time, preventing any potential re-ordering. Note that when retries are disabled, the broker preserves the order of writes it receives, but failed sends can create gaps in the message sequence.

### Batching and Compression

Kafka producers attempt to collect sent messages into batches to improve throughput and efficiency.

Use the following settings to control batching and compression:

#### `batch.size`

Controls the maximum size in bytes of each message batch for the Java client. When a batch reaches this size, it is sent to the broker.

#### `linger.ms`

Use this setting to control how long the producer waits to allow more messages to accumulate in a batch before sending it. A longer linger time increases the likelihood of filling batches but adds latency.

#### `compression.type`

Enable compression with this setting. Compression covers full message batches, so larger batches typically achieve higher compression ratios. When using Snappy compression, you need write access to the `/tmp` directory. If you don't have write access to the `/tmp` directory because it's set to `noexec`, you can specify an alternate directory path that you have write access to:

```
-Dorg.xerial.snappy.tempdir=<path>
```

#### `batch.num.messages`

Use this setting with the C/C++, Python, Go, and .NET clients to set a limit on the number of messages contained in each batch.

### Queuing Limit

#### `buffer.memory`

Controls the total memory available to the Java client for collecting unsent messages. When this limit is reached, the producer will block additional sends for up to `max.block.ms` before raising an exception.

#### `request.timeout.ms`

Sets a timeout to prevent records from being queued indefinitely. If this timeout expires before a message is successfully sent, the message will be removed from the queue and an exception will be thrown. The C/C++, Python, Go, and .NET clients have similar settings.


# Connect to your cluster using Lenses
Source: https://docs.streamnative.io/cloud/build/kafka-clients/integrations/cloud-connect-lenses



<Note title="Note">
  This QuickStart assumes that you have created a StreamNative cluster with the Kafka protocol enabled, created a service account, and granted the service account `produce` and `consume` permissions to a namespace for the target topic.
</Note>

This document shows how to connect to your StreamNative cluster using [Lenses](https://lenses.io/) through Token authentication.

## Before you begin

<Note title="Note">
  * Before using an API key, verify that the service account is authorized to access the resources, such as tenants, namespaces, and topics.
  * The password for different utilities as `kcat` will be equal to `token:<API KEY>`.
</Note>

You can follow the instructions to [create an API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#using-api-keys-to-connect-to-your-cluster) for the service account you choose to use.

* [Install Lenses](https://docs.lenses.io/5.1/installation/getting-started/).

## Steps

This section describes how to connect to your StreamNative cluster and export data from your StreamNative cluster using Lenses.

1. Go to the Lenses UI and configure your StreamNative cluster.

   * \[1] Bootstrap Servers: the Kafka service URL of your StreamNative cluster.
   * \[2] Security Protocol: the security protocol of your StreamNative cluster.
   * \[3] SASL Mechanism: the security mechanism of your StreamNative cluster.
   * \[4] JAAS Configuration
     * username: the tenant and namespace name, in the format of `<tenant-name>/<namespace-name>`.
     * password: the token of your service account, in the format of `token:<API key of your service account>`

   <img alt="configure pulsar cluster in Lenses" />

2. View data in the target topic.

   a. On the left navigation pane of the Lenses UI, click **Explore** to navigate to the **Explore** page.
   b. Select the target topic.

   <img alt="view data in the kop-topic topic" />

   The message in the topic `kop-topic` looks like the below:

   ```json theme={null}
   {
     "value": {
       "address": {
         "street": "Summer Place",
         "streetNumber": "79",
         "apartmentNumber": "",
         "postalCode": "96518",
         "city": "San Francisco"
       },
       "firstName": "Skylar",
       "middleName": "Skylar",
       "lastName": "Vega",
       "email": "skylarvega@gmail.com",
       "username": "skylarv",
       "password": "BAhSz5sB",
       "sex": "FEMALE",
       "telephoneNumber": "728-020-424",
       "dateOfBirth": -499301348331,
       "age": 69,
       "company": {
         "name": "Klein",
         "domain": "klein.biz",
         "email": "contact@klein.biz",
         "vatIdentificationNumber": "62-0006870"
       },
       "companyEmail": "skylar.vega@klein.biz",
       "nationalIdentityCardNumber": "860-15-3193",
       "nationalIdentificationNumber": "",
       "passportNumber": "TbfdoEIBP"
     }
   }
   ```

3. Query data in the target topic.

   a. From the **Header Bar** menu, go to the **Dashboard** panel.

   b. On the side navigation, select **SQL Studio** under the **Data** section.

   <img alt="query data in the kop-topic topic" />


# Build Kafka Client Applications on StreamNative Cloud
Source: https://docs.streamnative.io/cloud/build/kafka-clients/kafka-on-cloud



StreamNative Cloud provides two ways to use Kafka:

* **[Kafka Service](/kafka/overview)** — A fully managed, **native Apache Kafka** service. Kafka Service runs native Apache Kafka on the Ursa Engine with no compatibility layer. Your existing Kafka clients, tools, and ecosystems work without modification. This is the recommended option for new Kafka workloads.

* **Kafka Compatibility on Pulsar Clusters (via KSN)** — Pulsar Clusters can serve Kafka clients through KSN, which translates the Kafka protocol to underlying storage. This option lets you access your Pulsar data using Kafka clients. All clusters are powered by the Ursa Engine. The previously called "Ursa" clusters are cost-optimized Pulsar Clusters with Kafka compatibility.

<Tip>
  If you are starting a new Kafka project or migrating from Amazon MSK, Confluent, or self-managed Kafka, use [Kafka Service](/kafka/overview) for full native Kafka support.
</Tip>

## Kafka Compatibility on Pulsar Clusters (KSN)

The rest of this page covers Kafka compatibility when using Pulsar Clusters with KSN enabled. For native Kafka Service documentation, see the [Kafka Service overview](/kafka/overview).

KSN is a protocol handler that enables Kafka clients to connect to Pulsar Clusters. Because KSN translates Kafka requests to Pulsar operations, some Kafka features have differences in behavior. The following tables detail the features supported by KSN on Latency Optimized and Cost Optimized Pulsar Clusters.

### Kafka Protocol Support

<Note title="Note">
  KStreams and KSqlDB support on Cost-Optimized clusters has certain limitations. It does not support functionalities that require transactions and topic compaction.
</Note>

|                       | Latency Optimized | Cost Optimized        | Open-source KoP |
| --------------------- | ----------------- | --------------------- | --------------- |
| Publish & Consume     | YES               | YES                   | YES             |
| Kafka Schema Registry | YES               | YES                   | YES             |
| Transactions          | YES               | Coming Soon           | YES             |
| Compacted Topic       | YES               | Coming Soon           |                 |
| KStreams Integration  | YES               | Yes, with limitations |                 |
| KSqlDB Integration    | YES               | Yes, with limitations |                 |

### Production Readiness

|                                      | Latency Optimized | Cost Optimized | Open-source KoP |
| ------------------------------------ | ----------------- | -------------- | --------------- |
| OAuth Authentication                 | YES               | YES            | -               |
| Kubernetes Authentication            | YES               | YES            | -               |
| RBAC                                 | YES               | YES            | -               |
| Schema Registry OAuth Authentication | YES               | YES            | -               |
| Schema Registry RBAC                 | Yes               | Yes            | -               |
| TLS                                  | YES               | YES            | YES             |
| Authorization                        | YES               | YES            | YES             |
| Multi-AZ / Multi-Region Clusters     | YES               | YES            | -               |
| Geo-replication                      | YES               | YES            | -               |

### Deployments and Efficient Operations

|                        | Latency Optimized | Cost Optimized | Open-source KoP |
| ---------------------- | ----------------- | -------------- | --------------- |
| Serverless / Dedicated | YES               | YES            | -               |
| BYOC                   | YES               | YES            | -               |
| Private Cloud          | YES               | Coming Soon    | -               |
| On-prem (self-managed) | YES               | Coming Soon    | YES             |
| Auto-scaling           | YES               | YES            | -               |
| Cloud Console / UI     | YES               | YES            | -               |

You can use StreamNative Cloud for your existing Kafka applications and services without migrating the code. See a full list of [supported Kafka clients](/cloud/build/kafka-clients/kafka-on-cloud#kafka-clients) and [Kafka Compatibility](/cloud/build/kafka-clients/compatibility/kafka-compatibility).

## Get started

To get started with StreamNative Cloud using Kafka, see the [QuickStart guide](/cloud/get-started/quickstart-kafka) to learn how to set up a cluster with Kafka protocol enabled and configure a Kafka client for producing and consuming messages.

For language-specific setup instructions, refer to our [Kafka Client Guides](/clients/kafka-clients/kafka-clients-overview) which provide QuickStart tutorials for your preferred programming language.

## Use Kafka Client Setup Wizard

After provisioning your cluster, StreamNative Console provides a step-by-step wizard to help you set up Kafka client libraries and tools. The wizard guides you through the basic setup and configuration process, including selecting or creating service accounts, downloading key files or tokens, installing client libraries, and generating sample code to run.

To get started with the Kafka client setup wizard, follow these steps.

1. [Navigate to your StreamNative Cloud cluster](/cloud/get-started/cloud-console#switch-a-cluster).

2. On the left navigation pane, in the **Clients** section, click **Kafka Clients**.

   <img alt="gif of kafka client setup process through wizard" />

3. Follow the wizard to generate the sample code you need for connecting to your StreamNative cluster.

With a copy-and-paste, you can run the given sample code to produce and consume messages.

## Kafka Clients

<span />

| Language             | References                                                                                                                                                                                                                        | Description                                                                |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- |
| Kafka Java client    | [QuickStart](/cloud/build/kafka-clients/quick-starts/cloud-connect-kafka-java) \| [Client Guide](/clients/kafka-clients/java/tutorial/kafka-java-introduction) \| [Tutorial](/clients/kafka-clients/java/kafka-java-client-guide) | Java producer and consumer shipped with Apache Kafka.                      |
| Kafka C/C++ client   | [QuickStart](/cloud/build/kafka-clients/quick-starts/cloud-connect-kafka-c) \| [Tutorial](/clients/kafka-clients/c-and-c++/tutorial/kafka-c-introduction)                                                                         | librdkafka, a C/C++ library for Kafka.                                     |
| Kafka Python client  | [QuickStart](/cloud/build/kafka-clients/quick-starts/cloud-connect-kafka-python) \| [Tutorial](/clients/kafka-clients/python/tutorial/kafka-python-introduction)                                                                  | Python client that provides high-level producer, consumer and AdminClient. |
| Kafka Go client      | [QuickStart](/cloud/build/kafka-clients/quick-starts/cloud-connect-kafka-go) \| [Tutorial](/clients/kafka-clients/go/tutorial/kafka-go-introduction)                                                                              | Go client that offers a producer and consumer for Kafka.                   |
| Kafka Node.js client | [QuickStart](/cloud/build/kafka-clients/quick-starts/cloud-connect-kafka-nodejs) \| [Tutorial](/clients/kafka-clients/node.js/tutorial/kafka-js-introduction)                                                                     | Node.js client that provides Kafka APIs                                    |
| Kafka .NET client    | [Tutorial](/clients/kafka-clients/.net/tutorial/kafka-dotnet-introduction)                                                                                                                                                        | .NET client that provides a high-level producer, consumer and AdminClient. |

## Kafka CLI & Tools

You can use the Kafka CLI tools to connect to your StreamNative cluster. You can see a quickstart guide [here](/cloud/build/kafka-clients/quick-starts/cloud-connect-kafka-cli). For more information, see [Use Kafka Tools with StreamNative Cloud](/tools/cli/other-tools/use-kafka-tools-with-streamnative-cloud).

## Kafka Connect

StreamNative Cloud provides full compatibility with Kafka Connect, offering two deployment options: you can self-host Kafka Connect connectors in your own environment (see examples [here](/cloud/connect/self-host-connectors/cloud-connect-elasticsearch)), or leverage [fully managed Kafka Connect connectors](/cloud/connect/kafka-connect/kafka-connect-overview) running directly in StreamNative Cloud.

## Kafka Streams

You can also build data streaming applications using Kafka Streams. See the [Kafka Streams QuickStart](/cloud/process/kafka-streams-and-ksql/cloud-connect-kafka-stream) for more information. Please note that StreamNative Cloud doesn't host any Kafka Streams applications.

## KSQL

You can also build data streaming applications using KSQL. See the [KSQL QuickStart](/cloud/process/kafka-streams-and-ksql/cloud-connect-ksql) for more information. Please note that StreamNative Cloud doesn't host the KSQL service.

## Learn more advanced topics

* [Multi Tenancy](/cloud/build/kafka-clients/advanced-features/kafka-multi-tenancy)
* [Kafka Transactions](/cloud/build/kafka-clients/advanced-features/kafka-transaction)
* [Use Kafka Schema Registry](/cloud/governance/kafka-schemas/kafka-schema-registry)
* [Use Kafka Compacted Topic](/cloud/build/kafka-clients/advanced-features/kafka-compacted-topic)

## Integrations

* [Lenses](/cloud/build/kafka-clients/integrations/cloud-connect-lenses)

## Related topics

* [Kafka Service overview](/kafka/overview) — Native Apache Kafka on StreamNative Cloud
* [Understand Kafka-on-Pulsar](https://dzone.com/articles/understanding-kafka-on-pulsar-kop)
* [Build applications using Pulsar clients](/cloud/build/pulsar-clients/qs-connect)


# Kafka Rest API Quickstart
Source: https://docs.streamnative.io/cloud/build/kafka-clients/kafka-rest-api



The StreamNative Kafka REST API provides a comprehensive HTTP-based interface for interacting with your Kafka clusters. Apache Kafka itself does not come with a native REST API. This feature allows you to manage critical resources and produce/consume messages without needing native Kafka clients or complex library setups.

StreamNative's Kafka REST API implementation provides:

* **HTTP-based Kafka Operations**: Manage topics, produce/consume messages, and administer your cluster using any language and standard tools like curl, without needing native Kafka client libraries.
* **Full Protocol Compatibility**: Faithfully supports the Kafka protocol, ensuring seamless integration and expected behavior for all standard operations.
* **Built-in Security**: Integrated with StreamNative's authentication and authorization systems
* **Multi-tenancy Support**: Native support for StreamNative's tenant/namespace isolation model

## Prerequisites

Before using the Kafka REST API, ensure you have:

* A StreamNative Cloud account with an active Kafka-enabled cluster
* Appropriate permissions to create service accounts and manage Kafka resources
* Basic familiarity with REST APIs and HTTP tools like `curl`

### Step 1: Create a service account

<Note title="Note">
  Currently, you can't edit a service account. If you need a service account to have Super Admin access, make sure to enable it when creating the service account. By default, service accounts do not have Super Admin enabled.
</Note>

To create a service account, follow these steps.

1. On the left navigation pane, click **Service Accounts**.

2. Click **Create Service Account**.

3. (Optional) Select **Super Admin** to grant the service account with Super admin access to a namespace or tenant.

4. Enter a name for the service account, and then click **Confirm**.

### Step 2: Create an API key for your service account

<Note title="Note">
  Before using an API key, verify that the service account is authorized to access the resources, such as tenants, namespaces, and topics.
</Note>

You can follow the instructions to [create an API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#using-api-keys-to-connect-to-your-cluster) for the service account you choose to use.

### Step 3: Grant service account permissions

<Note title="Note">
  If you use a Super Admin service account, you can skip this step because a Super Admin service account already has the required permissions.
</Note>

You can grant permissions to the service account using RBAC. For a description of the available permissions, see the [predefined roles](/cloud/security/access/rbac/manage-rbac-roles#quick-reference). Granting permissions via the UI will be supported soon.

### Step 4: Get the HTTP Service URL of your StreamNative cluster

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

<Note>
  For the Kafka REST API, you need to use the **HTTP Service URL (TLS)** endpoint.
</Note>

### Step 5: Get topic list

The following example shows how to list topics using the Kafka REST API. For a complete list of all available API, see the full [Kafka REST API Reference](/api-references/kafka-rest-api).

```shell theme={null}
curl --location --request GET 'https://<your-http-service-url-tls>/rest-kafka/admin/v1/topics' \
--header 'Authorization: Bearer <your-token>'
```

<Note title="Security Best Practices">
  **Never hardcode authentication tokens in your applications.** Instead:

  * Store tokens in secure environment variables or secret management systems
  * Implement token rotation policies to regularly refresh credentials
  * Use service accounts with minimal required permissions following the principle of least privilege
  * Always use HTTPS (TLS) endpoints to encrypt data in transit
</Note>

Response
200 - A successful request returns a list of topic objects.

```json theme={null}
{
    "kind": "KafkaTopicList",
    "data": [
        {
            "kind": "KafkaTopic",
            "topic_name": "test-tenant.test-ns.topic-1",
            "is_internal": false,
            "partitions_count": 3
        },
        {
            "kind": "KafkaTopic",
            "topic_name": "topic-2",
            "is_internal": true,
            "partitions_count": 2
        },
        {
            "kind": "KafkaTopic",
            "topic_name": "topic-3",
            "is_internal": false,
            "partitions_count": 1
        }
    ]
}
```


# Migrating to StreamNative
Source: https://docs.streamnative.io/cloud/build/kafka-clients/migrating-to-streamnative



StreamNative adds native support for the Kafka protocol, but it does not mean StreamNative is an exact copy of Apache Kafka. StreamNative not only maintains compatibility with the Kafka protocol but also incorporates many of Pulsar's excellent features. It is crucial for Kafka users to understand these differences before migrating to StreamNative. By doing so, they can avoid any potential losses resulting from the disparities between the two systems.

### Data Retention

Pulsar has a different data retention policy by default. In Pulsar, consumed and acknowledged data from all subscriptions, or data from topics with no subscriptions, is systematically removed from the topic segment by segment. In contrast, Kafka retains data within the topic for a fixed duration of 7 days, regardless of whether it has been consumed or not. However, StreamNative adopts Pulsar's approach, offering users a data retention policy that can discern data consumption patterns. Therefore, when migrating to StreamNative, it is necessary to proactively adjust the data retention policy to prevent the deletion of data that has been written, after consumption, or in the absence of subscriptions. Certainly, if Pulsar's behavior aligns with your expectations, there would be no need to modify the policy.

You can follow the Pulsar Admin CLI, Pulsar Admin API or Pulsar Admin REST API to set the data retention policy for the namespace or topic.

In practice, you can see that the retention policy is not configured in the `public/default` namespace via the Pulsar admin CLI:

```bash theme={null}
$ ./bin/pulsar-admin namespaces get-retention public/default
null
```

`null` means the retention policy is not set. The default behavior is that all messages could be deleted after the retention period. You can set the retention policy for the namespace with the following command:

```bash theme={null}
$ ./bin/pulsar-admin namespaces set-retention -t 7d -s -1 public/default
```

This will result in the same behavior as Kafka, where messages will be deleted after 7 days. You can confirm this by running the following command:

```bash theme={null}
$ ./bin/pulsar-admin namespaces get-retention public/default
{
  "retentionTimeInMinutes" : 10080,
  "retentionSizeInMB" : -1
}
```

See more details [here](https://pulsar.apache.org/docs/next/cookbooks-retention-expiry/) for how Pulsar's retention policy works.

#### Set data retention for namespace

* [Admin CLI](https://pulsar.apache.org/reference/#/3.1.x/pulsar-admin/namespaces?id=set-retention)
* [Admin API](https://pulsar.apache.org/api/admin/3.1.x/org/apache/pulsar/client/admin/Namespaces.html#setRetention\(java.lang.String,org.apache.pulsar.common.policies.data.RetentionPolicies\))
* [Admin REST API](https://pulsar.apache.org/admin-rest-api/?version=3.1.0#operation/Namespaces_setRetention)

#### Set data retention for topic

* [Admin CLI](https://pulsar.apache.org/reference/#/3.1.x/pulsar-admin/topicPolicies?id=set-retention)
* [Admin API](https://pulsar.apache.org/api/admin/3.1.x/org/apache/pulsar/client/admin/TopicPolicies.html#setRetention\(java.lang.String,org.apache.pulsar.common.policies.data.RetentionPolicies\))
* [Admin REST API](https://pulsar.apache.org/admin-rest-api/?version=3.1.0#operation/PersistentTopics_setRetention)


# Optimize Kafka Client for Availability
Source: https://docs.streamnative.io/cloud/build/kafka-clients/optimize-and-tune/optimize-kafka-availability



To optimize for high availability, you need to tune your Kafka application to recover quickly from failure scenarios. This involves configuring parameters that control failure detection, recovery, and state restoration.

The configuration parameters discussed in this guide have varying ranges of values. The optimal settings depend on your chosen [data streaming engine](/cloud/overview/data-streaming-engine), specific requirements, and environmental factors such as average message size, number of partitions, and other system characteristics. Therefore, [benchmarking](/cloud/build/kafka-clients/optimize-and-tune/optimize-kafka-clients#benchmarking) is essential to validate and fine-tune the configuration for your particular application and environment.

## Write Quorum and Acknowledgment Quorum Size

<Note title="Note">
  The following configurations apply to the **Classic Engine** only. They do not apply to the **Ursa Engine**.
</Note>

When a producer sets `acks=all` or `acks=-1`, two configuration parameters control message replication:

* `managedLedgerDefaultWriteQuorum`: Specifies the replication factor for storing messages (number of replicas)
* `managedLedgerDefaultAckQuorum`: Specifies the minimum number of replicas that must acknowledge a write before it is considered successful

If the minimum acknowledgment quorum cannot be met, the producer raises an exception. To improve data availability:

* Increase the write quorum to maintain more replicas of the data
* Increase the difference between write quorum and acknowledgment quorum sizes to improve write availability while maintaining durability guarantees

These configurations can be set on a per-namespace or per-topic basis based on your specific requirements.

## Consumer failures

Consumers in a consumer group can share processing load. If a consumer unexpectedly fails, StreamNative Cloud detects the failure and rebalances the partitions amongst the remaining consumers in the consumer group. Consumer failures can be either hard failures (for example, `SIGKILL`) or soft failures (for example, **expired session timeouts**). These failures are detected when consumers fail to send heartbeats or `poll()` calls.

Consumer liveness is maintained with a heartbeat (running in a background thread since [KIP-62](https://cwiki.apache.org/confluence/display/KAFKA/KIP-62%3A+Allow+consumer+to+send+heartbeats+from+a+background+thread)). The `session.timeout.ms` configuration parameter dictates the timeout used to detect failed heartbeats. You can increase the session timeout to account for potential network delays and avoid soft failures.

Soft failures most commonly occur in two cases:

* When `poll()` returns a batch of messages that take too long to process
* When a JVM GC pause takes too long

If you have a `poll()` loop that spends significant time processing messages, you can:

* Increase `max.poll.interval.ms` to allow more time between fetching records
* Reduce `max.poll.records` to decrease the batch size returned

While higher session timeouts increase the time needed to detect and recover from consumer failures, failed client incidents are generally less frequent than network issues.

## Summary

Here's a summary of key configurations for optimizing availability:

### Consumer Configurations

| Configuration        | Recommended Value | Default Value | Description                               |
| -------------------- | ----------------- | ------------- | ----------------------------------------- |
| `session.timeout.ms` | 30000-60000       | 45000         | Time before consumer is considered failed |


# Optimize and Tune Kafka Clients
Source: https://docs.streamnative.io/cloud/build/kafka-clients/optimize-and-tune/optimize-kafka-clients



Before you roll out your Kafka client applications to the production, you can and should benchmark and optimize your applications based on your application's SLAs to tune and optimize performance.

## Benchmarking

Benchmark testing is essential because there is no one-size-fits-all configuration for Kafka applications. The optimal configuration depends on your specific use case, enabled features, data profile, and other factors. You should run benchmark tests when planning to tune Kafka clients beyond the default settings. Understanding your application's performance profile is crucial, especially when choosing the right [data streaming engine](/cloud/overview/data-streaming-engine) and optimizing for throughput or latency. Benchmark test results can also help determine the right size of your StreamNative Cloud cluster and the appropriate number of partitions and producer/consumer processes.

<Note title="Note">
  If you need help with sizing your StreamNative Cloud cluster, you can always [contact us](https://streamnative.io/contact) for assistance.
</Note>

### Initial Performance Baseline

Start by measuring baseline performance using:

* Kafka tools `kafka-producer-perf-test` and `kafka-consumer-perf-test` that are bundled in the Kafka distribution for JVM clients
* [`rdkafka_performance`](https://github.com/confluentinc/librdkafka/blob/master/examples/rdkafka_performance.c) interface for non-JVM clients using [librdkafka](https://github.com/confluentinc/librdkafka)

These tools provide a baseline performance measurement without application logic. Note that these performance tools do not support Schema Registry.

### Application Testing

1. Test your application using default Kafka configuration parameters first

2. Establish producer baseline performance:

   * Remove upstream dependencies
   * Use mock data generation or sanitized production data
   * Ensure test data reflects production data characteristics
   * When testing with compression, be mindful that unrealistic mock data (repeated patterns, zero padding) may show better compression than production data

3. Producer benchmarking:

   * Start with a single producer on one server
   * Measure throughput using producer metrics
   * Incrementally increase producer processes to find optimal count per server

4. Consumer benchmarking:
   * Follow similar process as producer testing
   * Start with single consumer, then increase processes
   * Determine optimal number of consumer processes per server

### Tuning Process

1. Run benchmark tests with different configuration parameters aligned with your application's SLAs
2. Focus on a subset of parameters - avoid changing defaults without understanding system impact
3. Iterate through: adjust settings, test, analyze results, and repeat
4. Continue until meeting throughput and latency requirements

## Defining Application SLAs

While getting a Kafka client application running is relatively quick, proper tuning is essential before production deployment. Different use cases have different requirements, so you must identify your primary service goals and align them with your application's SLAs. For a modern cloud data streaming platform, it is impossible to achieve all three properties of `Cost`, `Availability`, and `Performance` based on the [New CAP Theorem](https://streamnative.io/blog/cap-theorem-for-data-streaming), so you need to find the right balance among them.

* [Throughput](/cloud/build/kafka-clients/optimize-and-tune/optimize-kafka-throughput)
* [Latency](/cloud/build/kafka-clients/optimize-and-tune/optimize-kafka-latency)
* [Durability](/cloud/build/kafka-clients/optimize-and-tune/optimize-kafka-durability)
* [Availability](/cloud/build/kafka-clients/optimize-and-tune/optimize-kafka-availability)

### Considerations

Consider these factors when determining service goals to align with your application's SLAs:

1. The specific use cases your Kafka applications serve
2. Critical application and business requirements
3. Kafka's role in your business applications and services

Before tuning your Kafka client application, it's crucial to discuss business requirements and goals with your team to determine which metrics to optimize. There are two key reasons for this:

First, there are inherent trade-offs between different performance goals. You cannot simultaneously maximize throughput, latency, durability, and availability. For example, improving throughput often comes at the cost of increased latency, while maximizing durability can impact availability. While optimizing one metric doesn't completely sacrifice the others, these goals are interconnected and require careful balance.

Second, identifying your applications SLAs helps guide Kafka configuration tuning. By understanding user expectations, you can optimize the system appropriately. Consider which of these goals is most important for your use case:

**High Throughput (maximizing data movement rate):**

* Best for: High-volume data processing applications that need to handle millions of writes per second
* Example: Log aggregation systems, batch processing pipelines

**Low Latency (minimizing end-to-end message delivery time):**

* Best for: Real-time applications requiring immediate data delivery
* Examples: Chat applications, interactive websites, IoT device monitoring

**High Durability (ensuring no data loss):**

* Best for: Systems where data integrity is critical
* Examples: Financial transactions, audit logging, event sourcing systems

**High Availability (maximizing uptime):**

* Best for: Mission-critical applications that cannot tolerate downtime
* Examples: Payment processing systems, user authentication services


# Optimize Kafka Client for Durability
Source: https://docs.streamnative.io/cloud/build/kafka-clients/optimize-and-tune/optimize-kafka-durability



Durability refers to ensuring messages are not lost during transmission and storage. StreamNative Cloud provides durability through its storage layer in two ways:

* For **Classic Engine** clusters, messages are replicated across multiple storage nodes to protect against data loss
* For **Ursa Engine** clusters, messages are persisted to object storage

The configuration parameters discussed in this guide have varying ranges of values. The optimal settings depend on your chosen [data streaming engine](/cloud/overview/data-streaming-engine), specific requirements, and environmental factors such as average message size, number of partitions, and other system characteristics. Therefore, [benchmarking](/cloud/build/kafka-clients/optimize-and-tune/optimize-kafka-clients#benchmarking) is essential to validate and fine-tune the configuration for your particular application and environment.

## Producer Acknowledgments

Producers can control the durability of messages written to Kafka through the `acks` configuration parameter. Although you can use the `acks` parameter for throughput and latency optimization, it is primarily used to ensure message durability.

To optimize for high durability, set `acks=all` (equivalent to `acks=-1`). With this setting:

* For **Classic Engine** clusters, the broker waits for acknowledgment from all storage nodes before responding to the producer
* For **Ursa Engine** clusters, the broker waits for acknowledgment from object storage before responding to the producer

This provides the strongest available guarantees that messages won't be lost. The trade-off is higher latency since the broker must wait for all acknowledgments before responding to the producer.

## Duplication and Ordering

Producers can increase durability by retrying failed message sends to prevent data loss. The producer automatically retries sending messages up to the number specified by the `retries` parameter (default `MAX_INT`) and up to the time duration specified by `delivery.timeout.ms` (default `120000ms`). You can tune `delivery.timeout.ms` to set an upper bound on the total time between sending a message and receiving an acknowledgment from the broker, which should align with your business requirements for message validity.

There are two key considerations with automatic producer retries:

1. **Duplication**: Transient failures in StreamNative Cloud may cause the producer to send duplicate messages when retrying.
2. **Ordering**: Multiple sends may be "in flight" simultaneously, and a retry of a failed message may occur after a newer message has succeeded.

To address both concerns, configure the producer for idempotency by setting `enable.idempotence=true`. With idempotency enabled, brokers track messages using incrementing sequence numbers (similar to **TCP**). This prevents message duplication because brokers ignore duplicate sequence numbers, and preserves message ordering because on failures, the producer temporarily constrains to a single in-flight message until sequencing is restored. If idempotency guarantees cannot be satisfied, the producer raises a fatal error and rejects further sends. Applications should catch and handle these fatal errors appropriately.

If you don't configure producer idempotency but require these guarantees, you must handle potential duplication and ordering issues differently:

* For duplication: Build consumer application logic to handle duplicate messages
* For ordering: Either:
  * Set `max.in.flight.requests.per.connection=1` to allow only one request at a time
  * Set `retries=0` to preserve order while allowing pipelining (accepting potential message loss)

Instead of automatic retries, you can handle retries manually by coding exception handlers in the producer client (for example, using the `onCompletion()` method in the Java client's Callback interface). For manual retry handling, disable automatic retries with `retries=0`. Note that producer idempotency only works with automatic retries enabled - manual retries generate new sequence numbers that bypass duplication detection. While disabling automatic retries may create message gaps from individual failures, the broker still preserves the order of received writes.

## Consumer Offsets and Auto Commit

When optimizing for durability, you need to carefully consider how consumer offsets are managed, especially in the case of unexpected consumer failures. Consumer offsets track which messages have been consumed, making the timing and method of offset commits crucial for durability. A key scenario to avoid is when a consumer commits an offset for a message, begins processing that message, but then fails unexpectedly. In this case, when a new consumer takes over the partition, it won't reprocess any messages with offsets that were already committed, potentially leading to data loss.

By default, consumer offsets are automatically committed during the consumer's `poll()` call at regular intervals. While this default behavior works well for many use cases, you may need stronger guarantees if your consumer is part of a transactional chain. In such cases, you might want to ensure offsets are only committed after messages are fully processed.

You can control whether offset commits happen automatically or manually using the `enable.auto.commit` parameter:

* With `enable.auto.commit=true` (default), offsets are committed automatically during polling
* With `enable.auto.commit=false`, you must explicitly commit offsets in your consumer code using either:
  * `commitSync()` for synchronous commits
  * `commitAsync()` for asynchronous commits

For maximum durability, consider disabling automatic commits and explicitly managing offset commits in your application code after successful message processing.

## Exactly Once Semantics (EOS)

<Note title="Note">
  EOS transactions are supported for **Classic Engine** clusters only.
</Note>

For the strongest message delivery guarantees, you can configure your applications to use **Exactly Once Semantics (EOS)** transactions. EOS transactions enable atomic writes across multiple Kafka topics and partitions.

Since messages in the log may be in various states of a transaction, consumers can control which messages they receive using the `isolation.level` configuration parameter:

* Setting `isolation.level=read_committed` ensures consumers only receive:
  * Non-transactional messages
  * Committed transactional messages
  * No messages from open or aborted transactions

To implement transactional semantics in a consume-process-produce pattern and ensure exactly-once processing:

1. Set `enable.auto.commit=false` on the consumer
2. Manually commit offsets using the `sendOffsetsToTransaction()` method in the `KafkaProducer` interface
3. For event streaming applications, configure the `processing.guarantee` parameter for exactly-once processing

## Summary

Here's a summary of key configurations for optimizing durability:

### Producer Configurations

| Configuration                           | Recommended Value | Default Value                               | Description                               |
| --------------------------------------- | ----------------- | ------------------------------------------- | ----------------------------------------- |
| `replication.factor`                    | 3                 | -                                           | Number of replicas for each partition     |
| `acks`                                  | `all`             | `all`, default prior to Kafka 3.0: `1`      | Number of acknowledgments required        |
| `enable.idempotence`                    | `true`            | `true`, default prior to Kafka 3.0: `false` | Enable exactly-once delivery semantics    |
| `max.in.flight.requests.per.connection` | 1                 | 5                                           | Maximum number of unacknowledged requests |

### Consumer Configurations

| Configuration        | Recommended Value                                                       | Default Value                                                                        | Description                               |
| -------------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | ----------------------------------------- |
| `enable.auto.commit` | `false`                                                                 | `true`                                                                               | Enable automatic offset commits           |
| `isolation.level`    | `read_committed` (**Ursa Engine** doesn't support `read_committed` yet) | `read_uncommitted` for Java client and `read_committed` for librdkafka based clients | Transaction isolation level for consumers |


# Optimize Kafka Client for Latency
Source: https://docs.streamnative.io/cloud/build/kafka-clients/optimize-and-tune/optimize-kafka-latency



StreamNative Cloud supports two different [data streaming engines](/cloud/overview/data-streaming-engine): **Classic Engine** and **Ursa Engine**. The **Classic Engine** uses BookKeeper for storage, providing lower latency (less than 100ms, typically single-digit milliseconds) but at a higher cost. The **Ursa Engine** uses object storage, offering reduced costs but with slightly higher latency (sub-second, typically in the range of 200-500ms). Based on your latency requirements, you need to choose the appropriate engine.

<Note title="Note">
  The following latency optimization recommendations are applicable for **Classic Engine** clusters. However, they are not recommended for **Ursa Engine** clusters. Even with client-side latency optimizations, the overall latency for Ursa Engine will remain in the sub-second range due to broker-side batching.
</Note>

In addition to the data streaming engine, the remaining configurations in this guide provide general recommendations for optimizing latency.

## Data Format

StreamNative supports storing data in different formats to achieve varying levels of interoperability between protocols: `kafka` and `pulsar`. Each format has different performance characteristics:

* `kafka` format: The **default format** for StreamNative Cloud. It provides the best performance with Kafka clients. However, Pulsar consumers cannot consume this format unless a payload processor is employed.
* `pulsar` format: Provides the highest interoperability between protocols. However, it incurs a performance penalty as it requires transforming data from Kafka producers into the Pulsar format before storage.

If you want to achieve the lowest latency with Kafka clients and don't need Pulsar clients to read the data, consider configuring your cluster to store data in the `kafka` format.

## Batching Messages

Producers automatically batch messages by collecting multiple messages to send together. To minimize latency when producing data to StreamNative Cloud, you can reduce the time spent waiting for batches to fill. By default, the producer is optimized for low latency with the `linger.ms` parameter set to 0, meaning the producer sends data as soon as it's available. While batching is always enabled—messages are always sent in batches—with `linger.ms=0`, a batch may contain only one message (unless messages arrive faster than the producer can send them).

## Compression

Consider whether you need to enable compression. Enabling compression requires additional CPU cycles but reduces network bandwidth usage. Disabling compression (setting `compression.type=none`) spares CPU cycles but increases network bandwidth usage. While a good compression codec may potentially reduce latency by decreasing network transfer time, the CPU overhead of compression could offset those gains. Evaluate your specific use case - if CPU is your bottleneck, consider disabling compression; if network bandwidth is constrained, compression may help reduce overall latency.

## Producer Acknowledgements

You can tune the number of acknowledgments the producer requires from the designated broker in StreamNative Cloud before considering a request complete.

<Note title="Note">
  This producer acknowledgment is separate from when a message is considered durably committed to storage.
</Note>

The sooner the designated broker responds, the sooner the producer can send the next batch of messages, reducing producer latency. You can configure this using the `acks` parameter:

* `acks=0`: Producer doesn't wait for any acknowledgment, providing lowest latency but no durability guarantees
* `acks=1`: Producer waits for acknowledgment from the designated broker after receiving at least one acknowledgment from storage
* `acks=all`: Producer waits for acknowledgment from the designated broker after receiving all acknowledgments from storage

By default, `acks=all` provides the strongest durability guarantees but higher latency. For latency-sensitive applications that can tolerate potential message loss, you can set `acks=0`, but be aware that messages may be lost silently if broker failures occur.

## Consumer Fetching

Similar to producer batching, you can tune consumers for lower latency by adjusting how much data a consumer gets from each fetch from the designated broker in StreamNative Cloud. The consumer configuration parameter `fetch.min.bytes` defaults to `1`, which means fetch requests are answered as soon as a single byte of data is available or the fetch request times out (controlled by `fetch.max.wait.ms`). These two parameters work together to control both the size of fetch requests (`fetch.min.bytes`) and how long to wait for data (`fetch.max.wait.ms`).

For lowest latency, keep `fetch.min.bytes` at its default of `1` and reduce `fetch.max.wait.ms` from its default of `500ms`. This ensures consumers receive data as soon as it's available, though at the cost of potentially more frequent fetch requests.

## Summary

Here's a summary of key configurations for optimizing latency:

### Producer Configurations

| Configuration      | Recommended Value | Default Value | Description                        |
| ------------------ | ----------------- | ------------- | ---------------------------------- |
| `linger.ms`        | 0                 | 0             | Time to wait for batches to fill   |
| `compression.type` | `none`            | `none`        | Compression codec to use           |
| `acks`             | `1`               | `all`         | Number of acknowledgments required |

### Consumer Configurations

| Configuration     | Recommended Value | Default Value | Description                           |
| ----------------- | ----------------- | ------------- | ------------------------------------- |
| `fetch.min.bytes` | 1                 | 1             | Minimum data size for fetch responses |


# Optimize Kafka Client for Throughput
Source: https://docs.streamnative.io/cloud/build/kafka-clients/optimize-and-tune/optimize-kafka-throughput



To optimize for throughput, producers and consumers need to move as much data as possible within a given time period. This means maximizing the data transfer rate to achieve the highest possible throughput.

The configuration parameters discussed in this guide have varying ranges of values. The optimal settings depend on your chosen [data streaming engine](/cloud/overview/data-streaming-engine), specific requirements, and environmental factors such as *average message size*, *number of partitions*, and other system characteristics. Therefore, [benchmarking](/cloud/build/kafka-clients/optimize-and-tune/optimize-kafka-clients#benchmarking) is essential to validate and fine-tune the configuration for your particular application and environment.

## Data Streaming Engine

Both the **Classic Engine** and **Ursa Engine** support high-throughput data streaming with different cost and latency trade-offs. The Classic Engine uses BookKeeper for storage, providing lower latency but at a higher cost. The Ursa Engine uses object storage, offering reduced costs but with slightly higher latency. Choose the engine that best aligns with your specific requirements - Classic Engine for latency-sensitive workloads, or Ursa Engine for more cost-effective solutions.

## Data Format

StreamNative supports storing data in different formats to achieve varying levels of interoperability between protocols: `kafka` and `pulsar`. Each format has different performance characteristics:

* `kafka` format: Provides the best performance with Kafka clients. However, Pulsar consumers cannot consume this format unless a payload processor is employed.
* `pulsar` format: Provides the highest interoperability between protocols. However, it incurs a performance penalty as it requires transforming data from Kafka producers into the Pulsar format before storage.

If you want to achieve the highest throughput with Kafka clients and don't need Pulsar clients to read the data, consider configuring your cluster to store data in the `kafka` format.

## Number of Partitions

A topic partition is the unit of parallelism in Kafka. Producers can send messages to different partitions in parallel, brokers can process different partitions in parallel, and consumers can read from different partitions in parallel. In general, a higher number of topic partitions results in higher throughput. To maximize throughput, you need enough partitions to effectively distribute the workload across all brokers in your StreamNative Cloud cluster.

However, there are trade-offs to increasing the number of partitions. When choosing the partition count, consider both producer and consumer throughput requirements, and validate performance through benchmarking in your environment. Additionally, carefully design your data patterns and key assignments to ensure messages are distributed evenly across topic partitions. This prevents hotspots where certain partitions become overloaded while others remain underutilized.

## Batching Messages

Kafka producers can batch messages going to the same partition by collecting multiple messages to send together in a single request. One of the most important steps to optimize throughput is tuning producer batching to increase both the batch size and the time spent waiting for batches to fill with messages. Larger batch sizes result in fewer requests to the broker, which reduces load on producers and decreases broker CPU overhead for processing requests.

With the Java client, you can configure the `batch.size` parameter to increase the maximum size in bytes of each message batch. To give more time for batches to fill, you can configure the `linger.ms` parameter to have the producer wait longer before sending. This delay allows the producer to wait for the batch to reach the configured `batch.size`. The trade-off is higher latency since messages aren't sent immediately when they're ready.

For **Ursa Engine** clusters, it is recommended to use large batch sizes and higher `linger.ms` values to achieve better throughput.

## Compression

To optimize for throughput, you can enable compression on the producer to reduce the number of bytes transmitted over the network. Enable compression by configuring the `compression.type` parameter to one of the following standard compression codecs:

* `lz4` (recommended for performance)
* `snappy`
* `zstd`
* `gzip`
* `none` (default, meaning no compression)

Use `lz4` for optimal performance instead of `gzip`, which is more compute-intensive and may impact application performance. Compression is applied on full batches of data, so better batching results in better compression ratios.

## Producer Acknowledgments

When a producer sends a message to StreamNative Cloud, the message is routed to a designated broker based on the underlying data streaming engine for the target partition. By default, the producer waits for an acknowledgment from the broker before sending subsequent messages. However, if `acks=0` is configured, the producer sends messages without waiting for acknowledgment.

The `acks` configuration parameter controls how many acknowledgments the designated broker must receive before responding to the producer:

* `acks=0`: Producer sends messages without waiting for any acknowledgment
* `acks=1`: Designated broker acknowledges after receiving at least one acknowledgment from the underlying storage. In StreamNative Cloud, due to its storage architecture differing from Apache Kafka, this setting behaves the same as `acks=all`
* `acks=all`: Designated broker waits for all acknowledgments from the underlying storage

**Notes for Classic Engine clusters**

In StreamNative Cloud, message durability is not determined by the `acks` setting (`acks=1` or `acks=all`). Instead, durability is controlled by the underlying storage settings and namespace policies that define write quorum and ack quorum sizes. From a Kafka protocol perspective, both `acks=1` and `acks=all` behave identically in StreamNative Cloud - they wait for acknowledgments based on the cluster and namespace configurations.

To adjust throughput, modify the write quorum size and acknowledgment quorum size in either the namespace policies or cluster configuration. Cluster-level settings will apply globally to all namespaces.

## Memory Allocation

Kafka producers automatically allocate memory for the Java client to store unsent messages. If that memory limit is reached, the producer blocks additional sends until memory frees up or until `max.block.ms` time passes. You can adjust how much memory is allocated with the `buffer.memory` configuration parameter.

If you don't have many partitions, you may not need to adjust this parameter at all. However, if you have many partitions, you can tune `buffer.memory`—while taking into account the message size, linger time, and partition count—to maintain pipelines across more partitions. This enables better use of bandwidth across more brokers.

## Consumer Fetching

Another way to optimize for throughput is to adjust how much data consumers receive from each fetch from the designated broker in StreamNative Cloud. You can increase how much data the consumers get from the designated broker for each fetch request by increasing the configuration parameter `fetch.min.bytes`. This parameter sets the minimum number of bytes expected for a fetch response from a consumer.

Increasing `fetch.min.bytes` reduces the number of fetch requests made to StreamNative Cloud, reducing the broker CPU overhead to process each fetch, thereby improving throughput. Similar to increasing batching on the producer side, there may be a resulting trade-off with higher latency when increasing this parameter on the consumer. This is because the broker won't send the consumer new messages until either:

* The fetch request has enough messages to fulfill the size requirement (`fetch.min.bytes`)
* The wait time expires (`fetch.max.wait.ms`)

### Consumer Groups

Assuming your application allows it, use consumer groups with multiple consumers to parallelize consumption. Parallelizing consumption can improve throughput because multiple consumers can balance the load by processing multiple partitions simultaneously. The upper limit on this parallelization is the number of partitions in the topic.

## Summary

Here's a summary of key configurations for optimizing throughput:

### Producer Configurations

| Configuration      | Recommended Value           | Default Value | Description                               |
| ------------------ | --------------------------- | ------------- | ----------------------------------------- |
| `batch.size`       | 100000-200000               | 16384         | Maximum size in bytes for message batches |
| `linger.ms`        | 10-100                      | 0             | Time to wait for batches to fill          |
| `compression.type` | `lz4`                       | `none`        | Compression codec to use                  |
| `acks`             | `all`                       | `all`         | Number of acknowledgments required        |
| `buffer.memory`    | Increase if many partitions | 33554432      | Memory buffer size for unsent messages    |

### Consumer Configurations

| Configuration       | Recommended Value | Default Value | Description                             |
| ------------------- | ----------------- | ------------- | --------------------------------------- |
| `fetch.min.bytes`   | \~100000          | 1             | Minimum data size for fetch responses   |
| `fetch.max.wait.ms` | 500               | 500           | Maximum time to wait for fetch response |


# Connect to your cluster using the Kafka C client (rdkafka)
Source: https://docs.streamnative.io/cloud/build/kafka-clients/quick-starts/cloud-connect-kafka-c



<Note title="Note">
  This QuickStart assumes that you have created a StreamNative cluster with the Kafka protocol enabled, created a service account, and granted the service account `produce` and `consume` permissions to a namespace for the target topic.
</Note>

This document describes how to connect to your StreamNative cluster using the Kafka C client using [API Keys](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#kafka-clients) authentication.

## Before you begin

<Note title="Note">
  * Before using an API key, verify that the service account is authorized to access the resources, such as tenants, namespaces, and topics.
  * The password for different utilities as `kcat` will be equal to `token:<API KEY>`.
</Note>

You can follow the instructions to [create an API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#using-api-keys-to-connect-to-your-cluster) for the service account you choose to use.

## Steps

1. Install the Kafka C client. See [https://github.com/confluentinc/librdkafka?tab=readme-ov-file#installation](https://github.com/confluentinc/librdkafka?tab=readme-ov-file#installation)

2. Build a C application to produce and consume messages.

   ```c theme={null}
   // Note: for code simplicity, this example does not check the return value of
   // the rd_kafka_* APIs.
   #include <librdkafka/rdkafka.h>
   #include <stdatomic.h>
   #include <string.h>

   static void dr_msg_cb(rd_kafka_t *rk, const rd_kafka_message_t *msg, void *opaque) {
     if (msg->err == RD_KAFKA_RESP_ERR_NO_ERROR) {
       printf("Sent message %s to %lu\n", (char *)msg->payload, (unsigned long)msg->offset);
       atomic_fetch_add((atomic_int *)opaque, 1);
     } else {
       fprintf(stderr, "Received error: %s\n", rd_kafka_err2str(msg->err));
     }
   }

   int main(int argc, char *argv[]) {
     const char *bootstrap_servers = "<bootstrap-servers>";
     const char *topic = "my-topic";
     const char *username = "public";  // the tenant name
     const char *token = "token:<api-key>";
     const char *group = "my-group";
     char errstr[1024];

     rd_kafka_conf_t *consumer_conf = rd_kafka_conf_new();
     rd_kafka_conf_set(consumer_conf, "bootstrap.servers", bootstrap_servers, errstr, sizeof(errstr));
     rd_kafka_conf_set(consumer_conf, "group.id", group, errstr, sizeof(errstr));
     rd_kafka_conf_set(consumer_conf, "auto.offset.reset", "earliest", errstr, sizeof(errstr));
     rd_kafka_conf_set(consumer_conf, "sasl.mechanisms", "PLAIN", errstr, sizeof(errstr));
     rd_kafka_conf_set(consumer_conf, "security.protocol", "SASL_SSL", errstr, sizeof(errstr));
     rd_kafka_conf_set(consumer_conf, "sasl.username", username, errstr, sizeof(errstr));
     rd_kafka_conf_set(consumer_conf, "sasl.password", token, errstr, sizeof(errstr));
     // Note: Ursa does not support read_committed for now
     rd_kafka_conf_set(consumer_conf, "isolation.level", "read_uncommitted", errstr, sizeof(errstr));

     rd_kafka_t *consumer = rd_kafka_new(RD_KAFKA_CONSUMER, consumer_conf, errstr, sizeof(errstr));
     if (!consumer) {
       fprintf(stderr, "Failed to create consumer: %s\n", errstr);
       return 1;
     }

     rd_kafka_topic_partition_list_t *topics = rd_kafka_topic_partition_list_new(1);
     rd_kafka_topic_partition_list_add(topics, topic, RD_KAFKA_PARTITION_UA);
     rd_kafka_subscribe(consumer, topics);
     rd_kafka_topic_partition_list_destroy(topics);

     rd_kafka_conf_t *producer_conf = rd_kafka_conf_new();
     rd_kafka_conf_set(producer_conf, "bootstrap.servers", bootstrap_servers, errstr, sizeof(errstr));
     rd_kafka_conf_set(producer_conf, "sasl.mechanisms", "PLAIN", errstr, sizeof(errstr));
     rd_kafka_conf_set(producer_conf, "security.protocol", "SASL_SSL", errstr, sizeof(errstr));
     rd_kafka_conf_set(producer_conf, "sasl.username", username, errstr, sizeof(errstr));
     rd_kafka_conf_set(producer_conf, "sasl.password", token, errstr, sizeof(errstr));

     rd_kafka_conf_set_dr_msg_cb(producer_conf, dr_msg_cb);
     atomic_int count = ATOMIC_VAR_INIT(0);
     rd_kafka_conf_set_opaque(producer_conf, &count);

     rd_kafka_t *producer = rd_kafka_new(RD_KAFKA_PRODUCER, producer_conf, errstr, sizeof(errstr));
     if (!producer) {
       fprintf(stderr, "Failed to create producer: %s\n", errstr);
       return 1;
     }

     const int num_messages = 10;
     for (int i = 0; i < 10; i++) {
       char value[128];
       snprintf(value, sizeof(value), "msg-%d", i);
       rd_kafka_producev(producer, RD_KAFKA_V_TOPIC(topic), RD_KAFKA_V_MSGFLAGS(RD_KAFKA_MSG_F_COPY),
                         RD_KAFKA_V_VALUE(&value[0], strlen(value)), RD_KAFKA_V_OPAQUE(NULL),
                         RD_KAFKA_V_END);
     }
     // Poll until the message is sent
     int num_produce_done;
     while ((num_produce_done = atomic_load(&count)) < num_messages) {
       rd_kafka_poll(producer, 1);
     }
     printf("%d messages are loaded\n", num_messages);

     for (int i = 0; i < num_messages;) {
       rd_kafka_message_t *msg = rd_kafka_consumer_poll(consumer, 10);
       if (!msg) continue;

       printf("Received msg %s from %s-%d@%lu\n", (char *)msg->payload, topic, msg->partition,
              (unsigned long)msg->offset);
       i++;
       rd_kafka_message_destroy(msg);
     }

     rd_kafka_destroy(producer);
     rd_kafka_consumer_close(consumer);
     rd_kafka_destroy(consumer);
     return 0;
   }
   ```

   * `<bootstrap-servers>`: the Kafka service URL of your StreamNative cluster.
   * `<api-key>`: an API key of your service account.

   The code example uses C11 standard, so you need to compile it like:

   ```bash theme={null}
   gcc main.c -std=c11 -lrdkafka
   ```

3. Run the Go application and you should see the following output:

   ```bash theme={null}
   Sent message msg-0 to 0
   Sent message msg-1 to 1
   Sent message msg-2 to 2
   Sent message msg-3 to 3
   Sent message msg-4 to 4
   Sent message msg-5 to 5
   Sent message msg-6 to 6
   Sent message msg-7 to 7
   Sent message msg-8 to 8
   Sent message msg-9 to 9
   10 messages are loaded
   Received msg msg-0 from my-topic-0@0
   Received msg msg-1 from my-topic-0@1
   Received msg msg-2 from my-topic-0@2
   Received msg msg-3 from my-topic-0@3
   Received msg msg-4 from my-topic-0@4
   Received msg msg-5 from my-topic-0@5
   Received msg msg-6 from my-topic-0@6
   Received msg msg-7 from my-topic-0@7
   Received msg msg-8 from my-topic-0@8
   Received msg msg-9 from my-topic-0@9
   ```


# Connect to your cluster using Kafka CLI
Source: https://docs.streamnative.io/cloud/build/kafka-clients/quick-starts/cloud-connect-kafka-cli



This document describes how to connect to your StreamNative cluster using the Kafka CLI tool (v3.1.0) using either [OAuth2](#use-oauth2) or [API Keys](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#kafka-clients) authentication.

<Note title="Note">
  This QuickStart assumes that you have created a StreamNative cluster with the Kafka protocol enabled, created a service account, and granted the service account `produce` and `consume` permissions to a namespace for the target topic.
</Note>

## Connect to your cluster using API keys

<span />

This section describes how to connect to your StreamNative cluster using the Kafka Java client with [SASL/PLAIN authentication](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#kafka-clients).

### Before you begin

<Note title="Note">
  * Before using an API key, verify that the service account is authorized to access the resources, such as tenants, namespaces, and topics.
  * The password for different utilities as `kcat` will be equal to `token:<API KEY>`.
</Note>

You can follow the instructions to [create an API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#using-api-keys-to-connect-to-your-cluster) for the service account you choose to use.

### Steps

1. Download Kafka 3.1.0 release and extract it to the `~/kafka` folder.

   ```bash theme={null}
   mkdir -p ~/kafka && cd ~/kafka
   # download Kafka 3.1.0
   curl -O https://archive.apache.org/dist/kafka/3.1.0/kafka_2.13-3.1.0.tgz
   tar xzf ./kafka_2.13-3.1.0.tgz
   ```

2. Download the supplementary libraries for the Kafka client.

   ```bash theme={null}
   cd ~/kafka/kafka_2.13-3.1.0
   # download supplementary libraries
   curl -O https://repo1.maven.org/maven2/io/streamnative/pulsar/handlers/oauth-client/3.1.0.1/oauth-client-3.1.0.1.jar --output-dir ./libs
   ```

3. Create a token configuration file.

   This example creates a file named `kafka-token.properties`, substituting `API-KEY` with an API key of your service account.

   Remind that the password is: `token:API-KEY`

   ```
   # configure kafka-token.properties file.
   echo 'security.protocol=SASL_SSL
   sasl.mechanism=PLAIN
   sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="public/default" password="token:YOUR-TOKEN";' > ~/kafka-token.properties
   ```

4. Connect to the cluster through the SASL/PLAIN authentication method.

   a. Open a terminal and run a Kafka consumer to receive a message from the `kop-test-topic` topic.

   ```bash theme={null}
   # run consumer
   ~/kafka/kafka_2.13-3.1.0/bin/kafka-console-consumer.sh \
       --bootstrap-server "your-pulsar-service-url" \
       --consumer.config ~/kafka/kafka-token.properties \
       --topic kop-test-topic
   ```

   * `bootstrap-server`: the Kafka service URL of your StreamNative cluster.

   b. Open another terminal and run a Kafka producer to send a message to the `test-topic` topic.

   ```bash theme={null}
   # run producer
   ~/kafka/kafka_2.13-3.1.0/bin/kafka-console-producer.sh \
       --bootstrap-server "your-pulsar-service-url" \
       --producer.config ~/kafka/kafka-token.properties \
       --topic kop-test-topic
   ```

   You can type some messages, for example `Hello, Kafka on Pulsar!` and then press the **Enter** key to produce the message to the `kop-test-topic` topic. Then, you should see this message on the consumer terminal.

## Connect to your cluster using OAuth2 authentication

<span />

This section describes how to connect to your StreamNative cluster using the Kafka CLI tool through OAuth2 authentication.

### Before you begin

* Get the OAuth2 credential file.

  1. On the left navigation pane, click **Service Accounts**.
  2. In the row of the service account you want to use, in the **Key File** column, click the **Download** icon to download the OAuth2 credential file to your local directory.

* Get the service URL of your StreamNative cluster.

  1. On the left navigation pane, in the **Admin** area, click **Pulsar Clusters**.
  2. Select the **Details** tab, and in the **Access Points** area, click **Copy** at the end of the row of the **Kafka Service URL (TCP)**.

### Steps

1. Download Kafka 3.1.0 release and extract it to the `~/kafka` folder.

   ```bash theme={null}
   mkdir -p ~/kafka && cd ~/kafka
   # download Kafka 3.1.0
   curl -O https://archive.apache.org/dist/kafka/3.1.0/kafka_2.13-3.1.0.tgz
   tar xzf ./kafka_2.13-3.1.0.tgz
   ```

2. Download the supplementary libraries for the Kafka client.

   ```bash theme={null}
   cd ~/kafka/kafka_2.13-3.1.0
   # download supplementary libraries
   curl -O https://repo1.maven.org/maven2/io/streamnative/pulsar/handlers/oauth-client/2.9.1.5/oauth-client-2.9.1.5.jar --output-dir ./libs
   curl -O https://repo1.maven.org/maven2/org/apache/pulsar/pulsar-client-admin-api/2.9.2/pulsar-client-admin-api-2.9.2.jar --output-dir ./libs
   curl -O https://repo1.maven.org/maven2/org/apache/pulsar/pulsar-client/2.9.2/pulsar-client-2.9.2.jar --output-dir ./libs
   curl -O https://repo1.maven.org/maven2/org/apache/pulsar/pulsar-client-api/2.9.2/pulsar-client-api-2.9.2.jar --output-dir ./libs
   ```

3. Create an OAuth2 configuration file.

   This example creates a file named `kafka.properties`, substituting the path to your downloaded OAuth2 credential file and the audience respectively.

   ```
   # configure kafka.properties file.
   echo 'sasl.login.callback.handler.class=io.streamnative.pulsar.handlers.kop.security.oauth.OauthLoginCallbackHandler
   security.protocol=SASL_SSL
   sasl.mechanism=OAUTHBEARER
   sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule \
     required oauth.issuer.url="https://auth.streamnative.cloud/"\
     oauth.credentials.url="file:///YOUR-KEY-FILE-PATH"\
     oauth.audience="YOUR-AUDIENCE-STRING";' > ~/kafka/kafka.properties
   ```

   * `oauth.issuer.url`: the OAuth2 authentication provider. You can get the value from your downloaded OAuth2 credential file.
   * `oauth.credentials.url`: the path to your downloaded OAuth2 credential file.
   * `oauth.audience`: the `audience` parameter is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.

4. Connect to the cluster through the OAuth2 authentication method.

   a. Open a terminal and run a Kafka consumer to receive a message from the `test-topic` topic.

   ```bash theme={null}
   # run consumer
   ~/kafka/kafka_2.13-3.1.0/bin/kafka-console-consumer.sh \
       --bootstrap-server "your-pulsar-service-url" \
       --consumer.config ~/kafka/kafka.properties \
       --topic test-topic
   ```

   * `bootstrap-server`: the Kafka service URL of your StreamNative cluster.

   b. Open another terminal and run a Kafka producer to send a message to the `test-topic` topic.

   ```bash theme={null}
   # run producer
   ~/kafka/kafka_2.13-3.1.0/bin/kafka-console-producer.sh \
       --bootstrap-server "your-pulsar-service-url" \
       --producer.config ~/kafka/kafka.properties \
       --topic test-topic
   ```

   You can type some messages, for example `Hello, Kafka on Pulsar!` and then press the **Enter** key to produce the message to the `test-topic` topic. Then, you should see this message on the consumer terminal.


# Connect to your cluster using the Kafka Go client
Source: https://docs.streamnative.io/cloud/build/kafka-clients/quick-starts/cloud-connect-kafka-go



<Note title="Note">
  This QuickStart assumes that you have created a StreamNative cluster with the Kafka protocol enabled, created a service account, and granted the service account `produce` and `consume` permissions to a namespace for the target topic.
</Note>

This document describes how to connect to your StreamNative cluster using the Kafka Go client using [API Keys](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#kafka-clients) authentication.

## Before you begin

<Note title="Note">
  * Before using an API key, verify that the service account is authorized to access the resources, such as tenants, namespaces, and topics.
  * The password for different utilities as `kcat` will be equal to `token:<API KEY>`.
</Note>

You can follow the instructions to [create an API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#using-api-keys-to-connect-to-your-cluster) for the service account you choose to use.

## Steps

1. Install the Kafka Go client.

   ```bash theme={null}
   go get -u github.com/confluentinc/confluent-kafka-go/v2/kafka
   ```

2. Build a Go application to produce and consume messages.

   ```go theme={null}
   package main

   import (
     "fmt"
     "github.com/confluentinc/confluent-kafka-go/v2/kafka"
     "time"
   )

   func main() {
     // Step 1: replace with your configurations
     serverUrl := "SERVER-URL"
     jwtToken := "API-KEY"
     topicName := "test-go-topic"
     namespace := "public/default"
     password := "token:" + jwtToken

     // Step 2: create a producer to send messages
     producer, err := kafka.NewProducer(&kafka.ConfigMap{
       "bootstrap.servers": serverUrl,
       "security.protocol": "SASL_SSL",
       "sasl.mechanism":    "PLAIN",
       "sasl.username":     namespace,
       "sasl.password":     password,
     })
     if err != nil {
       panic(err)
     }
     defer producer.Close()

     err = producer.Produce(&kafka.Message{
       TopicPartition: kafka.TopicPartition{Topic: &topicName, Partition: kafka.PartitionAny},
       Value:          []byte("hello world"),
     }, nil)

     if err != nil {
       panic(err)
     }
     producer.Flush(1000)

     // wait for delivery report
     e := <-producer.Events()

     message := e.(*kafka.Message)
     if message.TopicPartition.Error != nil {
       fmt.Printf("failed to deliver message: %v\n",
         message.TopicPartition)
     } else {
       fmt.Printf("delivered to topic %s [%d] at offset %v\n",
         *message.TopicPartition.Topic,
         message.TopicPartition.Partition,
         message.TopicPartition.Offset)
     }

     // Step 3: create a consumer to read messages
     consumer, err := kafka.NewConsumer(&kafka.ConfigMap{
       "bootstrap.servers":  serverUrl,
       "security.protocol":  "SASL_SSL",
       "sasl.mechanisms":    "PLAIN",
       "sasl.username":      namespace,
       "sasl.password":      password,
       "session.timeout.ms": 6000,
       "group.id":           "my-group",
       "auto.offset.reset":  "earliest",
       "isolation.level":    "read_uncommitted", // Note: Ursa does not support read_committed for now
     })

     if err != nil {
       panic(fmt.Sprintf("Failed to create consumer: %s", err))
     }
     defer consumer.Close()

     topics := []string{topicName}
     err = consumer.SubscribeTopics(topics, nil)
     if err != nil {
       panic(fmt.Sprintf("Failed to subscribe topics: %s", err))
     }

     // read one message then exit
     for {
       fmt.Println("polling...")
       message, err = consumer.ReadMessage(1 * time.Second)
       if err == nil {
         fmt.Printf("consumed from topic %s [%d] at offset %v: %+v",
           *message.TopicPartition.Topic,
           message.TopicPartition.Partition, message.TopicPartition.Offset, string(message.Value))
         break
       }
     }
   }
   ```

   * `SERVER-URL`: the Kafka service URL of your StreamNative cluster.
   * `API-KEY`: an API key of your service account.

3. Run the Go application and you should see the following output:

   ```bash theme={null}
   delivered to topic test-go-topic [0] at offset 29
   polling...
   polling...
   polling...
   polling...
   polling...
   polling...
   polling...
   consumed from topic test-go-topic [0] at offset 15: hello world
   ```


# Connect to your cluster using the Kafka Java client
Source: https://docs.streamnative.io/cloud/build/kafka-clients/quick-starts/cloud-connect-kafka-java



<Note title="Note">
  This QuickStart assumes that you have created a StreamNative cluster with the Kafka protocol enabled, created a service account, and granted the service account `produce` and `consume` permissions to a namespace for the target topic.
</Note>

This document describes how to connect to a StreamNative cluster through a Kafka Java client, and use the Java producer and consumer to produce and consume messages to and from a topic. The Java client supports connecting to a StreamNative cluster using either [OAuth2](#use-oauth2) or [API Keys](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#kafka-clients) authentication.

## Prerequisites

* Install [Java 17](https://www.oracle.com/java/technologies/downloads/#java17). For details, see [overview of JDK installation](https://docs.oracle.com/en/java/javase/17/install/overview-jdk-installation.html).

* Install Maven. For details, see [installing Apache Maven](https://maven.apache.org/install.html).

## Connect to your cluster using API keys

<span />

This section describes how to connect to your StreamNative cluster using the Kafka Java client with [SASL/PLAIN authentication](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#kafka-clients).

### Before you begin

<Note title="Note">
  * Before using an API key, verify that the service account is authorized to access the resources, such as tenants, namespaces, and topics.
  * The password for different utilities as `kcat` will be equal to `token:<API KEY>`.
</Note>

You can follow the instructions to [create an API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#using-api-keys-to-connect-to-your-cluster) for the service account you choose to use.

### Steps

1. Add Maven dependencies.

   ```xml theme={null}
   <dependency>
       <groupId>org.apache.kafka</groupId>
       <artifactId>kafka-clients</artifactId>
       <version>3.6.1</version>
   </dependency>
   ```

2. Open a terminal and run a Kafka consumer to receive a message from the `test-kafka-topic` topic.

   ```java theme={null}
   package org.example;

   import java.io.IOException;
   import java.time.Duration;
   import java.util.Collections;
   import java.util.Properties;
   import java.util.concurrent.ExecutionException;
   import java.util.concurrent.Future;
   import org.apache.kafka.clients.CommonClientConfigs;
   import org.apache.kafka.clients.consumer.ConsumerConfig;
   import org.apache.kafka.clients.consumer.ConsumerRecords;
   import org.apache.kafka.clients.consumer.KafkaConsumer;
   import org.apache.kafka.clients.producer.KafkaProducer;
   import org.apache.kafka.clients.producer.ProducerConfig;
   import org.apache.kafka.clients.producer.ProducerRecord;
   import org.apache.kafka.clients.producer.RecordMetadata;
   import org.apache.kafka.common.serialization.StringDeserializer;
   import org.apache.kafka.common.serialization.StringSerializer;

   /**
    * A JWT token authentication example of Kafka consumer to StreamNative Cloud
    */
   public class SNCloudJWTTokenConsumer {
       public static void main(String[] args) throws ExecutionException, InterruptedException, IOException {
           BasicConfigurator.configure();
           Logger.getRootLogger().setLevel(Level.INFO);

           // replace these configs for your cluster
           String serverUrl = "SERVER-URL";
           String jwtToken = "API-KEY";
           String token = "token:" + jwtToken;

           final String topicName = "test-kafka-topic";
           String namespace = "public/default";

           final Properties props = new Properties();
           props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, serverUrl);
           props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
           props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
           props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
           props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
           props.put(ConsumerConfig.GROUP_ID_CONFIG, "hello-world");
           props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
           props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_SSL");
           props.put("sasl.mechanism", "PLAIN");
           props.put("sasl.jaas.config", String.format(
                   "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"%s\" password=\"%s\";",
                   namespace, token));

           // Create a consumer
           final KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
           consumer.subscribe(Collections.singleton(topicName));

           // Consume some messages and quit immediately
           boolean running = true;
           while (running) {
               System.out.println("running");
               final ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1));
               if (!records.isEmpty()) {
                   records.forEach(record -> System.out.println("Receive record: " + record.value() + " from "
                           + record.topic() + "-" + record.partition() + "@" + record.offset()));
                   running = false;
               }
           }
           consumer.close();
       }
   }
   ```

   * `SERVER-URL`: the Kafka service URL of your StreamNative cluster.
   * `API-KEY`: an API key of your service account.

3. Open another terminal and run a Kafka producer to send a message to the `test-kafka-topic` topic.

   ```java theme={null}
   package org.example;

   import java.io.IOException;
   import java.util.Properties;
   import java.util.concurrent.ExecutionException;
   import java.util.concurrent.Future;
   import org.apache.kafka.clients.CommonClientConfigs;
   import org.apache.kafka.clients.producer.KafkaProducer;
   import org.apache.kafka.clients.producer.ProducerConfig;
   import org.apache.kafka.clients.producer.ProducerRecord;
   import org.apache.kafka.clients.producer.RecordMetadata;
   import org.apache.kafka.common.serialization.StringDeserializer;
   import org.apache.kafka.common.serialization.StringSerializer;

   /**
    * A JWT token authentication example of Kafka producer to StreamNative Cloud
   */
   public class SNCloudJWTTokenProducer {
       public static void main(String[] args) throws ExecutionException, InterruptedException, IOException {
           BasicConfigurator.configure();
           Logger.getRootLogger().setLevel(Level.INFO);

           // replace these configs for your cluster
           String serverUrl = "SERVER-URL";
           String jwtToken = "API-KEY";
           String token = "token:" + jwtToken;
           final String topicName = "test-kafka-topic";
           String namespace = "public/default";

           // 2. Create a producer with token authentication, which is equivalent to SASL/PLAIN mechanism in Kafka
           final Properties props = new Properties();
           props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, serverUrl);
           props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
           props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
           props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_SSL");
           props.put("sasl.mechanism", "PLAIN");
           props.put("sasl.jaas.config", String.format(
                   "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"%s\" password=\"%s\";",
                   namespace, token));

           // 2. Create a producer
           final KafkaProducer<String, String> producer = new KafkaProducer<>(props);

           // 3. Produce one message
           for (int i = 0; i < 5; i++) {
               String value = "hello world";
               final Future<RecordMetadata> recordMetadataFuture = producer.send(new ProducerRecord<>(topicName, value));
               final RecordMetadata recordMetadata = recordMetadataFuture.get();
               System.out.println("Send " + value + " to " + recordMetadata);
           }
           producer.close();
       }
   }
   ```

   * `SERVER-URL`: the Kafka service URL of your StreamNative cluster.
   * `API-KEY`: an API key of your service account.

## Connect to your cluster using OAuth2 authentication

<span />

This section describes how to connect to your StreamNative cluster using the Kafka Java client with OAuth2 authentication.

### Before you begin

* Get the OAuth2 credential file.

  1. On the left navigation pane, click **Service Accounts**.
  2. In the row of the service account you want to use, in the **Key File** column, click the **Download** icon to download the OAuth2 credential file to your local directory.

* Get the service URL of your StreamNative cluster.

  1. On the left navigation pane, in the **Admin** area, click **Pulsar Clusters**.
  2. Select the **Details** tab, and in the **Access Points** area, click **Copy** at the end of the row of the **Kafka Service URL (TCP)**.

### Steps

1. Add Maven dependencies.

   ```xml theme={null}
   <dependency>
       <groupId>org.apache.kafka</groupId>
       <artifactId>kafka-clients</artifactId>
       <version>3.4.0</version>
   </dependency>
   <dependency>
       <groupId>io.streamnative.pulsar.handlers</groupId>
       <artifactId>oauth-client</artifactId>
       <version>3.1.0.1</version>
   </dependency>
   ```

2. Open a terminal and run a Kafka consumer to receive a message from the `test-topic` topic.

   ```java theme={null}
   package org.example;

   import io.streamnative.pulsar.handlers.kop.security.oauth.OauthLoginCallbackHandler;
   import org.apache.kafka.clients.consumer.ConsumerConfig;
   import org.apache.kafka.clients.consumer.ConsumerRecords;
   import org.apache.kafka.clients.consumer.KafkaConsumer;
   import org.apache.kafka.common.serialization.StringDeserializer;
   import java.time.Duration;
   import java.util.Collections;
   import java.util.Properties;

   /**
   * An OAuth2 authentication example of Kafka consumer to StreamNative Cloud
   */
   public class SNCloudOAuth2Consumer {
       public static void main(String[] args) {
           // replace these configs with your cluster
           String serverUrl = "YOUR-KAFKA-SERVICE-URL";
           String keyPath = "YOUR-KEY-FILE-ABSOLUTE-PATH";
           String audience = "YOUR-AUDIENCE-STRING";

           // 1. Create properties of oauth2 authentication, which is equivalent to SASL/PLAIN mechanism in Kafka
           final Properties props = new Properties();
           props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, serverUrl);
           props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
           props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
           props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
           props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
           props.put(ConsumerConfig.GROUP_ID_CONFIG, "hello-world");
           props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
           props.setProperty("sasl.login.callback.handler.class", OauthLoginCallbackHandler.class.getName());
           props.setProperty("security.protocol", "SASL_SSL");
           props.setProperty("sasl.mechanism", "OAUTHBEARER");
           final String jaasTemplate = "org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required"
                   + " oauth.issuer.url=\"%s\""
                   + " oauth.credentials.url=\"%s\""
                   + " oauth.audience=\"%s\";";
           props.setProperty("sasl.jaas.config", String.format(jaasTemplate,
                   "https://auth.streamnative.cloud/",
                   "file://" + keyPath,
                   audience
           ));

           // 2. Create a consumer
           final KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
           final String topicName = "test-topic";
           consumer.subscribe(Collections.singleton(topicName));

           // 2. Consume some messages and quit immediately
           boolean running = true;
           while (running) {
               System.out.println("running");
               final ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1));
               if (!records.isEmpty()) {
                   records.forEach(record -> System.out.println("Receive record: " + record.value() + " from "
                           + record.topic() + "-" + record.partition() + "@" + record.offset()));
                   running = false;
               }
           }
           consumer.close();
       }
   }
   ```

   * `YOUR-KEY-FILE-ABSOLUTE-PATH`: the path to your downloaded OAuth2 credential file.
   * `YOUR-KAFKA-SERVICE-URL`: the Kafka service URL of your StreamNative cluster.
   * `YOUR-AUDIENCE-STRING`: the `audience` parameter is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.

3. Open another terminal and run a Kafka producer to send a message to the `test-topic` topic.

   ```java theme={null}
   package org.example;

   import io.streamnative.pulsar.handlers.kop.security.oauth.OauthLoginCallbackHandler;
   import org.apache.kafka.clients.producer.KafkaProducer;
   import org.apache.kafka.clients.producer.ProducerConfig;
   import org.apache.kafka.clients.producer.ProducerRecord;
   import org.apache.kafka.clients.producer.RecordMetadata;
   import org.apache.kafka.common.serialization.StringSerializer;
   import java.io.IOException;
   import java.util.Properties;
   import java.util.concurrent.ExecutionException;
   import java.util.concurrent.Future;

   /**
   * An OAuth2 authentication example of Kafka producer to StreamNative Cloud
   */
   public class SNCloudOAuth2Producer {
       public static void main(String[] args) throws ExecutionException, InterruptedException, IOException {
           // 1. Create a producer with oauth2 authentication, which is equivalent to SASL/PLAIN mechanism in Kafka
           final Properties props = new Properties();
           // replace these configs with your cluster
           String serverUrl = "YOUR-KAFKA-SERVICE-URL";
           String keyPath = "YOUR-KEY-FILE-ABSOLUTE-PATH";
           String audience = "YOUR-AUDIENCE-STRING";

           props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, serverUrl);
           props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
           props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
           props.setProperty("sasl.login.callback.handler.class", OauthLoginCallbackHandler.class.getName());
           props.setProperty("security.protocol", "SASL_SSL");
           props.setProperty("sasl.mechanism", "OAUTHBEARER");
           final String jaasTemplate = "org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required"
                   + " oauth.issuer.url=\"%s\""
                   + " oauth.credentials.url=\"%s\""
                   + " oauth.audience=\"%s\";";
           props.setProperty("sasl.jaas.config", String.format(jaasTemplate,
                   "https://auth.streamnative.cloud/",
                   "file://" + keyPath,
                   audience
           ));
           final KafkaProducer<String, String> producer = new KafkaProducer<>(props);

           // 2. Produce one message
           final String topicName = "test-topic";
           final Future<RecordMetadata> recordMetadataFuture = producer.send(new ProducerRecord<>(topicName, "hello"));
           final RecordMetadata recordMetadata = recordMetadataFuture.get();
           System.out.println("Send hello to " + recordMetadata);
           producer.close();
       }
   }
   ```

   * `YOUR-KEY-FILE-ABSOLUTE-PATH`: the path to your downloaded OAuth2 credential file.
   * `YOUR-KAFKA-SERVICE-URL`: the Kafka service URL of your StreamNative cluster.
   * `YOUR-AUDIENCE-STRING`: the `audience` parameter is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name.


# Connect to your cluster using the Kafka Node.js client
Source: https://docs.streamnative.io/cloud/build/kafka-clients/quick-starts/cloud-connect-kafka-nodejs



<Note title="Note">
  This QuickStart assumes that you have created a StreamNative cluster with the Kafka protocol enabled, created a service account, and granted the service account `produce` and `consume` permissions to a namespace for the target topic.
</Note>

This document describes how to connect to your StreamNative cluster using the Kafka Node.js client using [API Keys](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#kafka-clients) authentication.

## Before you begin

<Note title="Note">
  * Before using an API key, verify that the service account is authorized to access the resources, such as tenants, namespaces, and topics.
  * The password for different utilities as `kcat` will be equal to `token:<API KEY>`.
</Note>

You can follow the instructions to [create an API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#using-api-keys-to-connect-to-your-cluster) for the service account you choose to use.

## Steps

1. Install the Kafka Nodejs client.

   ```bash theme={null}
   npm install kafkajs
   ```

2. Build a Nodejs application to produce and consume messages.

   ```js theme={null}
   const { Kafka } = require('kafkajs')

   // Step 1: replace with your configurations
   let serverUrl = 'SERVER-URL'
   let jwtToken = 'API-KEY'
   let topicName = 'test-js-topic'
   let namespace = 'public/default'

   // Step 2: create the kafka client
   const kafka = new Kafka({
     clientId: 'my-app',
     brokers: [serverUrl],
     ssl: true,
     sasl: {
       mechanism: 'plain',
       username: namespace,
       password: 'token:' + jwtToken,
     },
   })

   // Step 3: send a message to your topic
   async function send() {
     const producer = kafka.producer()
     await producer.connect()

     let resp = await producer.send({
       topic: topicName,
       messages: [
         {
           value: 'Hello KafkaJS user!',
         },
       ],
     })

     console.log(`Send message:`, resp)

     await producer.disconnect()
   }

   // Step 4: read messages from the beginning of your topic
   async function receive() {
     const consumer = kafka.consumer({
       groupId: 'my-group',
       readUncommitted: true, // Note: Ursa does not support read_committed for now
     })
     await consumer.connect()
     console.log('Connected to Kafka')

     await consumer.subscribe({ topic: topicName, fromBeginning: true })

     await consumer.run({
       eachMessage: async ({ topic, partition, message }) => {
         console.log(`Received message:`, {
           value: message.value.toString(),
           headers: message.headers,
           topic: topic,
           partition: partition,
           offset: message.offset,
         })
       },
     })
   }

   // Step 5: send one message then receive
   send().then(function () {
     receive()
   })
   ```

   * `SERVER-URL`: the Kafka service URL of your StreamNative cluster.
   * `API-KEY`: an API key of your service account.

3. Run the Node.js application with your saved scripts.

   This example runs the Node.js application assuming that you save the scripts as `kop_test.js`.

   ```bash theme={null}
   node kop_test.js
   ```

   You should see the following output:

   ```bash theme={null}
   Send message: [
     {
       topicName: 'test-js-topic',
       partition: 0,
       errorCode: 0,
       baseOffset: '7',
       logAppendTime: '-1',
       logStartOffset: '-1'
     }
   ]
   Connected to Kafka
   Received message: {
     value: 'Hello KafkaJS user!',
     headers: {},
     topic: 'test-js-topic',
     partition: 0,
     offset: '0'
   }
   ...
   ```


# Connect to your cluster using the Kafka Python client
Source: https://docs.streamnative.io/cloud/build/kafka-clients/quick-starts/cloud-connect-kafka-python



<Note title="Note">
  This QuickStart assumes that you have created a StreamNative cluster with the Kafka protocol enabled, created a service account, and granted the service account `produce` and `consume` permissions to a namespace for the target topic.
</Note>

This document describes how to connect to your StreamNative cluster using the Kafka Python client using [API Keys](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#kafka-clients) authentication.

## Before you begin

<Note title="Note">
  * Before using an API key, verify that the service account is authorized to access the resources, such as tenants, namespaces, and topics.
  * The password for different utilities as `kcat` will be equal to `token:<API KEY>`.
</Note>

You can follow the instructions to [create an API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#using-api-keys-to-connect-to-your-cluster) for the service account you choose to use.

## Steps

1. Install the Kafka Python client.

   ```bash theme={null}
   pip install confluent-kafka
   ```

2. Build a Python application to produce and consume messages.

   ```python theme={null}
   from confluent_kafka import Producer, Consumer, KafkaError, KafkaException

   # Step 1: replace with your configurations
   serverUrl = "SERVER-URL"
   jwtToken = "YOUR-API-KEY"
   topicName = "test-py-topic"
   namespace = "public/default"
   password = "token:" + jwtToken

   def error_cb(err):
       print("Client error: {}".format(err))
       if err.code() == KafkaError._ALL_BROKERS_DOWN or \
               err.code() == KafkaError._AUTHENTICATION:
           raise KafkaException(err)


   # Step 2: create a producer to send messages
   p = Producer({
       'bootstrap.servers': serverUrl,
       'sasl.mechanism': 'PLAIN',
       'security.protocol': 'SASL_SSL',
       'sasl.username': namespace,
       'sasl.password': password,
   })


   def acked(err, msg):
       if err is not None:
           print('Failed to deliver message: {}'.format(err.str()))
       else:
           print('Produced to: {} [{}] @ {}'.format(msg.topic(), msg.partition(), msg.offset()))


   # send messages
   p.produce(topicName, value='hello python', callback=acked)
   p.flush(10)

   # Step 3: create a consumer to consume messages
   c = Consumer({
       'bootstrap.servers': serverUrl,
       'sasl.mechanism': 'PLAIN',
       'security.protocol': 'SASL_SSL',
       'sasl.username': namespace,
       'sasl.password': password,
       'group.id': 'test_group_id',  # this will create a new consumer group on each invocation.
       'auto.offset.reset': 'earliest',
       'error_cb': error_cb,
       'isolation.level': 'read_uncommitted', # Note: Ursa does not support read_committed for now
   })

   c.subscribe([topicName])

   try:
       while True:
           print('polling...')
           # Wait for message or event/error
           msg = c.poll(1)
           if msg is None:
               continue
           print('Consumed: {}'.format(msg.value()))
           break

   except KeyboardInterrupt:
       pass

   finally:
       c.close()
   ```

   * `SERVER-URL`: the Kafka service URL of your StreamNative cluster.
   * `YOUR-API-KEY`: an API key of your service account.

3. Run the Python application and you should see the following output:

   ```bash theme={null}
   Produced to: test-py-topic [0] @ 30
   polling...
   polling...
   polling...
   polling...
   polling...
   polling...
   Consumed: b'hello world'
   ```


# Test Kafka Protocol as code
Source: https://docs.streamnative.io/cloud/build/kafka-clients/test-kafka-protocol



[TestContainers](https://testcontainers.com/) provides a lightweight, disposable containers for integration testing purposes. Ursa allows for the seamless initiation of a Ursa service alongside TestContainers, simplifying the testing process. You can use the Ursa testcontainer module for:

* **Local testing**: Ursa testcontainer module allows you to start a Ursa service with one line code. This will facilitate the convenience of conducting tests and experiencing it locally at any time.
* **Integration Testing**: Ursa testcontainer module is particularly useful for integration testing, where you need to test the interactions between your application and Ursa dependencies. Instead of relying on mocked or stubbed versions of these dependencies, Ursa testcontainer module allows you to spin up real, isolated containers for each test run. This ensures that your tests closely resemble the actual runtime environment and can catch issues that may not be apparent in unit tests.
* **Cross-Platform Testing**: If your application needs to run on different platforms or environments, Ursa testcontainer module can help ensure consistent behavior across these environments. By encapsulating your Ursa dependency in containers, you can run your tests on any machine that supports Docker, regardless of the underlying operating system or infrastructure.
* **Dependency Management**: Ursa testcontainer module simplifies the management of dependencies for your tests. Instead of manually setting up and tearing down external resources, Testcontainers handles the lifecycle of the containers, automatically starting them before your tests and stopping them afterward. This saves you from the hassle of maintaining complex setup and teardown code.
* **Parallel Testing**: Ursa testcontainer module supports parallel test execution, allowing you to run multiple tests concurrently without conflicts. Each test can have its own isolated container, ensuring independence and avoiding interference between tests.
* **Continuous Integration/Continuous Delivery (CI/CD)**: Ursa testcontainer module integrates well with CI/CD pipelines, enabling you to run your integration tests as part of your automated build and deployment process. By including Ursa testcontainer module in your CI/CD pipeline, you can ensure that your application's integration points are thoroughly tested before deployment.

### Step 1: Import Ursa testcontainer

You can import the Ursa testcontainer dependency as well as the testcontainers dependency by adding the followings to your pom.xml

```xml theme={null}
<dependency>
  <groupId>org.testcontainers</groupId>
  <artifactId>testcontainers</artifactId>
  <version>1.19.1</version>
</dependency>
<dependency>
  <groupId>io.streamnative.ksn</groupId>
  <artifactId>ksn-testcontainer</artifactId>
  <version>0.1.0</version>
</dependency>
```

### Step 2: Set up a Ursa cluster

```java theme={null}
final KsnCluster cluster = new KsnCluster();
```

### Step 3: Use the created Ursa cluster in your tests

```java theme={null}
// Create a Kafka admin
final Properties adminProps = new Properties();
adminProps.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, cluster.getBootstrapServers());
final AdminClient client = AdminClient.create(adminProps);

// Create a Kafka producer
final Properties producerProps = new Properties();
producerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, cluster.getBootstrapServers());
producerProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
final KafkaProducer<String, String> producer = new KafkaProducer<>(producerProps);

// Create a Kafka consumer
final Properties consumerProps = new Properties();
consumerProps.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, cluster.getBootstrapServers());
consumerProps.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
consumerProps.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
consumerProps.put(ConsumerConfig.GROUP_ID_CONFIG, "group");
consumerProps.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
final KafkaConsumer<String, String> consumer = new KafkaConsumer<>(consumerProps);
```

### Step 4: Close the Ursa cluster after testing

```java theme={null}
cluster.close();
```


# Data Governance on StreamNative Cloud
Source: https://docs.streamnative.io/cloud/governance/governance-overview



Data governance is a critical aspect of any data platform. StreamNative Cloud provides a comprehensive set of data governance capabilities to help you govern your data streams effectively. This section provides an overview of the data governance capabilities available on StreamNative Cloud.

## Schema Registry

Schema Registry allows teams to define and enforce universal data standards that enable scalable data compatibility while reducing operational complexity.

As a multi-protocol platform, StreamNative Cloud currently supports schema management for two different protocols: the built-in Pulsar schema registry and the Kafka Schema Registry, which is compatible with the open-source Confluent Schema Registry API. These two schema registries are not currently interoperable. When building your applications, ensure that producers and consumers use the same schema registry. Work is ongoing to make these two schema registries interoperable within StreamNative Cloud.

### Pulsar Schema Registry

The Pulsar schema registry is built into the brokers. You can use Pulsar CLI tools to manage your schemas. See [Pulsar Schema](https://pulsar.apache.org/docs/schema-overview/) for more information.

### Kafka Schema Registry

Kafka schema registry is introduced as part of the Kafka protocol support on StreamNative Cloud. Currently, it is compatible with the Confluent Schema Registry API.

See [Kafka Schema Registry](/cloud/governance/kafka-schemas/kafka-schema-registry) for more information.

## Related

To use Kafka schemas from Pulsar Java clients:

* [Use External JSON Schema with Pulsar clients](/cloud/governance/kafka-schemas/external-json-schema)
* [Use External Avro Schema with Pulsar clients](/cloud/governance/kafka-schemas/external-avro-schema)
* [Use External Protobuf Schema with Pulsar clients](/cloud/governance/kafka-schemas/external-protobuf-schema)


# Use External Avro Schema with Pulsar clients
Source: https://docs.streamnative.io/cloud/governance/kafka-schemas/external-avro-schema

Use Kafka Avro Schema and the Kafka Schema Registry from Pulsar Java clients with the kafka-schemas library.

External Avro Schema lets Pulsar Java clients produce and consume messages that use [Kafka Avro Schema](/cloud/governance/kafka-schemas/kafka-schema-registry) and the [Kafka Schema Registry](/cloud/governance/kafka-schemas/kafka-schema-registry) on StreamNative Cloud. Schemas are registered in and resolved from the Kafka Schema Registry, while your application uses the familiar Pulsar `Producer` and `Consumer` APIs.

Use External Avro Schema when you want to:

* Use Pulsar clients with Kafka Avro Schema and Schema Registry compatibility checks.
* Share Avro schemas between Kafka and Pulsar clients on the same topic.
* Work with Avro `SpecificRecord` classes generated from `.avsc` schema files.

The [`kafka-schemas`](https://github.com/streamnative/external-schemas) library provides a Pulsar `Schema` implementation backed by the Kafka Avro serializer. The same library also supports [External JSON Schema](/cloud/governance/kafka-schemas/external-json-schema) and [External Protobuf Schema](/cloud/governance/kafka-schemas/external-protobuf-schema).

## Prerequisites

* A StreamNative Pulsar cluster for message production and consumption.
* The [Kafka Schema Registry](/cloud/governance/kafka-schemas/kafka-schema-registry) service enabled on the cluster.
* A service account with `produce` and `consume` permissions on the target topic.
* RBAC permissions for the Kafka Schema Registry: assign the [`schema-writer`](/cloud/security/access/rbac/manage-rbac-roles#schema-writer) role to register schemas and the [`schema-reader`](/cloud/security/access/rbac/manage-rbac-roles#schema-reader) role to read schemas. See [Schema Registry RBAC roles](/cloud/security/access/rbac/manage-rbac-roles#schema-registry).
* Java 17 or higher.
* Pulsar Java client 4.1.0 or higher.

## Add the dependency

Add the following Maven dependencies to your project:

```xml theme={null}
<dependency>
  <groupId>org.apache.pulsar</groupId>
  <artifactId>pulsar-client</artifactId>
  <version>4.1.0</version>
  <exclusions>
      <exclusion>
          <groupId>javax.validation</groupId>
          <artifactId>validation-api</artifactId>
      </exclusion>
  </exclusions>
</dependency>
<dependency>
  <groupId>io.streamnative.schemas.external</groupId>
  <artifactId>kafka-schemas</artifactId>
  <version>1.0.0</version>
</dependency>
<dependency>
  <groupId>io.confluent</groupId>
  <artifactId>kafka-avro-serializer</artifactId>
  <version>8.0.0</version>
</dependency>
<dependency>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro</artifactId>
  <version>1.12.0</version>
</dependency>
```

The `pulsar-client` dependency provides the Pulsar `Producer`, `Consumer`, and `PulsarClient` APIs used in the examples below. The `kafka-avro-serializer` dependency is required at runtime because `kafka-schemas` declares it with `provided` scope. Declare `avro` explicitly so you control its version; `kafka-schemas` also pulls it in transitively.

### Add the Kafka Maven repository

`pulsar-client`, `kafka-schemas`, and `avro` are available from [Maven Central](https://repo1.maven.org/maven2/). You do not need to add a repository for those dependencies.

`kafka-avro-serializer` is not published to Maven Central. Add the following repository to your `pom.xml`:

```xml theme={null}
<repositories>
  <repository>
    <id>kafka</id>
    <url>https://packages.confluent.io/maven/</url>
  </repository>
</repositories>
```

<Note>
  If your organization already mirrors `kafka-avro-serializer` artifacts in an internal repository, configure that mirror instead of adding the public repository directly.
</Note>

## Define an Avro schema

External Avro Schema works with Avro `SpecificRecord` classes. Define a schema in an `.avsc` file and generate the Java class with the Avro Maven plugin.

### Step 1: Create a schema file

Create `src/main/avro/Player.avsc`:

```json theme={null}
{
  "namespace": "com.example.avro",
  "type": "record",
  "name": "Player",
  "doc": "A player's profile information.",
  "fields": [
    {
      "name": "name",
      "type": "string",
      "doc": "The player's full name."
    },
    {
      "name": "number",
      "type": ["null", "int"],
      "default": null,
      "doc": "The player's number (optional)."
    },
    {
      "name": "favorite_color",
      "type": ["null", "string"],
      "default": null,
      "doc": "The player's favorite color (optional)."
    }
  ]
}
```

### Step 2: Generate the SpecificRecord class

Add the Avro Maven plugin to your `pom.xml`:

```xml theme={null}
<plugin>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro-maven-plugin</artifactId>
  <version>1.12.0</version>
  <executions>
    <execution>
      <phase>generate-sources</phase>
      <goals>
        <goal>schema</goal>
      </goals>
      <configuration>
        <sourceDirectory>${project.basedir}/src/main/avro</sourceDirectory>
      </configuration>
    </execution>
  </executions>
</plugin>
```

Run `mvn generate-sources` to generate the `Player` class in the `com.example.avro` package.

## Configure Schema Registry authentication

`KafkaSchemaFactory` accepts the same Schema Registry configuration properties as `KafkaAvroSerializerConfig`. The [`external-schemas`](https://github.com/streamnative/external-schemas) examples authenticate to the Schema Registry with Basic authentication. Define the helper method as `private static` so you can call it from `main`.

Use your service account API key as the password. The username can be any non-empty string.

```java theme={null}
private static Map<String, Object> getSchemaRegistryConfigs(String schemaRegistryUrl, String apiKey) {
    Map<String, Object> configs = new HashMap<>();
    configs.put(KafkaAvroSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl);
    configs.put(KafkaAvroSerializerConfig.BASIC_AUTH_CREDENTIALS_SOURCE, "USER_INFO");
    configs.put(
            KafkaAvroSerializerConfig.USER_INFO_CONFIG,
            String.format("%s:%s", "public", apiKey));
    return configs;
}
```

For additional serializer options, see the `KafkaAvroSerializerConfig` class in the `kafka-avro-serializer` dependency.

## Produce and consume messages

Use `KafkaSchemaFactory` to create a Pulsar `Schema` backed by Kafka Avro Schema, then create a producer and consumer with the same schema instance.

```java theme={null}
import com.example.avro.Player;
import io.confluent.kafka.serializers.KafkaAvroSerializerConfig;
import io.streamnative.schemas.external.KafkaSchemaFactory;
import java.util.HashMap;
import java.util.Map;
import org.apache.pulsar.client.api.AuthenticationFactory;
import org.apache.pulsar.client.api.Consumer;
import org.apache.pulsar.client.api.Message;
import org.apache.pulsar.client.api.Producer;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.client.api.SubscriptionInitialPosition;

public class ExternalAvroSchemaExample {

    public static void main(String[] args) throws Exception {
        String serviceUrl = "<YOUR-PULSAR-SERVICE-URL>";
        String schemaRegistryUrl = "<YOUR-SCHEMA-REGISTRY-URL>";
        String apiKey = "<YOUR-API-KEY>";
        String topic = "persistent://public/default/players";

        KafkaSchemaFactory schemaFactory =
                new KafkaSchemaFactory(getSchemaRegistryConfigs(schemaRegistryUrl, apiKey));
        Schema<Player> schema = schemaFactory.avro(Player.class);

        PulsarClient client = PulsarClient.builder()
                .serviceUrl(serviceUrl)
                .authentication(AuthenticationFactory.token(apiKey))
                .build();

        Producer<Player> producer = client.newProducer(schema).topic(topic).create();
        Consumer<Player> consumer = client.newConsumer(schema)
                .topic(topic)
                .subscriptionName("my-subscription")
                .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
                .subscribe();

        for (int i = 0; i < 10; i++) {
            Player player = new Player();
            player.setName("name-" + i);
            player.setNumber(i);
            player.setFavoriteColor("color-" + i);
            producer.send(player);
        }

        for (int i = 0; i < 10; i++) {
            Message<Player> message = consumer.receive();
            consumer.acknowledge(message);
            Player player = message.getValue();
            System.out.println("name=>" + player.getName()
                    + ", number=>" + player.getNumber()
                    + ", favoriteColor=>" + player.getFavoriteColor());
        }

        consumer.close();
        producer.close();
        client.close();
    }

    private static Map<String, Object> getSchemaRegistryConfigs(String schemaRegistryUrl, String apiKey) {
        Map<String, Object> configs = new HashMap<>();
        configs.put(KafkaAvroSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl);
        configs.put(KafkaAvroSerializerConfig.BASIC_AUTH_CREDENTIALS_SOURCE, "USER_INFO");
        configs.put(
                KafkaAvroSerializerConfig.USER_INFO_CONFIG,
                String.format("%s:%s", "public", apiKey));
        return configs;
    }
}
```

When the producer sends the first message, the schema is automatically registered in the Kafka Schema Registry. The consumer resolves the schema from the registry when reading messages.

## Schema compatibility

External Avro Schema registers schemas with the Pulsar schema type `EXTERNAL`. A topic cannot mix `EXTERNAL` schemas with native Pulsar schemas such as `JSON`, `AVRO`, or `PROTOBUF` on the same topic.

For example, if a topic already uses Pulsar's built-in `Schema.AVRO(Player.class)`, creating a producer with External Avro Schema on the same topic fails with an incompatible schema error:

```
Incompatible schema: exists schema type AVRO, new schema type EXTERNAL
```

<Warning>
  Plan your schema strategy before publishing to a topic. Once a topic uses External Avro Schema, all producers and consumers on that topic must use the same External Avro Schema type.
</Warning>

Schema compatibility modes for Avro in the Kafka Schema Registry are described in [Configurable compatibility modes](/cloud/governance/kafka-schemas/kafka-schema-registry#configurable-compatibility-modes). StreamNative Cloud supports a subset of the Kafka Schema Registry REST API. See the [REST API](/cloud/governance/kafka-schemas/kafka-schema-registry#rest-api) section for supported operations.

## Related resources

<CardGroup>
  <Card title="External JSON Schema" icon="brackets-curly" href="/cloud/governance/kafka-schemas/external-json-schema">
    Use Kafka JSON Schema from Pulsar Java clients.
  </Card>

  <Card title="External Protobuf Schema" icon="code" href="/cloud/governance/kafka-schemas/external-protobuf-schema">
    Use Kafka Protobuf Schema from Pulsar Java clients.
  </Card>

  <Card title="Kafka Schema Registry" icon="database" href="/cloud/governance/kafka-schemas/kafka-schema-registry">
    Configure authentication, compatibility modes, and REST API access.
  </Card>

  <Card title="external-schemas repository" icon="github" href="https://github.com/streamnative/external-schemas">
    View source code, tests, and release notes for the kafka-schemas library.
  </Card>
</CardGroup>


# Use External JSON Schema with Pulsar clients
Source: https://docs.streamnative.io/cloud/governance/kafka-schemas/external-json-schema

Use Kafka JSON Schema and the Kafka Schema Registry from Pulsar Java clients with the kafka-schemas library.

External JSON Schema lets Pulsar Java clients produce and consume messages that use [Kafka JSON Schema](/cloud/governance/kafka-schemas/kafka-schema-registry) and the [Kafka Schema Registry](/cloud/governance/kafka-schemas/kafka-schema-registry) on StreamNative Cloud. Schemas are registered in and resolved from the Kafka Schema Registry, while your application uses the familiar Pulsar `Producer` and `Consumer` APIs.

Use External JSON Schema when you want to:

* Use Pulsar clients with Kafka JSON Schema and Schema Registry compatibility checks.
* Share JSON schemas between Kafka and Pulsar clients on the same topic.
* Build Key-Value messages where the key, value, or both use Kafka JSON Schema.

The [`kafka-schemas`](https://github.com/streamnative/external-schemas) library (previously published as `kafka-json-schema`) provides a Pulsar `Schema` implementation backed by the Kafka JSON Schema serializer. The same library also supports [External Avro Schema](/cloud/governance/kafka-schemas/external-avro-schema) and [External Protobuf Schema](/cloud/governance/kafka-schemas/external-protobuf-schema).

## Prerequisites

* A StreamNative Pulsar cluster for message production and consumption.
* The [Kafka Schema Registry](/cloud/governance/kafka-schemas/kafka-schema-registry) service enabled on the cluster.
* A service account with `produce` and `consume` permissions on the target topic.
* RBAC permissions for the Kafka Schema Registry: assign the [`schema-writer`](/cloud/security/access/rbac/manage-rbac-roles#schema-writer) role to register schemas and the [`schema-reader`](/cloud/security/access/rbac/manage-rbac-roles#schema-reader) role to read schemas. See [Schema Registry RBAC roles](/cloud/security/access/rbac/manage-rbac-roles#schema-registry).
* Java 17 or higher.
* Pulsar Java client 4.1.0 or higher.

## Add the dependency

Add the following Maven dependencies to your project:

```xml theme={null}
<dependency>
  <groupId>org.apache.pulsar</groupId>
  <artifactId>pulsar-client</artifactId>
  <version>4.1.0</version>
  <exclusions>
      <exclusion>
          <groupId>javax.validation</groupId>
          <artifactId>validation-api</artifactId>
      </exclusion>
  </exclusions>
</dependency>
<dependency>
  <groupId>io.streamnative.schemas.external</groupId>
  <artifactId>kafka-schemas</artifactId>
  <version>1.0.0</version>
</dependency>
<dependency>
  <groupId>io.confluent</groupId>
  <artifactId>kafka-json-schema-serializer</artifactId>
  <version>8.0.0</version>
</dependency>
```

The `pulsar-client` dependency provides the Pulsar `Producer`, `Consumer`, and `PulsarClient` APIs used in the examples below. The `kafka-json-schema-serializer` dependency is required at runtime because `kafka-schemas` declares it with `provided` scope.

### Add the Kafka Maven repository

`pulsar-client` and `kafka-schemas` are available from [Maven Central](https://repo1.maven.org/maven2/). You do not need to add a repository for those dependencies.

`kafka-json-schema-serializer` is not published to Maven Central. Add the following repository to your `pom.xml`:

```xml theme={null}
<repositories>
  <repository>
    <id>kafka</id>
    <url>https://packages.confluent.io/maven/</url>
  </repository>
</repositories>
```

<Note>
  If your organization already mirrors `kafka-json-schema-serializer` artifacts in an internal repository, configure that mirror instead of adding the public repository directly.
</Note>

## Configure Schema Registry authentication

`KafkaSchemaFactory` accepts the same Schema Registry configuration properties as `KafkaJsonSchemaSerializerConfig`. The [`external-schemas`](https://github.com/streamnative/external-schemas) examples authenticate to the Schema Registry with Basic authentication. Define the helper method as `private static` so you can call it from `main`.

Use your service account API key as the password. The username can be any non-empty string.

```java theme={null}
private static Map<String, Object> getSchemaRegistryConfigs(String schemaRegistryUrl, String apiKey) {
    Map<String, Object> configs = new HashMap<>();
    configs.put(KafkaJsonSchemaSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl);
    configs.put(KafkaJsonSchemaSerializerConfig.BASIC_AUTH_CREDENTIALS_SOURCE, "USER_INFO");
    configs.put(
            KafkaJsonSchemaSerializerConfig.USER_INFO_CONFIG,
            String.format("%s:%s", "public", apiKey));
    return configs;
}
```

For additional serializer options, see the `KafkaJsonSchemaSerializerConfig` class in the `kafka-json-schema-serializer` dependency.

## Produce and consume messages

The following example shows how to create a producer and consumer with External JSON Schema.

### Step 1: Define your message class

Define a POJO for your message payload. Lombok annotations are optional.

```java theme={null}
public class User {
    private String name;
    private Integer age;

    public User() {}

    public User(String name, Integer age) {
        this.name = name;
        this.age = age;
    }

    public String getName() { return name; }
    public void setName(String name) { this.name = name; }
    public Integer getAge() { return age; }
    public void setAge(Integer age) { this.age = age; }
}
```

### Step 2: Create a schema and connect to your cluster

Use `KafkaSchemaFactory` to create a Pulsar `Schema` backed by Kafka JSON Schema, then create a producer and consumer with the same schema instance.

```java theme={null}
import io.confluent.kafka.serializers.json.KafkaJsonSchemaSerializerConfig;
import io.streamnative.schemas.external.KafkaSchemaFactory;
import java.util.HashMap;
import java.util.Map;
import org.apache.pulsar.client.api.AuthenticationFactory;
import org.apache.pulsar.client.api.Consumer;
import org.apache.pulsar.client.api.Message;
import org.apache.pulsar.client.api.Producer;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.client.api.SubscriptionInitialPosition;

public class ExternalJsonSchemaExample {

    public static void main(String[] args) throws Exception {
        String serviceUrl = "<YOUR-PULSAR-SERVICE-URL>";
        String schemaRegistryUrl = "<YOUR-SCHEMA-REGISTRY-URL>";
        String apiKey = "<YOUR-API-KEY>";
        String topic = "persistent://public/default/users";

        KafkaSchemaFactory schemaFactory =
                new KafkaSchemaFactory(getSchemaRegistryConfigs(schemaRegistryUrl, apiKey));
        Schema<User> schema = schemaFactory.json(User.class);

        PulsarClient client = PulsarClient.builder()
                .serviceUrl(serviceUrl)
                .authentication(AuthenticationFactory.token(apiKey))
                .build();

        Producer<User> producer = client.newProducer(schema).topic(topic).create();
        Consumer<User> consumer = client.newConsumer(schema)
                .topic(topic)
                .subscriptionName("my-subscription")
                .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
                .subscribe();

        for (int i = 0; i < 10; i++) {
            producer.send(new User("name-" + i, 10 + i));
        }

        for (int i = 0; i < 10; i++) {
            Message<User> message = consumer.receive();
            consumer.acknowledge(message);
            System.out.println(message.getValue().getName());
        }

        consumer.close();
        producer.close();
        client.close();
    }

    private static Map<String, Object> getSchemaRegistryConfigs(String schemaRegistryUrl, String apiKey) {
        Map<String, Object> configs = new HashMap<>();
        configs.put(KafkaJsonSchemaSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl);
        configs.put(KafkaJsonSchemaSerializerConfig.BASIC_AUTH_CREDENTIALS_SOURCE, "USER_INFO");
        configs.put(
                KafkaJsonSchemaSerializerConfig.USER_INFO_CONFIG,
                String.format("%s:%s", "public", apiKey));
        return configs;
    }
}
```

When the producer sends the first message, the schema is automatically registered in the Kafka Schema Registry. The consumer resolves the schema from the registry when reading messages.

## Use Key-Value schemas

`KafkaSchemaFactory` also supports Key-Value messages. You can combine a native Pulsar schema for the key with External JSON Schema for the value, or use External JSON Schema for both key and value.

### Native Pulsar key with External JSON Schema value

Use a native Pulsar `Schema.STRING` key and an External JSON Schema value:

```java theme={null}
import org.apache.pulsar.common.schema.KeyValue;
import org.apache.pulsar.common.schema.KeyValueEncodingType;

Schema<KeyValue<String, User>> schema = schemaFactory.kv(
        Schema.STRING,
        schemaFactory.json(User.class),
        KeyValueEncodingType.INLINE);

Producer<KeyValue<String, User>> producer = client.newProducer(schema).topic(topic).create();
producer.send(new KeyValue<>("user-1", new User("Alice", 30)));
```

### External JSON Schema for both key and value

Use External JSON Schema for both the key and value:

```java theme={null}
public class UserKey {
    private Integer userId;
    private String name;
    // constructors and getters/setters omitted
}

Schema<KeyValue<UserKey, User>> schema = schemaFactory.kv(
        schemaFactory.json(UserKey.class),
        schemaFactory.json(User.class),
        KeyValueEncodingType.SEPARATED);
```

`KeyValueEncodingType` supports both `INLINE` and `SEPARATED` encoding, matching the behavior of Pulsar Key-Value schemas.

## Schema compatibility

External JSON Schema registers schemas with the Pulsar schema type `EXTERNAL`. A topic cannot mix `EXTERNAL` schemas with native Pulsar schemas such as `JSON`, `AVRO`, or `PROTOBUF` on the same topic.

For example, if a topic already uses Pulsar's built-in `Schema.JSON(User.class)`, creating a producer with External JSON Schema on the same topic fails with an incompatible schema error:

```
Incompatible schema: exists schema type JSON, new schema type EXTERNAL
```

<Warning>
  Plan your schema strategy before publishing to a topic. Once a topic uses External JSON Schema, all producers and consumers on that topic must use the same External JSON Schema type.
</Warning>

Schema compatibility modes for JSON Schema in the Kafka Schema Registry are described in [Configurable compatibility modes](/cloud/governance/kafka-schemas/kafka-schema-registry#configurable-compatibility-modes). StreamNative Cloud supports a subset of the Kafka Schema Registry REST API. See the [REST API](/cloud/governance/kafka-schemas/kafka-schema-registry#rest-api) section for supported operations.

## Related resources

<CardGroup>
  <Card title="External Avro Schema" icon="file-code" href="/cloud/governance/kafka-schemas/external-avro-schema">
    Use Kafka Avro Schema from Pulsar Java clients.
  </Card>

  <Card title="External Protobuf Schema" icon="code" href="/cloud/governance/kafka-schemas/external-protobuf-schema">
    Use Kafka Protobuf Schema from Pulsar Java clients.
  </Card>

  <Card title="Kafka Schema Registry" icon="database" href="/cloud/governance/kafka-schemas/kafka-schema-registry">
    Configure authentication, compatibility modes, and REST API access.
  </Card>

  <Card title="external-schemas repository" icon="github" href="https://github.com/streamnative/external-schemas">
    View source code, tests, and release notes for the kafka-schemas library.
  </Card>
</CardGroup>


# Use External Protobuf Schema with Pulsar clients
Source: https://docs.streamnative.io/cloud/governance/kafka-schemas/external-protobuf-schema

Use Kafka Protobuf Schema and the Kafka Schema Registry from Pulsar Java clients with the kafka-schemas library.

External Protobuf Schema lets Pulsar Java clients produce and consume messages that use [Kafka Protobuf Schema](/cloud/governance/kafka-schemas/kafka-schema-registry) and the [Kafka Schema Registry](/cloud/governance/kafka-schemas/kafka-schema-registry) on StreamNative Cloud. Schemas are registered in and resolved from the Kafka Schema Registry, while your application uses the familiar Pulsar `Producer` and `Consumer` APIs.

Use External Protobuf Schema when you want to:

* Use Pulsar clients with Kafka Protobuf Schema and Schema Registry compatibility checks.
* Share Protobuf schemas between Kafka and Pulsar clients on the same topic.
* Work with Protobuf message classes generated from `.proto` files.

The [`kafka-schemas`](https://github.com/streamnative/external-schemas) library provides a Pulsar `Schema` implementation backed by the Kafka Protobuf serializer. The same library also supports [External JSON Schema](/cloud/governance/kafka-schemas/external-json-schema) and [External Avro Schema](/cloud/governance/kafka-schemas/external-avro-schema).

## Prerequisites

* A StreamNative Pulsar cluster for message production and consumption.
* The [Kafka Schema Registry](/cloud/governance/kafka-schemas/kafka-schema-registry) service enabled on the cluster.
* A service account with `produce` and `consume` permissions on the target topic.
* RBAC permissions for the Kafka Schema Registry: assign the [`schema-writer`](/cloud/security/access/rbac/manage-rbac-roles#schema-writer) role to register schemas and the [`schema-reader`](/cloud/security/access/rbac/manage-rbac-roles#schema-reader) role to read schemas. See [Schema Registry RBAC roles](/cloud/security/access/rbac/manage-rbac-roles#schema-registry).
* Java 17 or higher.
* Pulsar Java client 4.1.0 or higher.

## Add the dependency

Add the following Maven dependencies to your project:

```xml theme={null}
<dependency>
  <groupId>org.apache.pulsar</groupId>
  <artifactId>pulsar-client</artifactId>
  <version>4.1.0</version>
  <exclusions>
      <exclusion>
          <groupId>javax.validation</groupId>
          <artifactId>validation-api</artifactId>
      </exclusion>
  </exclusions>
</dependency>
<dependency>
  <groupId>io.streamnative.schemas.external</groupId>
  <artifactId>kafka-schemas</artifactId>
  <version>1.0.0</version>
</dependency>
<dependency>
  <groupId>io.confluent</groupId>
  <artifactId>kafka-protobuf-serializer</artifactId>
  <version>8.0.0</version>
</dependency>
<dependency>
  <groupId>com.google.protobuf</groupId>
  <artifactId>protobuf-java</artifactId>
  <version>4.29.5</version>
</dependency>
```

The `pulsar-client` dependency provides the Pulsar `Producer`, `Consumer`, and `PulsarClient` APIs used in the examples below. Declare `kafka-protobuf-serializer` and `protobuf-java` explicitly so you control their versions. `kafka-schemas` also pulls them in transitively.

### Add the Kafka Maven repository

`pulsar-client`, `kafka-schemas`, and `protobuf-java` are available from [Maven Central](https://repo1.maven.org/maven2/). You do not need to add a repository for those dependencies.

`kafka-protobuf-serializer` is not published to Maven Central. Add the following repository to your `pom.xml`:

```xml theme={null}
<repositories>
  <repository>
    <id>kafka</id>
    <url>https://packages.confluent.io/maven/</url>
  </repository>
</repositories>
```

<Note>
  If your organization already mirrors `kafka-protobuf-serializer` artifacts in an internal repository, configure that mirror instead of adding the public repository directly.
</Note>

## Define a Protobuf schema

External Protobuf Schema works with Protobuf message classes generated from `.proto` files.

### Step 1: Create Protobuf definition files

Create `src/main/proto/other.proto`:

```protobuf theme={null}
syntax = "proto3";
package com.example.protobuf;

option java_multiple_files = true;
option java_package = "com.example.protobuf";

message OtherRecord {
  int32 other_id = 1;
}
```

Create `src/main/proto/myRecord.proto`:

```protobuf theme={null}
syntax = "proto3";
package com.example.protobuf;

option java_multiple_files = true;
option java_package = "com.example.protobuf";

import "other.proto";

message MyRecord {
  string f1 = 1;
  OtherRecord f2 = 2;
}
```

### Step 2: Generate the Protobuf classes

Add a Protobuf Maven plugin to your `pom.xml`. The following example uses the `protobuf-maven-plugin`:

```xml theme={null}
<plugin>
  <groupId>io.github.ascopes</groupId>
  <artifactId>protobuf-maven-plugin</artifactId>
  <version>3.10.1</version>
  <configuration>
    <protocVersion>4.29.5</protocVersion>
  </configuration>
  <executions>
    <execution>
      <goals>
        <goal>generate</goal>
      </goals>
    </execution>
  </executions>
</plugin>
```

Run `mvn generate-sources` to generate the `MyRecord` and `OtherRecord` classes in the `com.example.protobuf` package.

## Configure Schema Registry authentication

`KafkaSchemaFactory` accepts the same Schema Registry configuration properties as `KafkaProtobufSerializerConfig`. The [`external-schemas`](https://github.com/streamnative/external-schemas) examples authenticate to the Schema Registry with Basic authentication. Define the helper method as `private static` so you can call it from `main`.

Use your service account API key as the password. The username can be any non-empty string.

```java theme={null}
private static Map<String, Object> getSchemaRegistryConfigs(String schemaRegistryUrl, String apiKey) {
    Map<String, Object> configs = new HashMap<>();
    configs.put(KafkaProtobufSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl);
    configs.put(KafkaProtobufSerializerConfig.BASIC_AUTH_CREDENTIALS_SOURCE, "USER_INFO");
    configs.put(
            KafkaProtobufSerializerConfig.USER_INFO_CONFIG,
            String.format("%s:%s", "public", apiKey));
    return configs;
}
```

For additional serializer options, see the `KafkaProtobufSerializerConfig` class in the `kafka-protobuf-serializer` dependency.

## Produce and consume messages

Use `KafkaSchemaFactory` to create a Pulsar `Schema` backed by Kafka Protobuf Schema, then create a producer and consumer with the same schema instance.

```java theme={null}
import com.example.protobuf.MyRecord;
import com.example.protobuf.OtherRecord;
import io.confluent.kafka.serializers.protobuf.KafkaProtobufSerializerConfig;
import io.streamnative.schemas.external.KafkaSchemaFactory;
import java.util.HashMap;
import java.util.Map;
import org.apache.pulsar.client.api.AuthenticationFactory;
import org.apache.pulsar.client.api.Consumer;
import org.apache.pulsar.client.api.Message;
import org.apache.pulsar.client.api.Producer;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.Schema;
import org.apache.pulsar.client.api.SubscriptionInitialPosition;

public class ExternalProtobufSchemaExample {

    public static void main(String[] args) throws Exception {
        String serviceUrl = "<YOUR-PULSAR-SERVICE-URL>";
        String schemaRegistryUrl = "<YOUR-SCHEMA-REGISTRY-URL>";
        String apiKey = "<YOUR-API-KEY>";
        String topic = "persistent://public/default/protobuf-records";

        KafkaSchemaFactory schemaFactory =
                new KafkaSchemaFactory(getSchemaRegistryConfigs(schemaRegistryUrl, apiKey));
        Schema<MyRecord> schema = schemaFactory.protobuf(MyRecord.class);

        PulsarClient client = PulsarClient.builder()
                .serviceUrl(serviceUrl)
                .authentication(AuthenticationFactory.token(apiKey))
                .build();

        Producer<MyRecord> producer = client.newProducer(schema).topic(topic).create();
        Consumer<MyRecord> consumer = client.newConsumer(schema)
                .topic(topic)
                .subscriptionName("my-subscription")
                .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
                .subscribe();

        for (int i = 0; i < 10; i++) {
            MyRecord myRecord = MyRecord.newBuilder()
                    .setF1("name-" + i)
                    .setF2(OtherRecord.newBuilder().setOtherId(i).build())
                    .build();
            producer.send(myRecord);
        }

        for (int i = 0; i < 10; i++) {
            Message<MyRecord> message = consumer.receive();
            consumer.acknowledge(message);
            MyRecord myRecord = message.getValue();
            System.out.println("f1=>" + myRecord.getF1()
                    + ", f2.otherId=>" + myRecord.getF2().getOtherId());
        }

        consumer.close();
        producer.close();
        client.close();
    }

    private static Map<String, Object> getSchemaRegistryConfigs(String schemaRegistryUrl, String apiKey) {
        Map<String, Object> configs = new HashMap<>();
        configs.put(KafkaProtobufSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl);
        configs.put(KafkaProtobufSerializerConfig.BASIC_AUTH_CREDENTIALS_SOURCE, "USER_INFO");
        configs.put(
                KafkaProtobufSerializerConfig.USER_INFO_CONFIG,
                String.format("%s:%s", "public", apiKey));
        return configs;
    }
}
```

When the producer sends the first message, the schema is automatically registered in the Kafka Schema Registry. The consumer resolves the schema from the registry when reading messages.

## Schema compatibility

External Protobuf Schema registers schemas with the Pulsar schema type `EXTERNAL`. A topic cannot mix `EXTERNAL` schemas with native Pulsar schemas such as `JSON`, `AVRO`, or `PROTOBUF` on the same topic.

For example, if a topic already uses Pulsar's built-in Protobuf schema, creating a producer with External Protobuf Schema on the same topic fails with an incompatible schema error:

```
Incompatible schema: exists schema type PROTOBUF, new schema type EXTERNAL
```

<Warning>
  Plan your schema strategy before publishing to a topic. Once a topic uses External Protobuf Schema, all producers and consumers on that topic must use the same External Protobuf Schema type.
</Warning>

Schema compatibility modes for Protobuf in the Kafka Schema Registry are described in [Configurable compatibility modes](/cloud/governance/kafka-schemas/kafka-schema-registry#configurable-compatibility-modes). StreamNative Cloud supports a subset of the Kafka Schema Registry REST API. See the [REST API](/cloud/governance/kafka-schemas/kafka-schema-registry#rest-api) section for supported operations.

## Related resources

<CardGroup>
  <Card title="External JSON Schema" icon="brackets-curly" href="/cloud/governance/kafka-schemas/external-json-schema">
    Use Kafka JSON Schema from Pulsar Java clients.
  </Card>

  <Card title="External Avro Schema" icon="file-code" href="/cloud/governance/kafka-schemas/external-avro-schema">
    Use Kafka Avro Schema from Pulsar Java clients.
  </Card>

  <Card title="Kafka Schema Registry" icon="database" href="/cloud/governance/kafka-schemas/kafka-schema-registry">
    Configure authentication, compatibility modes, and REST API access.
  </Card>

  <Card title="external-schemas repository" icon="github" href="https://github.com/streamnative/external-schemas">
    View source code, tests, and release notes for the kafka-schemas library.
  </Card>
</CardGroup>


# Kafka Schema Registry
Source: https://docs.streamnative.io/cloud/governance/kafka-schemas/kafka-schema-registry



Kafka Schema Registry provides an interface for storing and managing schemas. Producers and consumers can register the schemas within the registry and retrieve them when necessary. Schemas are versioned, and the registry supports configurable compatibility modes between different schema versions. When a producer or consumer attempts to register a new schema version, the registry performs a compatibility check and returns an error if an incompatible change is detected. This mechanism ensures consistency and compatibility among all producers and consumers when schema changes occur.

## Access Schema Registry in Kafka clients

To access the Kafka Schema Registry, you must configure how to authenticate. There are two ways to configure authentication to the Schema Registry:

* OAuth2 authentication: only available for Kafka Java client
* Basic authentication: available for all Kafka clients

### OAuth2 authentication

First, import the following dependencies:

```xml theme={null}
    <dependency>
      <groupId>org.apache.kafka</groupId>
      <artifactId>kafka-clients</artifactId>
      <version>3.6.1</version>
    </dependency>
    <dependency>
      <groupId>io.streamnative.pulsar.handlers</groupId>
      <artifactId>oauth-client</artifactId>
      <version>3.2.2.6</version>
    </dependency>
    <dependency>
      <groupId>io.confluent</groupId>
      <artifactId>kafka-avro-serializer</artifactId>
      <version>7.5.0</version>
    </dependency>
```

Minimum required versions:

* `kafka-clients`: 3.4.0
* `oauth-client`: 3.1.0.4
* `kafka-avro-serializer`: 7.5.0

<Note title="Note">
  Before 3.2.2.6, `oauth-client` requires Java 17 or higher.
</Note>

Then, in addition to the existing properties, you need to configure more properties like:

```java theme={null}
// props is the Properties object that has already configures the OAuth2 authentication
// See https://docs.streamnative.io/docs/cloud-connect-kafka-java for the necessary configs
props.put(KafkaAvroSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl);
props.put(KafkaAvroSerializerConfig.BEARER_AUTH_CUSTOM_PROVIDER_CLASS, "io.streamnative.pulsar.handlers.kop.security.oauth.schema.OauthCredentialProvider");
props.put(KafkaAvroSerializerConfig.BEARER_AUTH_CREDENTIALS_SOURCE, "CUSTOM");
```

### Basic authentication

Unlike the OAuth2 authentication, Basic authentication does not require the `oauth-client` dependency or `kafka-clients` >= 3.4.0.

The username can be any non-empty string. The password should be the the token (the `jwtToken` variable in the code below) of your account.

```java theme={null}
// props is the Properties object that has already configures the Token authentication
// See https://docs.streamnative.io/docs/cloud-connect-kafka-java for the necessary configs
props.put(KafkaAvroSerializerConfig.SCHEMA_REGISTRY_URL_CONFIG, schemaRegistryUrl);
props.put(KafkaAvroSerializerConfig.BASIC_AUTH_CREDENTIALS_SOURCE, "USER_INFO");
props.put(KafkaAvroSerializerConfig.USER_INFO_CONFIG, String.format("%s:%s", "any-user", jwtToken));
```

## Configurable compatibility modes

When using serialization and deserialization formats such as Avro, JSON Schema, and Protobuf, we need to remember that there are different configurable compatibility modes. In the Schema Registry, schema compatibility is managed by versioning each individual schema. The compatibility type determines how the Schema Registry compares the new schema with previous versions of a schema, for a given subject. Upon its initial creation within a subject, a schema is assigned a unique identifier and a version number, starting at version 1. If the schema is updated and successfully passes the compatibility checks, it is given a new unique identifier and an incremented version number, i.e., version 2.

|                      |      |      |          |
| -------------------- | ---- | ---- | -------- |
| Compatibility modes  | AVRO | JSON | Protobuf |
| NONE                 | YES  | YES  | YES      |
| BACKWARD             | YES  | YES  | YES      |
| BACKWARD\_TRANSITIVE | YES  | YES  | YES      |
| FORWARD              | YES  | YES  | -        |
| FORWARD\_TRANSITIVE  | YES  | YES  | -        |
| FULL                 | YES  | YES  | -        |
| FULL\_TRANSITIVE     | YES  | YES  | -        |

## REST API

Kafka schema registry provides REST API for managing schemas. The following table lists the supported methods and parameters, more details about the API, please refer to [Schema Registry API](https://docs.confluent.io/platform/current/schema-registry/develop/api.html).

| API                                                                      | Method | Support Parameters                       |
| ------------------------------------------------------------------------ | ------ | ---------------------------------------- |
| `/schemas/ids/{int: id}`                                                 | GET    |                                          |
| `/schemas/ids/{int: id}/schema`                                          | GET    |                                          |
| `/schemas/types`                                                         | GET    |                                          |
| `/schemas/ids/{int: id}/versions`                                        | GET    |                                          |
| `/schemas/ids/{int: id}/subjects`                                        | GET    |                                          |
| `/subjects`                                                              | GET    | deleted (boolean), deletedOnly (boolean) |
| `/subjects/(string: subject)`                                            | POST   | normalize (boolean), deleted (boolean)   |
| `/subjects/(string: subject)`                                            | DELETE | permanent (boolean)                      |
| `/subjects/(string: subject)/versions`                                   | POST   | normalize (boolean)                      |
| `/subjects/(string: subject)/versions`                                   | GET    | deleted (boolean), deletedOnly (boolean) |
| `/subjects/(string: subject)/versions/(versionId: version)`              | GET    | deleted (boolean)                        |
| `/subjects/(string: subject)/versions/(versionId: version)`              | DELETE | permanent (boolean)                      |
| `/subjects/(string: subject)/versions/(versionId: version)/schema`       | GET    |                                          |
| `/subjects/(string: subject)/versions/(versionId: version)/referencedby` | GET    |                                          |
| `/compatibility/subjects/(string: subject)/versions/latest`              | GET    |                                          |
| `/config/(string: subject)`                                              | PUT    | only support set compatibility           |
| `/config/(string: subject)`                                              | GET    | only support get compatibility           |
| `/mode`                                                                  | GET    | only support the mode READWRITE          |

## Use Schema Registry on Console

1. On the left navigation pane of StreamNative Console, in the **Admin** section, click **Kafka Clients**, and choose the Java client, then enable the Kafka Schema Registry by following switch.
   <img alt="enable-kafka-schema-registry.png" />

2. Please make sure you granted permission(produce) for topic `public/__kafka_schemaregistry/__schema-registry` in the following page.
   <img alt="granted-permission-for-schema-registry-topic.png" />

We need to mention that Now the Kafka Schemas can’t work with Pulsar schemas. This is the mission of the unified schema registry.

## Enable Broker-side Schemas IDs Validation

Broker-side Schema ID Validation allows broker to validate the schema ID of the messages they send against the schema ID registered in the Schema Registry. This feature helps ensure that producers are sending messages with the correct schema, reducing the risk of data inconsistencies and errors. For more information, see [Validate Broker-side Schemas IDs](https://docs.confluent.io/platform/current/schema-registry/schema-validation.html).

### Limitations

Schema validation feature does not reject tombstone records (messages with null value) even if there is no schema ID associated with the record. This is to ensure that delete operations can still be performed on compacted topics without being blocked by schema validation.

### Enable Schema ID Validation on a Topic

Create a topic with Schema ID Validation enabled you can set the topic property `kop.kafka.key.schema.validation=true` and `kop.kafka.value.schema.validation=true` when creating the topic. For example, to create a topic named `my-topic-sv` with value schema validation, run the following command:

```bash theme={null}
snctl kafka admin topics create my-topic-sv --partitions 4 --config kop.kafka.value.schema.validation=true
```

Or

```bash theme={null}
bin/pulsar-admin \
    --admin-url <Pulsar Service Http Url> \
    --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
    --auth-params token:<API Key> \
  topics create-partitioned-topic persistent://public/default/my-topic-sv -p 4 -m kop.kafka.value.schema.validation=true
```

With this property set, if the message value does not have a schema ID or has a schema ID that does not match the schema registered in the Schema Registry, the broker will reject the message and return an error to the producer. And the message will be discarded.

### Change the subject name strategy

By default, the subject name strategy is set to `TopicNameStrategy`, which means that the subject name is derived from the topic name. If you want to change the subject name strategy, you can set the topic property `kop.kafka.schema.subject.name.strategy` to one of the following values:

* `TopicNameStrategy`: The subject name is derived from the topic name. For example, for a topic named `my-topic`, the subject name will be `my-topic-value` for value schema and `my-topic-key` for key schema.
* `RecordNameStrategy`: The subject name is derived from the fully qualified name of the record
* `TopicRecordNameStrategy`: The subject name is derived from the topic name and the fully qualified name of the record

For example, to create a topic named `my-topic-sv` with value schema validation and `RecordNameStrategy`, run the following command:

```bash theme={null}
snctl kafka admin topics create my-other-topic-sv --partitions 4 --config kop.kafka.value.schema.validation=true --config kop.kafka.value.subject.name.strategy=io.confluent.kafka.serializers.subject.RecordNameStrategy
```

Or

```bash theme={null}
bin/pulsar-admin \
    --admin-url <Pulsar Service Http Url> \
    --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
    --auth-params token:<API Key> \
  topics create-partitioned-topic persistent://public/default/my-other-topic-sv -p 4 -m kop.kafka.value.schema.validation=true -m kop.kafka.value.subject.name.strategy=io.confluent.kafka.serializers.subject.RecordNameStrategy
```

## Related

To use Kafka schemas from Pulsar Java clients:

* [Use External JSON Schema with Pulsar clients](/cloud/governance/kafka-schemas/external-json-schema)
* [Use External Avro Schema with Pulsar clients](/cloud/governance/kafka-schemas/external-avro-schema)
* [Use External Protobuf Schema with Pulsar clients](/cloud/governance/kafka-schemas/external-protobuf-schema)


# Overview of StreamNative Catalogs
Source: https://docs.streamnative.io/cloud/lakehouse/catalogs/catalog-overview



The StreamNative Catalog feature allows you to create connections to external catalog providers, such as Databricks Unity Catalog, Snowflake Open Catalog, Snowflake Horizon (Polaris) Catalog, Google BigLake metastore, and Amazon S3 Tables, to stream topic data from StreamNative Cloud as Iceberg tables.

You can register a catalog by selecting a provider, entering the required catalog details, and completing the registration process. StreamNative Cloud supports registering multiple catalogs per cluster, so a single Pulsar or Kafka cluster can land topics into different governance domains across BigQuery and Snowflake.

StreamNative Catalog is scoped at the organization level within StreamNative Cloud. Once a catalog is registered, it can be accessed and used across multiple StreamNative Clusters, enabling centralized catalog management and streamlined data access for streaming workloads.

<img alt="StreamNative Catalogs" />


# Manage StreamNative Catalogs
Source: https://docs.streamnative.io/cloud/lakehouse/catalogs/manage-catalogs



**View Catalogs**

On the **Catalogs** page, you can view a list of all catalogs that have been registered in StreamNative Cloud. The **Catalogs** page displays the following information for each registered catalog: Name, Catalog Type, Table Format, Status, and an Actions menu.

<img alt="StreamNative Catalogs" />

**Catalog Actions**

Each catalog listed on the **Catalogs** page includes an actions menu with the following options: **View Details** and **Delete**.

<img alt="StreamNative Catalogs" />

**Catalog Details**

Clicking on a catalog listed on the **Catalogs** page navigates you to the **Catalog Details** page, where you can view detailed information about the selected catalog.

<img alt="StreamNative Catalogs" />


# Private Networking for Databricks Unity Catalog (Iceberg)
Source: https://docs.streamnative.io/cloud/lakehouse/catalogs/private-networking/databricks-iceberg



This guide describes how to configure private network connections between StreamNative Cloud and Databricks Unity Catalog for Iceberg. Private connectivity ensures that traffic between your StreamNative BYOC cluster and Databricks Unity Catalog does not traverse the public internet.

Databricks Unity Catalog uses the same private connectivity infrastructure for both Iceberg and Delta Lake. If you use Delta Lake with Databricks, see [Private Networking for Databricks Unity Catalog (Delta Lake)](/cloud/lakehouse/catalogs/private-networking/databricks-unity-catalog).

The following diagram shows the network path between your StreamNative BYOC cluster and Databricks Unity Catalog over private connectivity.

```mermaid theme={null}
flowchart TB
    subgraph BYOC["StreamNative BYOC VPC"]
        Cluster["BYOC Cluster"]
        Endpoint["Private Endpoint"]
    end

    Databricks["Databricks Unity Catalog<br/>(Iceberg)"]

    Cluster -->|"catalog API requests"| Endpoint --> Databricks

    classDef byoc fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
    classDef edge fill:#fde68a,stroke:#b45309,color:#451a03
    classDef ext  fill:#ede9fe,stroke:#7c3aed,color:#4c1d95
    class Cluster byoc
    class Endpoint edge
    class Databricks ext
```

## Prerequisites

* A StreamNative BYOC cluster deployed on AWS, GCP, or Azure.
* A Databricks workspace in the same cloud provider and region as your StreamNative BYOC cluster.
* A prepared Databricks Unity Catalog for Iceberg. See [Prepare Lakehouse Catalogs](/cloud/lakehouse/prepare-lakehouse-catalogs) for the cloud-specific setup guides.
* Storage private connectivity configured (if applicable). See [Private Networking for Catalog Integration](/cloud/lakehouse/catalogs/private-networking/overview) for storage connectivity details.

## Configure private connectivity

Configure private connectivity to Databricks Unity Catalog based on the cloud provider where your StreamNative BYOC cluster is deployed.

<Tabs>
  <Tab title="AWS">
    Use AWS PrivateLink to establish a private connection between your StreamNative Cloud environment and Databricks Unity Catalog.

    Follow the Databricks documentation to configure inbound PrivateLink:
    [Configure Inbound PrivateLink for Databricks](https://docs.databricks.com/en/security/network/front-end/front-end-private-connect.html).
  </Tab>

  <Tab title="GCP">
    Use Google Cloud Private Service Connect to establish a private connection between your StreamNative Cloud environment and Databricks Unity Catalog.

    Follow the Databricks documentation to configure front-end Private Service Connect:
    [Configure Front-end Private Service Connect for Databricks on GCP](https://docs.databricks.com/gcp/en/security/network/front-end/front-end-private-connect.html).
  </Tab>

  <Tab title="Azure">
    Use Azure Private Link to establish a private connection between your StreamNative Cloud environment and Databricks Unity Catalog.

    Follow the Databricks documentation to configure inbound Private Link:
    [Configure Inbound Private Link for Databricks on Azure](https://learn.microsoft.com/en-us/azure/databricks/security/network/front-end/front-end-private-connect).
  </Tab>
</Tabs>

## Update the catalog URI

After enabling private connectivity, you may need to update the catalog URI in StreamNative Cloud. The catalog URI is configured during [catalog registration](/cloud/lakehouse/catalogs/register-catalog) and can be updated through [Manage Catalogs](/cloud/lakehouse/catalogs/manage-catalogs).

<Note>
  If your Databricks workspace is configured with private DNS, the existing workspace URL resolves to the private endpoint automatically and no URI change is needed. Otherwise, update the catalog URI to use the private endpoint hostname.
</Note>


# Private Networking for Databricks Unity Catalog (Delta Lake)
Source: https://docs.streamnative.io/cloud/lakehouse/catalogs/private-networking/databricks-unity-catalog



This guide describes how to configure private network connections between StreamNative Cloud and Databricks Unity Catalog for Delta Lake. Private connectivity ensures that traffic between your StreamNative BYOC cluster and Databricks Unity Catalog does not traverse the public internet.

Databricks Unity Catalog uses the same private connectivity infrastructure for both Delta Lake and Iceberg. If you use Iceberg with Databricks, see [Private Networking for Databricks Unity Catalog (Iceberg)](/cloud/lakehouse/catalogs/private-networking/databricks-iceberg).

The following diagram shows the network path between your StreamNative BYOC cluster and Databricks Unity Catalog over private connectivity.

```mermaid theme={null}
flowchart TB
    subgraph BYOC["StreamNative BYOC VPC"]
        Cluster["BYOC Cluster"]
        Endpoint["Private Endpoint"]
    end

    Databricks["Databricks Unity Catalog<br/>(Delta Lake)"]

    Cluster -->|"catalog API requests"| Endpoint --> Databricks

    classDef byoc fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
    classDef edge fill:#fde68a,stroke:#b45309,color:#451a03
    classDef ext  fill:#ede9fe,stroke:#7c3aed,color:#4c1d95
    class Cluster byoc
    class Endpoint edge
    class Databricks ext
```

## Prerequisites

* A StreamNative BYOC cluster deployed on AWS, GCP, or Azure.
* A Databricks workspace in the same cloud provider and region as your StreamNative BYOC cluster.
* A prepared Databricks Unity Catalog for Delta Lake. See [Prepare Lakehouse Catalogs](/cloud/lakehouse/prepare-lakehouse-catalogs) for the cloud-specific setup guides.
* Storage private connectivity configured (if applicable). See [Private Networking for Catalog Integration](/cloud/lakehouse/catalogs/private-networking/overview) for storage connectivity details.

## Configure private connectivity

Configure private connectivity to Databricks Unity Catalog based on the cloud provider where your StreamNative BYOC cluster is deployed.

<Tabs>
  <Tab title="AWS">
    Use AWS PrivateLink to establish a private connection between your StreamNative Cloud environment and Databricks Unity Catalog.

    Follow the Databricks documentation to configure inbound PrivateLink:
    [Configure Inbound PrivateLink for Databricks](https://docs.databricks.com/en/security/network/front-end/front-end-private-connect.html).
  </Tab>

  <Tab title="GCP">
    Use Google Cloud Private Service Connect to establish a private connection between your StreamNative Cloud environment and Databricks Unity Catalog.

    Follow the Databricks documentation to configure front-end Private Service Connect:
    [Configure Front-end Private Service Connect for Databricks on GCP](https://docs.databricks.com/gcp/en/security/network/front-end/front-end-private-connect.html).
  </Tab>

  <Tab title="Azure">
    Use Azure Private Link to establish a private connection between your StreamNative Cloud environment and Databricks Unity Catalog.

    Follow the Databricks documentation to configure inbound Private Link:
    [Configure Inbound Private Link for Databricks on Azure](https://learn.microsoft.com/en-us/azure/databricks/security/network/front-end/front-end-private-connect).
  </Tab>
</Tabs>

## Update the catalog URI

After enabling private connectivity, you may need to update the catalog URI in StreamNative Cloud. The catalog URI is configured during [catalog registration](/cloud/lakehouse/catalogs/register-catalog) and can be updated through [Manage Catalogs](/cloud/lakehouse/catalogs/manage-catalogs).

<Note>
  If your Databricks workspace is configured with private DNS, the existing workspace URL resolves to the private endpoint automatically and no URI change is needed. Otherwise, update the catalog URI to use the private endpoint hostname.
</Note>


# Private Networking for Google BigLake
Source: https://docs.streamnative.io/cloud/lakehouse/catalogs/private-networking/google-biglake



This guide describes how to configure private network connections between StreamNative Cloud and Google BigLake metastore. Private connectivity ensures that traffic between your StreamNative BYOC cluster and Google BigLake does not traverse the public internet.

Google BigLake is available only on GCP.

The following diagram shows the network path between your StreamNative BYOC cluster and Google BigLake metastore over private connectivity.

```mermaid theme={null}
flowchart TB
    subgraph BYOC["StreamNative BYOC VPC (GCP)"]
        Cluster["BYOC Cluster"]
        PGA["Private Google API Access"]
    end

    BL["Google BigLake Metastore"]

    Cluster -->|"catalog API requests"| PGA --> BL

    classDef byoc fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
    classDef edge fill:#fde68a,stroke:#b45309,color:#451a03
    classDef ext  fill:#ede9fe,stroke:#7c3aed,color:#4c1d95
    class Cluster byoc
    class PGA edge
    class BL ext
```

## Prerequisites

* A StreamNative BYOC cluster deployed on GCP.
* A prepared Google BigLake catalog. See [Prepare Google BigLake (Iceberg)](/cloud/lakehouse/prepare-catalogs/biglake/iceberg).
* Storage private connectivity configured (if applicable). See [Private Networking for Catalog Integration](/cloud/lakehouse/catalogs/private-networking/overview) for storage connectivity details.

## Configure private connectivity

Google BigLake metastore is a Google-managed service that runs within the Google Cloud network. On GCP, StreamNative configures private network connections to Google Cloud APIs by default in all StreamNative environments.

**No additional action is required on your side** in most cases. All traffic between your StreamNative BYOC cluster and Google BigLake stays within the Google private network automatically.

If your BYOC cluster runs in a [Shared VPC](https://cloud.google.com/vpc/docs/shared-vpc), you must configure private connectivity yourself. See [Private Networking for Catalog Integration](/cloud/lakehouse/catalogs/private-networking/overview#storage-private-connectivity) for details.


# Private Networking for Catalog Integration
Source: https://docs.streamnative.io/cloud/lakehouse/catalogs/private-networking/overview



When integrating StreamNative Cloud with external catalog providers in production environments, you may need to configure private network connections to ensure that traffic between StreamNative Cloud and your catalog and storage services does not traverse the public internet.

Private networking for lakehouse catalog integration involves two components:

* **Storage connectivity** — Private network connections between StreamNative Cloud and your object storage (S3, GCS, or Azure Blob Storage).
* **Catalog connectivity** — Private network connections between StreamNative Cloud and your catalog provider.

The following diagram shows the network path between StreamNative Cloud and the catalog and storage services over private connectivity.

```mermaid theme={null}
flowchart TB
    subgraph BYOC["StreamNative BYOC VPC"]
        Cluster["BYOC Cluster"]
        subgraph Endpoints["Private Endpoints"]
            StorageEP["Storage Private Endpoint"]
            CatalogEP["Catalog Private Endpoint"]
        end
    end

    Storage["Object Storage"]
    Catalog["Catalog Provider"]

    Cluster -->|"storage traffic"| StorageEP --> Storage
    Cluster -->|"catalog traffic"| CatalogEP --> Catalog

    classDef byoc fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
    classDef edge fill:#fde68a,stroke:#b45309,color:#451a03
    classDef ext  fill:#ede9fe,stroke:#7c3aed,color:#4c1d95
    class Cluster byoc
    class StorageEP,CatalogEP edge
    class Storage,Catalog ext
    style Endpoints fill:#fffbeb,stroke:#f59e0b,color:#92400e
```

## Prerequisites

* A StreamNative BYOC cluster deployed on AWS, GCP, or Azure.
* A registered catalog in StreamNative Cloud. See [Register Catalog](/cloud/lakehouse/catalogs/register-catalog).
* An active integration with a supported catalog provider. See the [External Tables Integrations](/cloud/lakehouse/external-tables/external-tables-overview) for setup guides.

## Storage private connectivity

Private network connections between StreamNative Cloud and object storage are handled differently depending on the cloud provider.

<Tabs>
  <Tab title="AWS">
    On AWS, StreamNative configures private network connections to Amazon S3 endpoints by default in all StreamNative environments. The S3 VPC endpoint is configured per VPC.

    **No action is required on your side.** All traffic between your StreamNative BYOC cluster and S3 stays within the AWS private network automatically.
  </Tab>

  <Tab title="GCP">
    On GCP, StreamNative configures private network connections to Google Cloud Storage (GCS) endpoints by default in all StreamNative environments. The GCS private endpoint is configured per network.

    **No action is required on your side** in most cases. All traffic between your StreamNative BYOC cluster and GCS stays within the Google private network automatically.

    **Exception — Shared VPC:** If your BYOC cluster runs in a [Shared VPC](https://cloud.google.com/vpc/docs/shared-vpc), StreamNative cannot modify the network in the host project. You must configure private connectivity to GCS yourself by either:

    * Enabling [Private Google Access](https://cloud.google.com/vpc/docs/configure-private-google-access) on the subnets used by the cluster, or
    * Creating a [Private Service Connect endpoint for Google APIs](https://cloud.google.com/vpc/docs/configure-private-service-connect-apis) (see [Shared VPC deployment patterns](https://cloud.google.com/vpc/docs/private-service-connect-deployments) for host-project guidance).
  </Tab>

  <Tab title="Azure">
    On Azure, you must configure a private endpoint for your Azure Storage account to enable private connectivity between StreamNative Cloud and your storage.

    To set up a private endpoint for your storage account, follow the instructions in the Azure documentation:
    [Configure Azure Storage private endpoints](https://learn.microsoft.com/en-us/azure/storage/common/storage-private-endpoints).
  </Tab>
</Tabs>

## Catalog private connectivity

To establish private network connections between StreamNative Cloud and your catalog provider, follow the guide for your catalog provider:

<CardGroup>
  <Card title="Snowflake Open Catalog" icon="snowflake" href="/cloud/lakehouse/catalogs/private-networking/snowflake-open-catalog">
    Configure private connectivity to Snowflake Open Catalog using AWS PrivateLink, GCP Private Service Connect, or Azure Private Link.
  </Card>

  <Card title="Snowflake Horizon Catalog" icon="snowflake" href="/cloud/lakehouse/catalogs/private-networking/snowflake-horizon-catalog">
    Configure private connectivity to Snowflake Horizon Catalog using AWS PrivateLink, GCP Private Service Connect, or Azure Private Link.
  </Card>

  <Card title="Databricks Delta Lake" icon="database" href="/cloud/lakehouse/catalogs/private-networking/databricks-unity-catalog">
    Configure private connectivity to Databricks Unity Catalog for Delta Lake using AWS PrivateLink, GCP Private Service Connect, or Azure Private Link.
  </Card>

  <Card title="Databricks Iceberg" icon="database" href="/cloud/lakehouse/catalogs/private-networking/databricks-iceberg">
    Configure private connectivity to Databricks Unity Catalog for Iceberg using AWS PrivateLink, GCP Private Service Connect, or Azure Private Link.
  </Card>

  <Card title="Amazon S3 Tables" icon="aws" href="/cloud/lakehouse/catalogs/private-networking/s3-tables">
    Private connectivity for Amazon S3 Tables on AWS. Configured by default — no action required.
  </Card>

  <Card title="Google BigLake" icon="google" href="/cloud/lakehouse/catalogs/private-networking/google-biglake">
    Private connectivity for Google BigLake on GCP. Configured by default — no action required.
  </Card>
</CardGroup>


# Private Networking for Amazon S3 Tables
Source: https://docs.streamnative.io/cloud/lakehouse/catalogs/private-networking/s3-tables



This guide describes how to configure private network connections between StreamNative Cloud and Amazon S3 Tables. Private connectivity ensures that traffic between your StreamNative BYOC cluster and Amazon S3 Tables does not traverse the public internet.

Amazon S3 Tables is available only on AWS.

The following diagram shows the network path between your StreamNative BYOC cluster and Amazon S3 Tables over private connectivity.

```mermaid theme={null}
flowchart TB
    subgraph BYOC["StreamNative BYOC VPC (AWS)"]
        Cluster["BYOC Cluster"]
        VPCE["S3 Tables VPC Endpoint"]
    end

    S3T["Amazon S3 Tables"]

    Cluster -->|"catalog and data traffic"| VPCE --> S3T

    classDef byoc fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
    classDef edge fill:#fde68a,stroke:#b45309,color:#451a03
    classDef ext  fill:#ede9fe,stroke:#7c3aed,color:#4c1d95
    class Cluster byoc
    class VPCE edge
    class S3T ext
```

## Prerequisites

* A StreamNative BYOC cluster deployed on AWS.
* A prepared Amazon S3 Tables catalog. See [Prepare Amazon S3 Tables (Iceberg)](/cloud/lakehouse/prepare-catalogs/s3table/iceberg).
* Storage private connectivity configured (if applicable). See [Private Networking for Catalog Integration](/cloud/lakehouse/catalogs/private-networking/overview) for storage connectivity details.

## Configure private connectivity

Amazon S3 Tables uses S3 endpoints for both data storage and catalog operations. On AWS, StreamNative configures private network connections to Amazon S3 endpoints by default in all StreamNative environments. The S3 VPC endpoint is configured per VPC.

**No additional action is required on your side.** All traffic between your StreamNative BYOC cluster and Amazon S3 Tables stays within the AWS private network automatically.


# Private Networking for Snowflake Horizon Catalog
Source: https://docs.streamnative.io/cloud/lakehouse/catalogs/private-networking/snowflake-horizon-catalog



This guide describes how to configure private network connections between StreamNative Cloud and Snowflake Horizon Catalog. Private connectivity ensures that traffic between your StreamNative BYOC cluster and Snowflake Horizon Catalog does not traverse the public internet.

Snowflake Horizon Catalog uses the same Snowflake private connectivity infrastructure as Snowflake Open Catalog. The setup process is identical across both catalog types. See also [Private Networking for Snowflake Open Catalog](/cloud/lakehouse/catalogs/private-networking/snowflake-open-catalog).

The following diagram shows the network path between your StreamNative BYOC cluster and Snowflake Horizon Catalog over private connectivity.

```mermaid theme={null}
flowchart TB
    subgraph BYOC["StreamNative BYOC VPC"]
        Cluster["BYOC Cluster"]
        Endpoint["Private Endpoint"]
    end

    Snowflake["Snowflake Horizon Catalog"]

    Cluster -->|"catalog API requests"| Endpoint --> Snowflake

    classDef byoc fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
    classDef edge fill:#fde68a,stroke:#b45309,color:#451a03
    classDef ext  fill:#ede9fe,stroke:#7c3aed,color:#4c1d95
    class Cluster byoc
    class Endpoint edge
    class Snowflake ext
```

## Prerequisites

* A StreamNative BYOC cluster deployed on AWS, GCP, or Azure.
* A Snowflake account with Horizon Catalog enabled, in the same cloud provider and region as your StreamNative BYOC cluster.
* Storage private connectivity configured (if applicable). See [Private Networking for Catalog Integration](/cloud/lakehouse/catalogs/private-networking/overview) for storage connectivity details.

## Configure private connectivity

Configure private connectivity to Snowflake Horizon Catalog based on the cloud provider where your StreamNative BYOC cluster is deployed.

<Tabs>
  <Tab title="AWS">
    Use AWS PrivateLink to establish a private connection between your StreamNative Cloud environment and Snowflake Horizon Catalog.

    Follow the Snowflake documentation to configure PrivateLink:
    [Snowflake PrivateLink on AWS](https://docs.snowflake.com/en/user-guide/admin-security-privatelink).
  </Tab>

  <Tab title="GCP">
    Use Google Cloud Private Service Connect to establish a private connection between your StreamNative Cloud environment and Snowflake Horizon Catalog.

    Follow the Snowflake documentation to configure Private Service Connect:
    [Snowflake Private Service Connect on GCP](https://docs.snowflake.com/en/user-guide/private-service-connect-google).
  </Tab>

  <Tab title="Azure">
    Use Azure Private Link to establish a private connection between your StreamNative Cloud environment and Snowflake Horizon Catalog.

    Follow the Snowflake documentation to configure Private Link:
    [Snowflake Private Link on Azure](https://docs.snowflake.com/en/user-guide/privatelink-azure).
  </Tab>
</Tabs>

## Update the catalog URI

After enabling private connectivity, update the catalog URI in StreamNative Cloud to use the PrivateLink hostname. The catalog URI is configured during [catalog registration](/cloud/lakehouse/catalogs/register-catalog) and can be updated through [Manage Catalogs](/cloud/lakehouse/catalogs/manage-catalogs).

Change the URI from the public format:

```
https://<account-locator>.<catalog-region>.snowflakecomputing.com/polaris/api/catalog
```

to the PrivateLink format:

```
https://<account-locator>.privatelink.snowflakecomputing.com/polaris/api/catalog
```

<Note>
  The exact private endpoint hostname may vary by cloud provider. Refer to the Snowflake PrivateLink documentation for your cloud provider to determine the correct hostname.
</Note>


# Private Networking for Snowflake Open Catalog
Source: https://docs.streamnative.io/cloud/lakehouse/catalogs/private-networking/snowflake-open-catalog



This guide describes how to configure private network connections between StreamNative Cloud and Snowflake Open Catalog. Private connectivity ensures that traffic between your StreamNative BYOC cluster and Snowflake Open Catalog does not traverse the public internet.

Snowflake Open Catalog uses the same private connectivity infrastructure as Snowflake Horizon Catalog. If you use Horizon Catalog, see [Private Networking for Snowflake Horizon Catalog](/cloud/lakehouse/catalogs/private-networking/snowflake-horizon-catalog).

The following diagram shows the network path between your StreamNative BYOC cluster and Snowflake Open Catalog over private connectivity.

```mermaid theme={null}
flowchart TB
    subgraph BYOC["StreamNative BYOC VPC"]
        Cluster["BYOC Cluster"]
        Endpoint["Private Endpoint"]
    end

    Snowflake["Snowflake Open Catalog"]

    Cluster -->|"catalog API requests"| Endpoint --> Snowflake

    classDef byoc fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
    classDef edge fill:#fde68a,stroke:#b45309,color:#451a03
    classDef ext  fill:#ede9fe,stroke:#7c3aed,color:#4c1d95
    class Cluster byoc
    class Endpoint edge
    class Snowflake ext
```

## Prerequisites

* A StreamNative BYOC cluster deployed on AWS, GCP, or Azure.
* A Snowflake Open Catalog account in the same cloud provider and region as your StreamNative BYOC cluster.
* A prepared Snowflake Open Catalog. See [Prepare Lakehouse Catalogs](/cloud/lakehouse/prepare-lakehouse-catalogs) for the cloud-specific setup guides.
* Storage private connectivity configured (if applicable). See [Private Networking for Catalog Integration](/cloud/lakehouse/catalogs/private-networking/overview) for storage connectivity details.

## Configure private connectivity

Configure private connectivity to Snowflake Open Catalog based on the cloud provider where your StreamNative BYOC cluster is deployed.

<Tabs>
  <Tab title="AWS">
    Use AWS PrivateLink to establish a private connection between your StreamNative Cloud environment and Snowflake Open Catalog.

    Follow the Snowflake documentation to configure PrivateLink:
    [Snowflake PrivateLink on AWS](https://docs.snowflake.com/en/user-guide/admin-security-privatelink).
  </Tab>

  <Tab title="GCP">
    Use Google Cloud Private Service Connect to establish a private connection between your StreamNative Cloud environment and Snowflake Open Catalog.

    Follow the Snowflake documentation to configure Private Service Connect:
    [Snowflake Private Service Connect on GCP](https://docs.snowflake.com/en/user-guide/private-service-connect-google).
  </Tab>

  <Tab title="Azure">
    Use Azure Private Link to establish a private connection between your StreamNative Cloud environment and Snowflake Open Catalog.

    Follow the Snowflake documentation to configure Private Link:
    [Snowflake Private Link on Azure](https://docs.snowflake.com/en/user-guide/privatelink-azure).
  </Tab>
</Tabs>

## Update the catalog URI

After enabling private connectivity, update the catalog URI in StreamNative Cloud to use the PrivateLink hostname. The catalog URI is configured during [catalog registration](/cloud/lakehouse/catalogs/register-catalog) and can be updated through [Manage Catalogs](/cloud/lakehouse/catalogs/manage-catalogs).

Change the URI from the public format:

```
https://<account-locator>.<catalog-region>.snowflakecomputing.com/polaris/api/catalog
```

to the PrivateLink format:

```
https://<account-locator>.privatelink.snowflakecomputing.com/polaris/api/catalog
```

<Note>
  The exact private endpoint hostname may vary by cloud provider. Refer to the Snowflake PrivateLink documentation for your cloud provider to determine the correct hostname.
</Note>


# Register Catalog
Source: https://docs.streamnative.io/cloud/lakehouse/catalogs/register-catalog



To register a catalog in StreamNative Cloud, navigate to the **Catalogs** page and click **Register Catalog**.

<img alt="StreamNative Catalogs" />

On the **Register Catalog** form, provide the required details based on the selected Catalog Provider.

### Databricks Unity Catalog (Delta Lake and Iceberg)

* **Name** – Enter a unique name for the catalog in StreamNative Cloud.
* **Catalog Provider** – Select the catalog provider where you want to ingest data.
* **Catalog Name** – Enter the name of the catalog that exists in Databricks Unity Catalog.
* **URI** – Provide the URI of the catalog.
* **Authentication Type** – Select the appropriate authentication method for connecting to the catalog provider.

### Snowflake Open Catalog

* **Name** – Enter a unique name for the catalog in StreamNative Cloud.
* **Catalog Provider** – Select the catalog provider where you want to ingest data.
* **Warehouse** – Enter the name of the warehouse that exists in Snowflake Open Catalog.
* **URI** – Provide the URI of the catalog.
* **Authentication Type** – Select the appropriate authentication method for connecting to the catalog provider.

### Amazon S3 Tables

* **Name** – Enter a unique name for the catalog in StreamNative Cloud.
* **Catalog Provider** – Select the catalog provider where you want to ingest data.
* **S3 Table Bucket** – Enter the ARN of the S3 Table bucket.

### Google BigLake metastore

The Google BigLake metastore is an Iceberg REST catalog for Google Cloud that integrates with BigQuery, so streamed tables are queryable from BigQuery without an extra import step.

* **Name** – Enter a unique name for the catalog in StreamNative Cloud.
* **Catalog Provider** – Select Google BigLake as the catalog provider.
* **Google Project** – Enter the name of the Google project.
* **Warehouse** – Enter the name of the Google Cloud Storage Warehouse.

### Snowflake Horizon (Polaris) Catalog

The Snowflake Horizon Catalog is an Iceberg REST catalog hosted on Snowflake. Access is governed by Snowflake roles, so existing Snowflake RBAC policies apply to streamed tables.

* **Name** – Enter a unique name for the catalog in StreamNative Cloud.
* **Catalog Provider** – Select Snowflake Horizon as the catalog provider.
* **Account** – Enter the Snowflake account identifier hosting the Horizon catalog.
* **Warehouse** – Enter the name of the warehouse that exists in Snowflake Horizon.
* **URI** – Provide the URI of the Polaris REST catalog endpoint.
* **Authentication Type** – Select the appropriate authentication method (for example, OAuth2 client credentials) for connecting to Snowflake Horizon.

## Register multiple catalogs per cluster

A single StreamNative cluster can be associated with multiple registered catalogs. This lets you route different topics to different catalogs — for example, landing some topics into BigLake for BigQuery analytics and others into Snowflake Horizon for governed Snowflake access — all from one Pulsar or Kafka cluster. Repeat the registration steps above for each catalog you want to make available to the cluster.

<img alt="StreamNative Catalogs" />


# Enable Lakehouse Table
Source: https://docs.streamnative.io/cloud/lakehouse/enable-lakehouse-integration



You can enable the Lakehouse Table for a cluster, a namespace, or a single topic from the StreamNative Cloud Console. Once enabled, topic data is delivered to the configured external lakehouse catalog automatically.

> Currently, only the **Deliver to External Table** mode is supported. The **Expose Internal Table** option is coming soon.

## Prerequisites

* A cluster (Pulsar or Kafka) running on a profile that supports Lakehouse Table delivery. See [Lakehouse Table Overview](/cloud/lakehouse/lakehouse-table-overview) for the supported cluster types and profiles.
* A registered catalog in your organization. If you have not registered a catalog yet, see [Register Catalog](/cloud/lakehouse/catalogs/register-catalog).

## Enable Lakehouse Table at the cluster level

You can enable the Lakehouse Table at the cluster level either when you create a new cluster or for an existing cluster.

### Enable when creating a new cluster

When you create a new cluster, the cluster wizard includes a **Lakehouse Table** configuration step.

<Frame>
  <img alt="Lakehouse Table step in the cluster creation wizard" />
</Frame>

1. Toggle on **Enable Lakehouse Table**.

2. Select a **Catalog Provider**. StreamNative Cloud supports the following providers:
   * Databricks Unity Catalog for Iceberg (Iceberg)
   * Databricks Unity Catalog for Delta Lake (Delta Lake)
   * Snowflake Open Catalog (Iceberg)
   * Snowflake Horizon Catalog (Iceberg)
   * Amazon S3Table (Iceberg)
   * Google BigLake (Iceberg) -- can only be configured **after** the cluster is created. Skip the Lakehouse Table step at cluster creation and enable it later from the existing cluster page.

3. Select the target **Catalog** that belongs to the catalog provider you selected. If the catalog you need is not registered yet, click **Register new catalog** to open the registration page. See [Register Catalog](/cloud/lakehouse/catalogs/register-catalog).

   <Frame>
     <img alt="Select a registered catalog or register a new one" />
   </Frame>

4. (Optional) Enable **Apply lakehouse setting to all topics** to automatically deliver every existing and new topic in the cluster to the lakehouse table. If this option is disabled, you can enable Lakehouse delivery later at the namespace or topic level.

### Enable for an existing cluster

For an existing Pulsar cluster (latency-optimized profile or cost-optimized profile), open the cluster page and click **Enable Lakehouse Table**.

<Frame>
  <img alt="Enable Lakehouse Table button on the cluster page" />
</Frame>

In the dialog, select a target catalog from the dropdown. If the catalog is not registered yet, click **Register new catalog** to register one.

<Frame>
  <img alt="Enable Lakehouse Table dialog with catalog dropdown" />
</Frame>

After you confirm, every topic in the cluster begins delivering data to the lakehouse table automatically.

## Enable Lakehouse Table at the namespace level

You can also enable the Lakehouse Table at the namespace level. Open the namespace and click **Enable Lakehouse Table**, then select a registered target catalog. If the catalog you need is not registered, click **Register new catalog** to register one.

> Namespace-level enablement is supported only on Pulsar clusters (latency-optimized and cost-optimized profiles). Kafka clusters do not support namespace-level enablement; for Kafka clusters, enable Lakehouse Table at the cluster level instead.

<Frame>
  <img alt="Enable Lakehouse Table at the namespace level" />
</Frame>

After you confirm, every topic in the namespace begins delivering data to the lakehouse table automatically.

## Enable Lakehouse Table at the topic level

You can enable the Lakehouse Table for a single topic. Open the topic and click **Enable Lakehouse Table**, then select a registered target catalog. If the catalog you need is not registered, click **Register new catalog** to register one.

> Topic-level enablement is supported only on Pulsar clusters (latency-optimized and cost-optimized profiles). Kafka clusters do not support topic-level enablement; for Kafka clusters, enable Lakehouse Table at the cluster level instead.

<Frame>
  <img alt="Enable Lakehouse Table at the topic level" />
</Frame>

After you confirm, the topic data is delivered to the lakehouse table automatically.

## Configuration override priority

When the same setting is configured at multiple levels, the most specific level wins:

```
Topic settings
    ↓ (override)
Namespace settings
    ↓ (override)
Cluster settings
```

For example, if Lakehouse delivery is enabled at the cluster level but disabled at a specific topic, that topic's data is not delivered.

### Per-namespace and per-topic catalog selection

Different namespaces in the same cluster can deliver data to **different catalogs**, and individual topics within a namespace can also override the namespace's catalog and deliver to a different one. For example, namespace `analytics/finance` can be configured to use a Snowflake Open Catalog, namespace `analytics/marketing` in the same cluster can use a Databricks Unity Catalog, and a single topic `analytics/marketing/realtime-events` inside `analytics/marketing` can be redirected to an Amazon S3Tables catalog. To assign different catalogs, enable the Lakehouse Table separately on each namespace or topic and select the desired catalog.

### Catalog override order

The catalog used for a topic is resolved in the same most-specific-wins order:

```
Topic catalog
    ↓ (overrides)
Namespace catalog
    ↓ (overrides)
Cluster catalog
```

If a catalog is set on the topic, that catalog is used regardless of the namespace or cluster catalog. If only the namespace catalog is set, every topic in that namespace uses the namespace's catalog. If neither is set, the cluster catalog is used.

## Next Steps

* Monitor data delivery progress for a topic from the Cloud Console: [Lakehouse Observability -- Monitor data delivery progress in the Cloud Console](/cloud/lakehouse/lakehouse-observability#monitor-data-delivery-progress-in-the-cloud-console).
* See how Pulsar topic names are mapped to lakehouse namespace and table names: [Lakehouse Table Overview -- Topic to lakehouse identifier mapping](/cloud/lakehouse/lakehouse-table-overview#topic-to-lakehouse-identifier-mapping).
* Explore features:
  * [Schema Evolution](/cloud/lakehouse/features/schema-evolution)
  * [Variant Type](/cloud/lakehouse/features/variant-type)
  * [Partition Key](/cloud/lakehouse/features/iceberg-partition-key)
  * [Upsert](/cloud/lakehouse/features/iceberg-upsert)
  * [Persist Key](/cloud/lakehouse/features/persist-key)
  * [Persist Extra Metadata](/cloud/lakehouse/features/persist-extra-metadata)


# Overview - External Tables
Source: https://docs.streamnative.io/cloud/lakehouse/external-tables/external-tables-overview



**External Tables**

External Tables are tables backed by StreamNative topics but represented in open table formats, such as Apache Iceberg or Delta Lake, with both data and metadata stored directly in user-managed object storage.

**Key Characteristics**

* **Open Table Formats** -

  Data is written as Parquet files and managed using Iceberg or Delta metadata logs.

* **Object Storage–Backed** -
  Tables reside in S3, GCS, Azure Blob, or other object storage systems.

Externally Managed
StreamNative Cloud materializes the data, but table lifecycle, optimization, schema governance, and cataloging are typically managed by a Lakehouse vendor, such as:

* Databricks Unity Catalog
* Snowflake Open Catalog
* Amazon S3 Tables

**High Interoperability**

External Tables are fully queryable by external compute platforms, BI tools, and SQL engines.

**When to Use External Tables**

* You want deep integration with an existing Lakehouse platform.
* You require open, interoperable formats for analytics, AI/ML, or downstream workloads.
* You prefer to manage optimization (compaction, vacuum, retention) in systems like Databricks, Snowflake, or Polaris.


# Partition Key
Source: https://docs.streamnative.io/cloud/lakehouse/features/iceberg-partition-key



Partition keys control how data is organized into partitions within the Iceberg table. Partitioning improves query performance by enabling partition pruning.

`partition.key` is a [dynamic configuration](../dynamic-configuration) key that takes effect **only at the topic level**. Setting it at the cluster or namespace level has no effect.

> **Cluster-name prefix:** All dynamic configuration keys must be prefixed with the cluster name (for example, `<cluster-name>.partition.key`). The cluster name is the value of `clusterName` in `conf/broker.conf` -- see [Finding the Cluster Name](../dynamic-configuration#finding-the-cluster-name). The examples below use `private-cloud` as the cluster name; replace it with the name of your cluster.

> **Cardinality limit:** Keep the **total number of partition values across all levels under 10** (the cardinality of `key1 × key2 × ... × keyN` should not exceed 10). If the partitioning would produce more than 10 distinct partition values, use the `bucket[N]` transform to bound it. Excessive partitions cause many small files, which degrade write throughput and query performance.

## Configuration Format

The partition key is specified as a JSON array. Each element has three fields:

| Field          | Required | Description                                                   |
| -------------- | -------- | ------------------------------------------------------------- |
| `sourceColumn` | Yes      | The field name from the topic schema                          |
| `transform`    | No       | Iceberg partition transform function. Defaults to `identity`. |
| `targetName`   | No       | Custom name for the transformed partition column              |

### Supported Iceberg Transforms

| Transform     | Description                      |
| ------------- | -------------------------------- |
| `identity`    | Use the field value as-is        |
| `bucket[N]`   | Hash into N buckets              |
| `truncate[N]` | Truncate strings to N characters |
| `year`        | Extract year from timestamp      |
| `month`       | Extract month from timestamp     |
| `day`         | Extract day from timestamp       |
| `hour`        | Extract hour from timestamp      |

For full semantics, see the [Iceberg partition transforms specification](https://iceberg.apache.org/spec/#partition-transforms).

## Apply at Topic Level

```bash theme={null}
bin/pulsar-admin topics update-properties \
  -p private-cloud.partition.key='[{\"sourceColumn\":\"<field>\",\"transform\":\"<transform>\",\"targetName\":\"<name>\"}]' \
  persistent://<tenant>/<namespace>/<topic>
```

> Setting `partition.key` at the cluster or namespace level has no effect. Apply it on the topic only.

## Example

Configure two partition keys on a topic:

* `timestamp` -- bucketed by hour, named `ts_hour`
* `address` -- truncated to 7 characters, named `t_address`

```bash theme={null}
bin/pulsar-admin topics update-properties \
  -p private-cloud.partition.key="[{\"sourceColumn\":\"timestamp\",\"transform\":\"hour\",\"targetName\":\"ts_hour\"},{\"sourceColumn\":\"address\",\"transform\":\"truncate[7]\",\"targetName\":\"t_address\"}]" \
  persistent://public/default/events
```

## Important Notes

1. The `sourceColumn` value must reference a field that exists in the topic schema.
2. The `targetName` is the name produced after applying the transform; it does not need to exist in the topic schema.
3. The JSON value must be a valid JSON array. When passing it on the shell, escape inner double quotes (`\"`).
4. If the JSON cannot be parsed, the system falls back to a non-partitioned table.
5. **Keep the total cardinality of partition values under 10.** If a column has high cardinality, wrap it with `bucket[N]` to bound the number of partitions (for example, `{"sourceColumn":"userId","transform":"bucket[8]"}`). High partition counts produce many small files and degrade performance.

## Related

* [Dynamic Configuration Guide](../dynamic-configuration) -- Cluster-name prefix, override priority, and apply procedure
* [Upsert](/cloud/lakehouse/features/iceberg-upsert) -- Combining partition keys with upsert has compatibility constraints


# Upsert
Source: https://docs.streamnative.io/cloud/lakehouse/features/iceberg-upsert



Upsert mode enables deduplication of records based on a primary key. When multiple records with the same key arrive, only the latest value is retained in the lakehouse table.

`upsert.mode.enabled` and `identifier.fields` are [dynamic configuration](../dynamic-configuration) keys.

> **Cluster-name prefix:** All dynamic configuration keys must be prefixed with the cluster name (for example, `<cluster-name>.upsert.mode.enabled`). The cluster name is the value of `clusterName` in `conf/broker.conf` -- see [Finding the Cluster Name](../dynamic-configuration#finding-the-cluster-name). The examples below use `private-cloud` as the cluster name; replace it with the name of your cluster.

## Configuration Keys

| Key                                  | Scope                       | Description                                                                                                           | Default |
| ------------------------------------ | --------------------------- | --------------------------------------------------------------------------------------------------------------------- | ------- |
| `<cluster-name>.upsert.mode.enabled` | Cluster / Namespace / Topic | Enable upsert mode (`true` / `false`). More-specific scopes override broader ones.                                    | `false` |
| `<cluster-name>.identifier.fields`   | Topic only                  | Comma-separated list of fields used as the primary key. Setting this at the cluster or namespace level has no effect. | --      |

## How Upsert Works

StreamNative Ursa implements upsert using **Iceberg equal-delete** files. For each upsert write, the compaction service emits an equal-delete entry that matches the previous record by the configured identifier fields, followed by the new record. Readers reconcile the delete and the new value at query time.

### Catalog Compatibility

Because upsert depends on equal-delete files, the underlying catalog and query engine must support reading Iceberg equal deletes. Catalogs differ in their delete-file support:

| Catalog                                      | Equal Delete              | Upsert Support     |
| -------------------------------------------- | ------------------------- | ------------------ |
| Snowflake Open Catalog (Polaris)             | No (position delete only) | Not supported      |
| Snowflake Horizon Catalog                    | No (position delete only) | Not supported      |
| Databricks Unity Catalog (Managed Iceberg)   | No (position delete only) | Not supported      |
| AWS S3Table                                  | Yes                       | Supported          |
| Google BigLake                               | Under confirmation        | Under confirmation |
| Iceberg Hadoop catalog (no external catalog) | Yes                       | Supported          |

If you need upsert with a catalog that only supports position deletes, the recommended approach is to use the Hadoop catalog or AWS S3Table for the affected tables.

## Apply at Cluster Level

Cluster-level properties are stored in the `sn/system` namespace. Cluster-level upsert applies as the default for every namespace and topic that does not override it.

```bash theme={null}
bin/pulsar-admin namespaces set-properties \
  -p private-cloud.cluster.upsert.mode.enabled=true \
  sn/system
```

## Apply at Namespace Level

```bash theme={null}
bin/pulsar-admin namespaces set-properties \
  -p private-cloud.upsert.mode.enabled=true \
  <tenant>/<namespace>
```

## Apply at Topic Level

`identifier.fields` must be set on the topic; setting `upsert.mode.enabled` on the topic is also valid and overrides any namespace or cluster default.

```bash theme={null}
bin/pulsar-admin topics update-properties \
  -p private-cloud.upsert.mode.enabled=true \
  -p private-cloud.identifier.fields=<field1>,<field2> \
  persistent://<tenant>/<namespace>/<topic>
```

## Example

Enable upsert on `persistent://public/default/users` with `userId` and `email` as the primary key:

```bash theme={null}
bin/pulsar-admin topics update-properties \
  -p private-cloud.upsert.mode.enabled=true \
  -p private-cloud.identifier.fields=userId,email \
  persistent://public/default/users
```

## Requirements

* Identifier fields **must exist** in the topic schema.
* Identifier fields **must be marked as `required`** in the schema (not nullable).
* The catalog and query engine must support reading Iceberg equal-delete files. See [Catalog Compatibility](#catalog-compatibility).

## Commit Behavior

* **Append-only writes** (default) are batch-committed: multiple Parquet files are grouped and committed to the catalog in a single commit.
* **Upsert writes** are committed one-by-one to ensure correct data ordering. This may slightly reduce throughput compared to append-only mode.

## Limitations

* When upsert is combined with a [partition key](/cloud/lakehouse/features/iceberg-partition-key), identifier fields only deduplicate within the same partition. For example, if the table has `partition.key=region` and `identifier.fields=userId`, two records with the same `userId` but different `region` values are both kept.
* Upsert is supported only for External Tables (SDT).

## Related

* [Dynamic Configuration Guide](../dynamic-configuration) -- Cluster-name prefix, override priority, and apply procedure
* [Partition Key](/cloud/lakehouse/features/iceberg-partition-key) -- Combining partition keys with upsert has compatibility constraints


# Persist Extra Metadata
Source: https://docs.streamnative.io/cloud/lakehouse/features/persist-extra-metadata



The **Persist Extra Metadata** feature injects a `__meta` column into the lakehouse table that contains additional message metadata captured at write time. Use it when downstream queries need access to information that is part of the message envelope but not part of the message body -- for example, the message offset, publish time, event time, or producer-side properties.

## What Gets Persisted

When the feature is enabled, every record written to the lakehouse table includes a `__meta` column populated with the following fields:

| Field             | Description                                                                        |
| ----------------- | ---------------------------------------------------------------------------------- |
| `__messageOffset` | The original Pulsar / Kafka message ID or offset                                   |
| `__publishTime`   | Pulsar publish timestamp (epoch ms)                                                |
| `__eventTime`     | Pulsar event time (epoch ms), if set by the producer                               |
| `__schemaVersion` | Numeric schema version of the source message                                       |
| `__properties`    | Producer-supplied key/value properties (Pulsar message properties / Kafka headers) |

The column is stored as an Iceberg / Delta **Variant** value. This keeps the schema stable when properties evolve (new keys, removed keys) and lets downstream engines extract individual fields with standard Variant accessors (for example, `__meta.__publishTime` in Spark SQL).

## Configuration

| Property               | Default | Description                                                                                           |
| ---------------------- | ------- | ----------------------------------------------------------------------------------------------------- |
| `persistExtraMetadata` | `false` | Master switch. When `true`, the `__meta` column is added to the table and populated for every record. |

### Required Companion Settings

Because `__meta` is a Variant column, the feature shares the [Variant type](/cloud/lakehouse/features/variant-type) prerequisites:

| Property                   | Default | Required When                                                               |
| -------------------------- | ------- | --------------------------------------------------------------------------- |
| `variantTypeEnabled`       | `false` | Always (master switch for Variant support)                                  |
| `tableEvolveSchemaEnabled` | `true`  | Always (the column is added by schema evolution)                            |
| `allowIcebergV3`           | `false` | Required for Iceberg (Variant is an Iceberg V3 feature). Ignored for Delta. |

> **Important:** Variant support is gated by a feature flag. Contact the StreamNative Support Team to enable it before turning on `persistExtraMetadata`.

### Iceberg

Add the following to the compaction service `custom` config:

```yaml theme={null}
persistExtraMetadata: "true"
variantTypeEnabled: "true"
tableEvolveSchemaEnabled: "true"   # default; only override if previously disabled
allowIcebergV3: "true"
```

> **Downstream query engine compatibility:** When `allowIcebergV3` is enabled, your readers (Spark, Trino, Athena, etc.) must support Iceberg V3 to read tables that contain Variant columns. See [Variant Type](/cloud/lakehouse/features/variant-type#iceberg) for details.

### Delta Lake

```yaml theme={null}
persistExtraMetadata: "true"
variantTypeEnabled: "true"
tableEvolveSchemaEnabled: "true"
```

Delta does not require `allowIcebergV3`.

## Querying the Metadata

Once enabled, the `__meta` column appears in the lakehouse table alongside the user-defined fields. Examples:

**Spark SQL (Iceberg):**

```sql theme={null}
SELECT __meta:__publishTime AS publish_time,
       __meta:__messageOffset AS offset,
       __meta:__properties:trace_id AS trace_id,
       *
FROM iceberg_catalog.namespace.events
WHERE __meta:__publishTime >= 1700000000000
LIMIT 10;
```

**Spark SQL (Delta):**

```sql theme={null}
SELECT variant_get(__meta, '$.__publishTime', 'long') AS publish_time,
       variant_get(__meta, '$.__messageOffset', 'string') AS offset,
       *
FROM delta.`s3://bucket/path/events`
LIMIT 10;
```

The exact Variant accessor syntax depends on your engine version; consult the engine's Variant documentation for the canonical form.

## Behavior Notes

* **Adding metadata to existing tables.** Because the column is added through schema evolution (`tableEvolveSchemaEnabled=true`), enabling the flag on a topic that already has a lakehouse table appends the `__meta` column on the next compaction. Records written before the flag was enabled will have a `null` value in the column.
* **Disabling the feature.** Setting `persistExtraMetadata` back to `false` stops new records from receiving metadata, but the column itself is not removed. Older rows retain their values.
* **Performance impact.** The `__meta` column is small per-row but increases storage and write throughput slightly. The Variant encoding is efficient and supports predicate pushdown when the engine extracts a specific field.

## Related

* [Variant Type](/cloud/lakehouse/features/variant-type) -- Prerequisite feature for `persistExtraMetadata`
* [Schema Evolution](/cloud/lakehouse/features/schema-evolution) -- The mechanism by which the `__meta` column is added to the table
* [Persist Key](/cloud/lakehouse/features/persist-key) -- Companion feature that persists the message key as a separate column


# Persist Key
Source: https://docs.streamnative.io/cloud/lakehouse/features/persist-key



The **Persist Key** feature injects a `__key` column into the lakehouse table that contains the original message key from the source topic. Use it when downstream queries need access to the partition key produced by the application -- for example, joining the lakehouse table with another dataset on the producer's key, computing per-key aggregations, or auditing message routing.

## What Gets Persisted

When the feature is enabled, every record written to the lakehouse table includes a `__key` column populated with the source message's key:

| Source          | Key value persisted                                                                           |
| --------------- | --------------------------------------------------------------------------------------------- |
| Pulsar producer | The Pulsar message key (`MessageBuilder.key(...)`), or `null` if the producer did not set one |
| Kafka producer  | The Kafka record key bytes, or `null` if the producer did not set one                         |

The column is added to the table by schema evolution and stored as a `binary` value, preserving the exact bytes the producer sent.

## Configuration

| Property     | Default | Description                                                                                          |
| ------------ | ------- | ---------------------------------------------------------------------------------------------------- |
| `persistKey` | `false` | Master switch. When `true`, the `__key` column is added to the table and populated for every record. |

### Required Companion Settings

The `__key` column is added through schema evolution, so schema evolution must remain enabled:

| Property                   | Default | Description                                                                               |
| -------------------------- | ------- | ----------------------------------------------------------------------------------------- |
| `tableEvolveSchemaEnabled` | `true`  | Required (the column is added by schema evolution). Only override if previously disabled. |

`persistKey` does **not** require Variant support, Iceberg V3, or any other feature flag.

### Configuration

Add the following to the compaction service `custom` config:

```yaml theme={null}
persistKey: "true"
tableEvolveSchemaEnabled: "true"   # default; only override if previously disabled
```

The same configuration applies to both Iceberg and Delta Lake.

## Querying the Key

Once enabled, the `__key` column appears in the lakehouse table alongside the user-defined fields. Examples:

**Spark SQL:**

```sql theme={null}
SELECT CAST(__key AS STRING) AS key,
       *
FROM iceberg_catalog.namespace.events
WHERE CAST(__key AS STRING) = 'user-42'
LIMIT 10;
```

```sql theme={null}
-- Per-key counts
SELECT CAST(__key AS STRING) AS key, COUNT(*) AS message_count
FROM iceberg_catalog.namespace.events
GROUP BY CAST(__key AS STRING)
ORDER BY message_count DESC;
```

If your producer keys are UTF-8 strings, cast the column to `STRING` for readability. For binary keys, query the column directly as `BINARY`.

## Behavior Notes

* **Adding the key column to existing tables.** Because the column is added through schema evolution (`tableEvolveSchemaEnabled=true`), enabling the flag on a topic that already has a lakehouse table appends the `__key` column on the next compaction. Records written before the flag was enabled will have a `null` value in the column.
* **Disabling the feature.** Setting `persistKey` back to `false` stops new records from receiving the key, but the column itself is not removed. Older rows retain their values.
* **Null keys.** Messages without a key are written with `null` in the `__key` column.
* **Combining with upsert.** The `__key` column is independent of [identifier fields](/cloud/lakehouse/features/iceberg-upsert) used for upsert. It records the producer-side key as-is and is not used for deduplication.

## Related

* [Persist Extra Metadata](/cloud/lakehouse/features/persist-extra-metadata) -- Companion feature that persists message envelope metadata (offset, publish time, properties) as a Variant column
* [Schema Evolution](/cloud/lakehouse/features/schema-evolution) -- The mechanism by which the `__key` column is added to the table
* [Upsert](/cloud/lakehouse/features/iceberg-upsert) -- For deduplication semantics on a primary key, which is a different concern from persisting the source message key


# Schema Evolution
Source: https://docs.streamnative.io/cloud/lakehouse/features/schema-evolution



Schema evolution is enabled by default (`tableEvolveSchemaEnabled=true`). When a producer changes the schema of a topic, the lakehouse table is automatically updated to match.

> **Recommendation:** For topics that have Lakehouse integration enabled, set the Pulsar schema compatibility strategy to **`BACKWARD_TRANSITIVE`**. This guarantees that every reader (including downstream Iceberg / Delta consumers) can read data written with all previous schema versions, which aligns with how the lakehouse table is evolved. See the [Pulsar schema compatibility documentation](https://pulsar.apache.org/docs/2.10.x/schema-evolution-compatibility/#backward-and-backward_transitive) for details on `BACKWARD` vs `BACKWARD_TRANSITIVE`.
>
> Apply with `pulsar-admin` at the namespace or topic level:
>
> ```bash theme={null}
> # Namespace level
> bin/pulsar-admin namespaces set-schema-compatibility-strategy \
>   --compatibility BACKWARD_TRANSITIVE \
>   <tenant>/<namespace>
>
> # Topic level
> bin/pulsar-admin topicPolicies set-schema-compatibility-strategy \
>   --strategy BACKWARD_TRANSITIVE \
>   <tenant>/<namespace>/<topic-name>
> ```

## How It Works

Schema evolution is triggered automatically during the compaction process when the compaction service encounters messages with a schema version newer than what the lakehouse table currently has.

### Step 1: Detection

When processing messages, the compaction service extracts the schema version from each message. If the version is newer than the latest version recorded in the table's schema mapping, evolution is triggered.

### Step 2: Retrieve All Schema Versions

The compaction service retrieves all schema versions from the schema registry (Pulsar or Confluent) up to the current version. This ensures that intermediate schema versions are not skipped.

### Step 3: Convert Schemas

Each source schema (Avro, JSON, ProtobufNative(Pulsar) or Protobuf(Kafka)) is converted to the target table format:

* **Avro** is converted directly to Iceberg Schema or Delta StructType
* **JSON** is first converted to Avro, then to the target format (Pulsar internally represents JSON schemas using Avro)
* **ProtobufNative(Pulsar)** is converted via the Protobuf descriptor to Avro, then to the target format
* **Protobuf(Kafka)** is converted via the Protobuf descriptor to Avro, then to the target format

### Step 4: Apply Changes

Schema changes are applied to the table iteratively, version by version in ascending order. Each version is processed as a transaction.

For **Iceberg** tables, the following operations are supported:

* **Add columns** -- new fields are always added as optional for backward compatibility
* **Delete columns** -- field is removed from the schema (or made optional in soft-delete mode)
* **Type promotion** -- `int` to `long`, `float` to `double`, and other compatible promotions
* **Nullability changes** -- required fields can be made optional (reverse is not supported)
* **Nested struct evolution** -- the same rules apply recursively to nested fields

For **Delta** tables:

* Column Mapping mode (`NAME`) is automatically enabled to support safe renames and deletes
* Each column receives a unique physical ID for safe schema evolution
* Schema changes are committed as Delta transactions

### Step 5: Record Schema Mapping

After each successful evolution, a mapping from the source schema version to the table's internal schema ID is recorded as a table property (`streamnative.schema.mapping`). This prevents re-processing the same version.

## Supported Operations

| Operation                                                            | Iceberg       | Delta         |
| -------------------------------------------------------------------- | ------------- | ------------- |
| Add optional columns                                                 | Supported     | Supported     |
| Delete columns                                                       | Supported     | Supported     |
| Type promotion (`int` -> `long`, `float` -> `double`)                | Supported     | Supported     |
| Required -> Optional                                                 | Supported     | Supported     |
| Optional -> Required                                                 | Not supported | Not supported |
| Nested struct evolution                                              | Supported     | Supported     |
| Type category changes (`struct` \<-> `list`, `primitive` \<-> `map`) | Not supported | Not supported |

## Limitations

* **Adding required fields:** New fields added to existing tables are automatically converted to optional. Required fields can only be defined at table creation time.
* **Type category changes:** Changing a field from one type category to another (e.g., struct to list, primitive to map) is not supported.
* **Backward incompatibility:** If the table schema has been evolved beyond the message's schema version, the message cannot be processed. Update the producer to use the latest schema.
* **Soft-delete mode:** Enable `schema.evolution.soft-delete.enabled=true` to make deleted fields optional instead of removing them entirely. This preserves backward compatibility for readers.

## Configuration

| Property                               | Description                                      | Default |
| -------------------------------------- | ------------------------------------------------ | ------- |
| `tableEvolveSchemaEnabled`             | Enable automatic schema evolution                | `true`  |
| `schema.evolution.soft-delete.enabled` | Make deleted fields optional instead of removing | `false` |
| `dlt.suffix`                           | Suffix appended to Dead Letter Table name        | `_dlt`  |

## Dead Letter Table (DLT)

When a message cannot be successfully processed into the lakehouse table, it is routed to a **Dead Letter Table (DLT)** instead of failing the entire compaction job. This allows the pipeline to continue processing valid messages while preserving failed records for investigation.

### When Are Messages Routed to DLT?

Messages are routed to DLT in the following scenarios:

| Scenario                       | Example                                                                                                                                                    |
| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Schema incompatibility**     | 1) Schema evolution is disabled and the record schema does not match the table schema; 2) New topic schema is incompatible with the Lakehouse table schema |
| **Unsupported schema changes** | Evolving a non-Variant field to Variant, or a Variant field to another type                                                                                |
| **Type conversion errors**     | Record cannot be serialized to the target lakehouse format (Iceberg/Delta/Parquet)                                                                         |
| **Null values**                | Null payload/record values from Kafka entries                                                                                                              |
| **Deserialization errors**     | Failures deserializing Pulsar or Kafka source data                                                                                                         |
| **Parsing failures**           | Schema parsing errors for Pulsar message format                                                                                                            |

> **Note:** Fatal errors (such as catalog connectivity issues, permission errors, or infrastructure failures) are **not** routed to DLT -- they cause the compaction task to fail and retry.

### DLT Table Naming

The DLT table name is derived from the original topic name by appending the configurable DLT suffix (default: `_dlt`).

**Format:**

```
<original-topic-local-name><dlt-suffix>
```

**Example:**

| Original Topic                                    | DLT Table                                                 |
| ------------------------------------------------- | --------------------------------------------------------- |
| `persistent://my-tenant/my-namespace/user-events` | `user-events_dlt` (in namespace `my-tenant/my-namespace`) |
| `persistent://public/default/orders`              | `orders_dlt` (in namespace `public/default`)              |

For Iceberg tables, the DLT table is created in the same namespace as the original table. For Delta tables, the DLT topic is `<tenant>/<namespace>/<topic-name>_dlt`.

### DLT Schema

Both Iceberg and Delta DLT tables use a consistent 3-field schema:

| Field           | Type   | Nullable | Description                                        |
| --------------- | ------ | -------- | -------------------------------------------------- |
| `messageId`     | String | No       | The original Pulsar or Kafka message ID            |
| `payload`       | String | Yes      | Base64-encoded original message payload            |
| `failureReason` | String | Yes      | The error message explaining why the record failed |

This allows you to:

* Identify which messages failed and why
* Replay or reprocess the failed messages after fixing the underlying issue
* Debug schema or serialization problems

### Configuring the DLT Suffix

The default DLT suffix is `_dlt`. To customize it, set the `dlt.suffix` property in the compaction service configuration:

```yaml theme={null}
compactionScheduler:
  config:
    custom:
      dlt.suffix: "_deadletter"
```

With this configuration, the DLT table for topic `user-events` would be `user-events-deadletter`.

### Monitoring DLT Activity

When the compaction writer closes, it logs a summary of DLT activity including:

* Total number of records routed to DLT
* Breakdown by failure reason (up to 100 distinct reasons tracked)
* Up to 10 sample message IDs per failure reason

Example log output:

```
WARN Sent 42 record(s) to DLT for topic: persistent://public/default/events, partition: 0,
     failure reasons with messageIds: {"Schema evolution is disabled...": ["msgId1", "msgId2", ...]}
```

Use this information to diagnose schema mismatches, invalid data, or other data quality issues upstream.


# Variant Type
Source: https://docs.streamnative.io/cloud/lakehouse/features/variant-type



The Variant type allows a single column to hold values of different data types, enabling flexible handling of semi-structured data without defining a rigid schema upfront.

StreamNative Ursa supports the Variant type for both **Apache Iceberg (V3)** and **Delta Lake** tables.

> **Important:** Variant type support is disabled by default. Contact the StreamNative Support Team to enable the feature flag before using Variant types.

## Enabling Variant Support

Variant support is gated by a small set of broker properties. The required combination depends on the target table format.

| Property                   | Default | Description                                                                                                                     |
| -------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `variantTypeEnabled`       | `false` | Master switch for Variant support. Required for both Iceberg and Delta.                                                         |
| `tableEvolveSchemaEnabled` | `true`  | Schema evolution must remain enabled so Variant fields can be added/removed during writes. Required for both Iceberg and Delta. |
| `allowIcebergV3`           | `false` | Enables Iceberg V3 features (including Variant). **Required for Iceberg**, ignored for Delta.                                   |

### Iceberg

Set the following properties on the compaction service `custom` config:

```yaml theme={null}
variantTypeEnabled: "true"
tableEvolveSchemaEnabled: "true"   # default; only override if previously disabled
allowIcebergV3: "true"
```

> **Downstream query engine compatibility:** When `allowIcebergV3` is enabled, the downstream query engine reading the table must also support Iceberg V3. Older Spark / Trino / Athena versions that only support Iceberg V2 will fail to read tables that use Variant or other V3-only features. Verify your engine's Iceberg support level before enabling.

### Delta Lake

Delta Lake's Variant type is not gated by an Iceberg version flag. Only the master switch and schema evolution are required:

```yaml theme={null}
variantTypeEnabled: "true"
tableEvolveSchemaEnabled: "true"   # default; only override if previously disabled
```

## Supported Data Types

* **Primitives:** `string`, `int`, `long`, `float`, `double`, `boolean`, `bytes`
* **Complex types:** `map`, `list / array`, `set`
* **Nested POJOs** and **entire POJOs**

## Configure Variant in Pulsar

### Avro Schema

Use the `@AvroSchema` annotation with `logicalType: "variant"`:

```java theme={null}
@Data
public class Event {
    private String name;

    // Variant for primitive type
    @AvroSchema("{\"type\": \"string\", \"logicalType\": \"variant\"}")
    private String flexibleField;

    // Variant for Map
    @AvroSchema("{\"type\": \"map\", \"values\": \"string\", \"logicalType\": \"variant\"}")
    private Map<String, String> metadata;

    // Variant for List
    @AvroSchema("{\"type\": \"array\", \"items\": \"string\", \"logicalType\": \"variant\"}")
    private List<String> tags;

    // Variant for nested POJO with metadata fields for query optimization
    @AvroSchema("{\"type\": \"record\", \"name\": \"Address\", \"fields\": ["
            + "  {\"name\": \"city\", \"type\": \"string\"},"
            + "  {\"name\": \"zip\", \"type\": \"int\"}"
            + "],"
            + "\"logicalType\": \"variant\","
            + "\"variant-metadata-fields\": \"[\\\"zip\\\", \\\"city\\\"]\" "
            + "}")
    private Address address;
}
```

### JSON Schema

Use `@JsonPropertyDescription` with the Variant annotation:

```java theme={null}
@Data
public class Event {
    private String name;

    @JsonPropertyDescription("logicalType: variant")
    private String flexibleField;

    @JsonPropertyDescription("logicalType: variant")
    private Map<String, String> metadata;

    @JsonPropertyDescription("logicalType: variant")
    private List<String> tags;

    @JsonPropertyDescription("logicalType: variant")
    private Address address;
}
```

### ProtobufNative Schema

Define a custom field option named `logical_type`:

```protobuf theme={null}
syntax = "proto3";

import "google/protobuf/descriptor.proto";

extend google.protobuf.FieldOptions {
  string logical_type = 1001;
}

message Event {
  string name = 1;
  string flexible_field = 2 [(logical_type) = "variant"];
  map<string, string> metadata = 3 [(logical_type) = "variant"];
  repeated string tags = 4 [(logical_type) = "variant"];
  Address address = 5 [(logical_type) = "variant"];
}

message Address {
  string city = 1;
  int32 zip = 2;
}
```

## Configure Variant in Ursa (Kafka Protocol)

### Avro Schema -- POJO Annotation

Same `@AvroSchema` annotations as Pulsar. Produce data using `ReflectionAvroSerializer` (do not use `KafkaAvroSerializer`).

### Avro Schema -- Inline Definition

Define `logicalType: "variant"` directly in the Avro schema:

```json theme={null}
{
  "type": "record",
  "name": "Event",
  "fields": [
    {
      "name": "id",
      "type": ["null", "string"],
      "default": null
    },
    {
      "name": "score",
      "type": {
        "type": "double",
        "logicalType": "variant"
      }
    },
    {
      "name": "tags",
      "type": {
        "type": "array",
        "items": "string",
        "logicalType": "variant"
      },
      "default": []
    },
    {
      "name": "attributes",
      "type": {
        "type": "map",
        "values": "string",
        "logicalType": "variant"
      },
      "default": {}
    },
    {
      "name": "address",
      "type": {
        "type": "record",
        "name": "Address",
        "fields": [
          {"name": "street", "type": "string"},
          {"name": "city", "type": "string"}
        ],
        "logicalType": "variant",
        "variant-metadata-fields": "[\"street\", \"city\"]"
      }
    }
  ]
}
```

### JSON Schema

Same `@JsonPropertyDescription("logicalType: variant")` annotations as Pulsar.

### Protobuf Schema

Protobuf is supported via the same `logical_type` custom field option as Pulsar's ProtobufNative schema. Define the option once in your `.proto` file and tag each Variant field:

```protobuf theme={null}
syntax = "proto3";

import "google/protobuf/descriptor.proto";

extend google.protobuf.FieldOptions {
  string logical_type = 1001;
}

message Event {
  string name = 1;
  string flexible_field = 2 [(logical_type) = "variant"];
  map<string, string> metadata = 3 [(logical_type) = "variant"];
  repeated string tags = 4 [(logical_type) = "variant"];
  Address address = 5 [(logical_type) = "variant"];
}

message Address {
  string city = 1;
  int32 zip = 2;
}
```

## Performance Optimization

Use `variant-metadata-fields` to specify fields that should be extracted as top-level columns. This accelerates query performance by enabling predicate pushdown on those fields:

```json theme={null}
"variant-metadata-fields": "[\"zip\", \"city\"]"
```

## Schema Evolution Rules for Variant

| Operation                                | Supported                               |
| ---------------------------------------- | --------------------------------------- |
| Adding new Variant fields                | Yes                                     |
| Removing existing Variant fields         | Yes                                     |
| Converting non-Variant field to Variant  | No (messages sent to Dead Letter Table) |
| Converting Variant field to another type | No (messages sent to Dead Letter Table) |

For more information about the Dead Letter Table (DLT), see [Schema Evolution -- Dead Letter Table](/cloud/lakehouse/features/schema-evolution#dead-letter-table-dlt).


# Overview - Internal Tables
Source: https://docs.streamnative.io/cloud/lakehouse/internal-tables/internal-tables-overview



**Internal Tables (Coming Soon)**

Internal Tables are topics inside StreamNative Cloud that store data in Ursa Format, the native, high-performance storage layer used by StreamNative’s Cost Optimized Clusters. This capability is currently not available and will be released in an upcoming version of StreamNative Cloud.

**Key Characteristics**

* **Stored in Ursa Format** -
  Data is written using a compact, high-throughput format optimized for streaming and low-latency access.

* **Fully Managed by StreamNative** -
  StreamNative will handle all storage lifecycle operations — retention, compaction, indexing, and metadata.

* **StreamNative-Native Performance** -
  Designed for fast ingestion, low latency, and operational efficiency within the StreamNative cloud environment.

* **Not Externally Queryable via Open Table Formats** -
  Internal Tables are optimized for streaming workloads, not external analytics engines.

**When to Use Internal Tables**

* You need low-latency ingestion and high-throughput streaming.
* You prefer StreamNative to manage all storage operations end-to-end.
* You don’t require external interoperability with Iceberg/Delta or an external Lakehouse system.

Note: Internal Tables are under development and will be introduced soon. The feature description above outlines planned capabilities.


# Lakehouse Features Overview
Source: https://docs.streamnative.io/cloud/lakehouse/lakehouse-features



## Feature Matrix

| Feature          | Iceberg (SDT)  | Delta (SDT)  | Iceberg (SBT) | Delta (SBT) |
| ---------------- | -------------- | ------------ | ------------- | ----------- |
| Schema Evolution | Supported      | Supported    | Coming Soon   | Coming Soon |
| Partition Key    | Configurable   | Configurable | Coming Soon   | Coming Soon |
| Upsert           | Supported      | Supported    | Coming Soon   | Coming Soon |
| Variant Type     | Supported (V3) | Supported    | Coming Soon   | Coming Soon |
| Streaming Read   | --             | --           | Coming Soon   | Coming Soon |
| Analytical Query | Supported      | Supported    | Coming Soon   | Coming Soon |

## Supported Cloud Vendors

| Provider             | WAL Storage | Lakehouse Table |
| -------------------- | ----------- | --------------- |
| AWS S3               | Supported   | Supported       |
| Google Cloud Storage | Supported   | Supported       |
| Azure Blob Storage   | Supported   | Supported       |

## Supported Schemas

| Schema Format           | Supported |
| ----------------------- | --------- |
| Avro                    | Yes       |
| JSON                    | Yes       |
| ProtobufNative (Pulsar) | Yes       |
| Protobuf (Kafka)        | Yes       |

Supported schema registries:

* StreamNative Kafka Schema Registry
* Confluent Schema Registry

## Feature Guides

Each feature has its own detailed guide:

* [Schema Evolution](/cloud/lakehouse/features/schema-evolution) -- How the lakehouse table evolves when the topic schema changes, including Dead Letter Table (DLT) behavior
* [Variant Type](/cloud/lakehouse/features/variant-type) -- Handle semi-structured data with Iceberg / Delta Variant type
* [Partition Key](/cloud/lakehouse/features/iceberg-partition-key) -- Configure table partitioning with Iceberg transforms
* [Upsert](/cloud/lakehouse/features/iceberg-upsert) -- Deduplication and primary key-based updates
* [Persist Extra Metadata](/cloud/lakehouse/features/persist-extra-metadata) -- Persist message-envelope metadata (offset, publish time, properties) as a `__meta` Variant column
* [Persist Key](/cloud/lakehouse/features/persist-key) -- Persist the source message key as a `__key` column

## Next Steps

* [Observability](/cloud/lakehouse/lakehouse-observability) -- Monitor lakehouse tables with metrics and Grafana dashboards


# Lakehouse Observability
Source: https://docs.streamnative.io/cloud/lakehouse/lakehouse-observability



## Monitor data delivery progress in the Cloud Console

After you [enable the Lakehouse Table](/cloud/lakehouse/enable-lakehouse-integration) on a topic, the StreamNative Cloud Console shows a per-topic delivery dashboard. Use it to check the health of data delivery without any external monitoring setup.

Open a topic in the Cloud Console and select the **Lakehouse Table** tab.

<Frame>
  <img alt="Lakehouse Table tab on a topic, showing Streaming Lag, Last Success Commit Time, and Rejected Count" />
</Frame>

The dashboard reports three delivery indicators and the catalog the topic is delivering to.

| Indicator                    | What it means                                                                                                                                                                                                        | What to do if it looks wrong                                                                                                                                                                                                     |
| ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Streaming Lag**            | Number of messages produced to the topic but not yet committed to the lakehouse table. A small, steady value is expected; a continuously growing value indicates that delivery cannot keep up with the produce rate. | Check the produce rate, the catalog's availability, and recent **Rejected Count** changes. For deeper investigation, see the [Grafana dashboard](#grafana-dashboard) and the `pulsar_storage_compact_lag` metric.                |
| **Last Success Commit Time** | Time elapsed since the most recent successful commit to the lakehouse catalog. Updates regularly while the topic has traffic.                                                                                        | If this value keeps growing while the topic is actively receiving messages, delivery is stalled. Verify the catalog credentials and connectivity, then check the failure metrics in the [Grafana dashboard](#grafana-dashboard). |
| **Rejected Count**           | Number of messages that could not be written to the lakehouse table -- for example, messages that failed schema validation or exceeded size limits.                                                                  | A non-zero value means some messages were not delivered. Inspect the topic schema and producer payloads. Rejected messages are not retried automatically.                                                                        |

The **Catalog Settings** panel below confirms which catalog the topic is delivering to and shows whether the setting is inherited from the cluster, the namespace, or set directly on the topic. See [Configuration override priority](/cloud/lakehouse/enable-lakehouse-integration#configuration-override-priority) for how the effective catalog is resolved.

<Tip>
  The Cloud Console dashboard is the fastest way to verify delivery for a single topic. For fleet-wide monitoring, alerting, and historical trends, set up the [Grafana dashboard](#grafana-dashboard).
</Tip>

## Prerequisites

Before you can visualize Lakehouse metrics in your own Grafana instance, [enable Metrics Remote Write](/cloud/log-and-monitor/advanced-observability#metrics-remote-write-integration) on your Cloud Environment to forward StreamNative Cloud metrics to your Prometheus-compatible monitoring system or Datadog.

## Grafana Dashboard

A pre-built Grafana dashboard is available as [`CompactionScheduler.json`](https://github.com/streamnative/apache-pulsar-grafana-dashboard/tree/master/dashboards.kubernetes) in the [apache-pulsar-grafana-dashboard](https://github.com/streamnative/apache-pulsar-grafana-dashboard) repository. Import it into your Grafana instance for comprehensive monitoring.

### How to Import

1. Download [`CompactionScheduler.json`](https://github.com/streamnative/apache-pulsar-grafana-dashboard/tree/master/dashboards.kubernetes) from the repository.
2. Open Grafana -> **Dashboards** -> **Import**.
3. Upload `CompactionScheduler.json` or paste the JSON content.
4. Select your Prometheus data source.
5. Click **Import**.

### Dashboard Overview

The dashboard is organized into the following sections:

| Section                      | Description                                                                                                                                                                                                      |
| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Overview**                 | Topic count, task count, publish/compact/commit failed tasks, commit batch size                                                                                                                                  |
| **Compaction Write**         | Compaction lag, task publish lag, task stats, non-committable tasks, throughput (bytes/messages), latencies for compaction duration, WAL read, Parquet write, task commit, lakehouse commit, end-to-end pipeline |
| **Persistent API**           | Read throughput, read latencies (index+data, message, Oxia index, Oxia metadata)                                                                                                                                 |
| **WAL**                      | Read cache eviction/loading rate, WAL read latency, S3 cache loading latency                                                                                                                                     |
| **S3**                       | S3 read throughput, request rate, S3 read latency                                                                                                                                                                |
| **Compaction Read**          | Lakehouse read bytes/messages, read latency                                                                                                                                                                      |
| **Compaction Write Details** | Lakehouse write/encode/before-write/write-record latencies, Parquet write-record/write-metadata latencies                                                                                                        |
| **DLQ Tasks**                | Dead Letter Queue task statistics                                                                                                                                                                                |

***

## Key Alerts

These metrics should be monitored with alerting rules:

| Metric                                                                                       | Alert Condition                            | Severity |
| -------------------------------------------------------------------------------------------- | ------------------------------------------ | -------- |
| `pulsar_storage_compact_lag`                                                                 | Compaction lag exceeds threshold per topic | Warning  |
| `compaction_cluster_leaders_ratio`                                                           | Sum across cluster is not exactly 1        | Critical |
| `pulsar_storage_compact_quarantined_topics_count`                                            | Greater than 0                             | Warning  |
| `pulsar_storage_compact_topics_in_dlq`                                                       | Greater than 0                             | Critical |
| `pulsar_storage_compact_tasks_in_dlq`                                                        | Greater than 0                             | Critical |
| `pulsar_storage_compact_publish_task_failed_count_total`                                     | Increasing                                 | Warning  |
| `pulsar_storage_compact_failed_task_count_total`                                             | Increasing                                 | Warning  |
| `pulsar_storage_compact_task_commit_duration_seconds_count{pulsar_response_status="failed"}` | Increasing                                 | Critical |
| `pulsar_subscription_back_log`                                                               | Backlog exceeds threshold                  | Warning  |

***

## Compaction Service Metrics

The compaction service has three stages: task publishing (leader), WAL-to-Parquet conversion (worker), and commit to lakehouse (leader).

### Task Lifecycle

| Metric                                                  | Type  | Description                                      |
| ------------------------------------------------------- | ----- | ------------------------------------------------ |
| `pulsar_storage_compact_ongoing_topic_count`            | Gauge | Number of topics currently undergoing compaction |
| `pulsar_storage_compact_ongoing_task_count`             | Gauge | Number of active compaction tasks in progress    |
| `pulsar_storage_compact_tasks_in_init_state`            | Gauge | Tasks in initialization state                    |
| `pulsar_storage_compact_tasks_in_compacted_state`       | Gauge | Tasks in compacted state                         |
| `pulsar_storage_compact_tasks_in_prepared_commit_state` | Gauge | Tasks in prepared commit state                   |
| `pulsar_storage_compact_tasks_in_committed_state`       | Gauge | Tasks in committed state                         |

### Throughput

| Metric                                                | Type    | Description                                              |
| ----------------------------------------------------- | ------- | -------------------------------------------------------- |
| `pulsar_storage_compact_bytes_total`                  | Counter | Total bytes processed during compaction                  |
| `pulsar_storage_compact_messages_total`               | Counter | Total messages processed during compaction               |
| `pulsar_storage_compact_published_task_bytes`         | Gauge   | Size in bytes of messages batched in one compaction task |
| `pulsar_storage_compact_committed_parquet_file_bytes` | Gauge   | Size in bytes of committed Parquet files                 |
| `pulsar_storage_compact_commit_task_batch_size`       | Gauge   | Number of Parquet files in a single commit batch         |

### Offset Tracking

| Metric                                           | Type  | Description                                                        |
| ------------------------------------------------ | ----- | ------------------------------------------------------------------ |
| `pulsar_storage_compact_latest_message_offset`   | Gauge | Latest message offset for each topic                               |
| `pulsar_storage_compact_latest_published_offset` | Gauge | Latest published task's message offset                             |
| `pulsar_storage_compact_last_compacted_offset`   | Gauge | Latest offset confirmed as fully committed to lakehouse            |
| `pulsar_storage_compact_lag`                     | Gauge | Difference between latest message offset and last compacted offset |

### Latency

| Metric                                                                        | Type      | Description                                                            |
| ----------------------------------------------------------------------------- | --------- | ---------------------------------------------------------------------- |
| `pulsar_storage_compact_duration_seconds_bucket`                              | Histogram | Total latency of a compaction task                                     |
| `pulsar_storage_compact_read_messages_duration_seconds_bucket`                | Histogram | Latency for reading messages from WAL files                            |
| `pulsar_storage_compact_write_messages_duration_seconds_bucket`               | Histogram | Latency for decoding, converting, and writing to Parquet               |
| `pulsar_storage_compact_task_commit_duration_seconds_bucket`                  | Histogram | Latency for committing a task (includes Oxia index + catalog snapshot) |
| `pulsar_storage_compact_commit_to_lakehouse_duration_seconds_bucket`          | Histogram | Latency for committing snapshot to catalog service only                |
| `pulsar_storage_compact_message_from_ursa_to_parquet_duration_seconds_bucket` | Histogram | End-to-end latency: message write to Parquet file write                |
| `pulsar_storage_compact_message_end_to_end_duration_seconds_bucket`           | Histogram | End-to-end latency: message write to lakehouse commit                  |

### Failures

| Metric                                                               | Type      | Description                                   |
| -------------------------------------------------------------------- | --------- | --------------------------------------------- |
| `pulsar_storage_compact_publish_task_failed_count_total`             | Counter   | Total failed task publications                |
| `pulsar_storage_compact_failed_task_count_total`                     | Counter   | Total failed WAL-to-Parquet conversions       |
| `pulsar_storage_compact_quarantined_topics_count`                    | Gauge     | Topics quarantined due to compaction failures |
| `pulsar_storage_compact_topics_in_dlq`                               | Gauge     | Topics in Dead Letter Queue                   |
| `pulsar_storage_compact_tasks_in_dlq`                                | Gauge     | Tasks in Dead Letter Queue                    |
| `pulsar_storage_compact_non_committable_task_count`                  | Counter   | Non-committable tasks exceeding threshold     |
| `pulsar_storage_compact_non_committable_task_histogram_bytes_bucket` | Histogram | Size distribution of non-committable tasks    |

***

## WAL Storage Metrics

| Metric                                                         | Type      | Description                             |
| -------------------------------------------------------------- | --------- | --------------------------------------- |
| `pulsar_storage_wal_putEntry_count_total`                      | Counter   | Total entries written to WAL            |
| `pulsar_storage_wal_putEntry_rejected_count_total`             | Counter   | Total entries rejected during WAL write |
| `pulsar_storage_wal_putEntry_duration_seconds_bucket`          | Histogram | WAL write latency                       |
| `pulsar_storage_wal_putEntry_pending_duration_seconds_bucket`  | Histogram | Time entries wait in WAL buffer         |
| `pulsar_storage_wal_putEntry_cache_duration_seconds_bucket`    | Histogram | Write cache write latency               |
| `pulsar_storage_wal_getEntries_duration_seconds_bucket`        | Histogram | Batch read latency (cache or backend)   |
| `pulsar_storage_wal_getEntry_duration_seconds_bucket`          | Histogram | Single entry read latency               |
| `pulsar_storage_wal_writeCache_flush_duration_seconds_bucket`  | Histogram | Write cache flush latency               |
| `pulsar_storage_wal_readCache_loading_count_total`             | Counter   | Read cache loads from backend           |
| `pulsar_storage_wal_readCache_eviction_count_total`            | Counter   | Read cache evictions                    |
| `pulsar_storage_wal_readCache_loading_duration_seconds_bucket` | Histogram | Cache loading latency                   |
| `pulsar_storage_wal_read_cache_missed_total`                   | Counter   | Read cache misses                       |
| `pulsar_storage_wal_putEntry_pending_count`                    | Gauge     | Entries queued in WAL pending buffer    |
| `pulsar_storage_wal_writeCache_flushCallback_pending_count`    | Gauge     | Pending flush acknowledgments           |
| `pulsar_storage_wal_readCache_size_bytes`                      | Gauge     | Current read cache size                 |

### Write Cache Metrics

| Metric                                             | Type  | Description              |
| -------------------------------------------------- | ----- | ------------------------ |
| `pulsar_storage_wal_writeCache_used_bytes`         | Gauge | Write cache utilization  |
| `pulsar_storage_wal_writeCache_bufferSegment_used` | Gauge | Buffer segments in use   |
| `pulsar_storage_wal_writeCache_cacheSegment_used`  | Gauge | Cache segments in use    |
| `pulsar_storage_wal_writeCache_segment_count`      | Gauge | Total allocated segments |
| `pulsar_storage_wal_writeCache_capacity_bytes`     | Gauge | Max capacity per segment |

***

## File Storage Metrics

| Metric                                                         | Type      | Description                      |
| -------------------------------------------------------------- | --------- | -------------------------------- |
| `pulsar_storage_backend_storage_request_total`                 | Counter   | Total backend storage operations |
| `pulsar_storage_backend_write_duration_seconds_bucket`         | Histogram | Backend write latency            |
| `pulsar_storage_backend_read_duration_seconds_bucket`          | Histogram | Backend read latency             |
| `pulsar_storage_backend_metadata_read_duration_seconds_bucket` | Histogram | Metadata read latency            |
| `pulsar_storage_backend_crc_duration_seconds_bucket`           | Histogram | CRC calculation latency          |
| `pulsar_storage_backend_delete_duration_seconds_bucket`        | Histogram | Object deletion latency          |
| `pulsar_storage_backend_write_bytes_count_bytes_total`         | Counter   | Total bytes written to backend   |
| `pulsar_storage_backend_read_bytes_count_bytes_total`          | Counter   | Total bytes read from backend    |

***

## Lakehouse Read Metrics

| Metric                                                                | Type      | Description                                        |
| --------------------------------------------------------------------- | --------- | -------------------------------------------------- |
| `pulsar_storage_lakehouse_read_messages_total`                        | Counter   | Total messages read from lakehouse (Parquet files) |
| `pulsar_storage_lakehouse_read_bytes_bytes_total`                     | Counter   | Total bytes read from lakehouse                    |
| `pulsar_storage_lakehouse_read_request_total`                         | Counter   | Total read requests processed                      |
| `pulsar_storage_lakehouse_read_cache_hit_total`                       | Counter   | Parquet prefetch cache hits                        |
| `pulsar_storage_lakehouse_read_cache_miss_total`                      | Counter   | Parquet prefetch cache misses                      |
| `pulsar_storage_lakehouse_read_latency_seconds_bucket`                | Histogram | Read latency                                       |
| `pulsar_storage_lakehouse_read_request_queued_latency_seconds_bucket` | Histogram | Queue wait time before processing                  |

***

## Lakehouse Writer Metrics

| Metric                                                  | Type      | Description                     |
| ------------------------------------------------------- | --------- | ------------------------------- |
| `pulsar_storage_lakehouse_writer_before_write_duration` | Histogram | Pre-write operation latency     |
| `pulsar_storage_lakehouse_writer_write_all_duration`    | Histogram | Batch write latency             |
| `pulsar_storage_lakehouse_writer_write_record_duration` | Histogram | Individual record write latency |
| `pulsar_storage_lakehouse_writer_encode_duration`       | Histogram | Record encoding latency         |

## Lakehouse Reader Metrics

| Metric                                                 | Type      | Description                    |
| ------------------------------------------------------ | --------- | ------------------------------ |
| `pulsar_storage_lakehouse_reader_seek_duration`        | Histogram | Seek operation latency         |
| `pulsar_storage_lakehouse_reader_read_all_duration`    | Histogram | Batch read latency             |
| `pulsar_storage_lakehouse_reader_read_record_duration` | Histogram | Individual record read latency |
| `pulsar_storage_lakehouse_reader_decode_duration`      | Histogram | Record decoding latency        |

***

## Parquet File Metrics

### Writer

| Metric                                                     | Type      | Description                    |
| ---------------------------------------------------------- | --------- | ------------------------------ |
| `pulsar_storage_lakehouse_parquet_write_record_duration`   | Histogram | Parquet record write latency   |
| `pulsar_storage_lakehouse_parquet_write_metadata_duration` | Histogram | Parquet metadata write latency |

### Reader

| Metric                                                              | Type      | Description                     |
| ------------------------------------------------------------------- | --------- | ------------------------------- |
| `pulsar_storage_lakehouse_parquet_read_record_duration`             | Histogram | Parquet record read latency     |
| `pulsar_storage_lakehouse_parquet_read_metadata_duration`           | Histogram | Parquet metadata read latency   |
| `pulsar_storage_lakehouse_parquet_seek_by_offset_duration`          | Histogram | Seek by offset latency          |
| `pulsar_storage_lakehouse_parquet_seek_by_secondary_index_duration` | Histogram | Seek by secondary index latency |


# BigLake for Iceberg
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/biglake/iceberg



This guide describes how to prepare a Google BigLake metastore for use with StreamNative Ursa as an Iceberg REST catalog.

For background, see the Google Cloud documentation: [Use the BigLake metastore Iceberg REST catalog](https://docs.cloud.google.com/biglake/docs/blms-rest-catalog).

## Prerequisites

* A GCP project with permissions to create BigLake catalogs and modify IAM roles
* A StreamNative Ursa cluster running on GCP

## 1. Create a BigLake Catalog

In the Google Cloud Console, search for **BigLake**.

<img alt="Search BigLake" />

Create a new catalog by selecting a Cloud Storage bucket.

> **Important:**
>
> * The bucket must be in the **same region** as your StreamNative Ursa cluster, otherwise cross-region traffic and latency are introduced.
> * BigLake does not support sub-directories within a bucket. Each BigLake catalog maps to exactly one bucket (1:1 mapping).

In the **Authentication** section, choose **Credential vending mode**.

<img alt="Credential vending mode" />

After the catalog is created, view the catalog details to obtain the **REST Catalog URI**, **GCS Warehouse**, and **Project**.

<img alt="Catalog information" />

Click **Set bucket permissions** to grant the BigLake service account access to the bucket.

<img alt="Set bucket permissions" />

## 2. Grant IAM Roles to the Ursa Broker Service Account

Locate the StreamNative Ursa broker service account (the format is typically `iamaccount-<id>@<gcp-project>.iam.gserviceaccount.com`).

<img alt="Broker service account" />

In the GCP IAM console, grant the broker service account the following roles:

* **BigLake Editor**
* **Storage Object User**
* **Service Usage Consumer**

<img alt="Grant IAM roles" />

<img alt="Grant IAM roles" />

## Catalog Information Summary

When the steps above are complete, collect the following values for the StreamNative Ursa compaction service. Several values are fixed for any BigLake catalog -- copy them as-is.

| Value                                        | Description                                                                                              |
| -------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| `iceberg.catalog-backend`                    | `BIGLAKE`                                                                                                |
| `iceberg.type`                               | `rest`                                                                                                   |
| `iceberg.uri`                                | The **REST Catalog URI** from step 1 (typically `https://biglake.googleapis.com/iceberg/v1/restcatalog`) |
| `iceberg.warehouse`                          | The **GCS Warehouse** from step 1 (e.g., `gs://<bucket-name>`)                                           |
| `iceberg.header.x-goog-user-project`         | The **Project** from step 1                                                                              |
| `iceberg.rest.auth.type`                     | `org.apache.iceberg.gcp.auth.GoogleAuthManager` (fixed)                                                  |
| `iceberg.io-impl`                            | `org.apache.iceberg.gcp.gcs.GCSFileIO` (fixed)                                                           |
| `iceberg.rest-metrics-reporting-enabled`     | `false` (fixed -- BigLake does not yet support REST metrics reporting)                                   |
| `iceberg.header.X-Iceberg-Access-Delegation` | `vended-credentials` (fixed)                                                                             |

For the next steps, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog).


# Horizon Catalog for Iceberg on AWS
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/horizon-catalog/iceberg/aws



This guide describes how to prepare a Snowflake Horizon Catalog for use with StreamNative Ursa as an Iceberg catalog on AWS.

For background, see the Snowflake documentation: [Write to Apache Iceberg tables using an external query engine through Snowflake Horizon Catalog](https://docs.snowflake.com/en/LIMITEDACCESS/iceberg/tables-iceberg-write-using-external-write-engine-snowflake-horizon).

## Prerequisites

* A Snowflake account with Horizon Catalog enabled
* An AWS account with permissions to create S3 buckets and IAM roles
* The `ACCOUNTADMIN` role in Snowflake (required for several SQL operations below)

## 1. Create an External Volume

The Horizon Catalog uses a Snowflake **External Volume** to access object storage. Reference: [Tutorial: Create your first Apache Iceberg table](https://docs.snowflake.com/en/user-guide/tutorials/create-your-first-iceberg-table#create-a-table).

### 1.1 Identify the Snowflake Account Region

Find the region of your Snowflake account so the S3 bucket can be created in the same region.

<img alt="Snowflake region" />

### 1.2 Create an S3 Bucket

Create an S3 bucket in the AWS console, in the same region as your Snowflake account.

<img alt="Create S3 bucket" />

### 1.3 Create an IAM Policy

Create an IAM policy granting access to the bucket. Replace `<your-bucket>` with the bucket name from step 1.2.

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:DeleteObject",
        "s3:DeleteObjectVersion"
      ],
      "Resource": "arn:aws:s3:::<your-bucket>/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": "arn:aws:s3:::<your-bucket>",
      "Condition": {
        "StringLike": {
          "s3:prefix": ["*"]
        }
      }
    }
  ]
}
```

<img alt="IAM policy" />

### 1.4 Create an IAM Role

Create an IAM role with an External ID (for example, `iceberg_table_external_id`). The trust policy will be updated in step 1.6.

<img alt="Create IAM role" />

Attach the policy from step 1.3 to the role.

<img alt="Bind policy to role" />

### 1.5 Create the External Volume in Snowflake

Switch to the `ACCOUNTADMIN` role and run the following SQL, substituting your values:

```sql theme={null}
CREATE OR REPLACE EXTERNAL VOLUME <volume_name>
   STORAGE_LOCATIONS = (
      (
         NAME = '<location-name>'
         STORAGE_PROVIDER = 'S3'
         STORAGE_BASE_URL = 's3://<your-bucket>/'
         STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::<your-account-id>:role/<your-role-name>'
         STORAGE_AWS_EXTERNAL_ID = '<your-external-id>'
      )
   )
   ALLOW_WRITES = TRUE;
```

<img alt="External volume" />

If the command fails with a permission error, ensure you are using the `ACCOUNTADMIN` role:

<img alt="Permission issue" />

<img alt="Switch to ACCOUNTADMIN" />

### 1.6 Configure the Trust Relationship

After the external volume is created, retrieve the Snowflake-generated IAM user ARN:

```sql theme={null}
DESC EXTERNAL VOLUME <volume_name>;
```

The `STORAGE_AWS_IAM_USER_ARN` field contains the value (for example, `arn:aws:iam::<snowflake-account>:user/<id>`).

<img alt="Volume info" />

Return to the AWS IAM console and update the trust policy of the role created in step 1.4 to allow `STORAGE_AWS_IAM_USER_ARN` to assume the role.

<img alt="Update trust policy" />

<img alt="Update trust policy" />

## 2. Configure Access Control

> **Note:** If you already have roles configured with access to the Iceberg tables you want to use, you can skip this section.

For details, see [Configuring access control](https://docs.snowflake.com/en/user-guide/security-access-control-configure).

```sql theme={null}
GRANT ROLE ACCOUNTADMIN, SYSADMIN TO USER <username>;
```

<img alt="Grant roles" />

## 3. Create the Catalog Database

Switch to `ACCOUNTADMIN` and create the catalog database in the Snowflake UI. Then bind the database to the external volume:

```sql theme={null}
ALTER DATABASE <db_name> SET CATALOG = 'SNOWFLAKE';
ALTER DATABASE <db_name> SET EXTERNAL_VOLUME = '<volume_name>';
```

<img alt="Create catalog" />

## 4. Obtain an Access Token

Snowflake Horizon supports three authentication methods: External OAuth, Key-pair authentication, and Programmatic Access Token (PAT). This guide uses **PAT**.

### 4.1 Create an Authentication Policy

Switch to `ACCOUNTADMIN` and run:

```sql theme={null}
USE <db_name>;

CREATE AUTHENTICATION POLICY <policy_name>
  PAT_POLICY=(NETWORK_POLICY_EVALUATION = ENFORCED_NOT_REQUIRED);

ALTER ACCOUNT SET AUTHENTICATION POLICY <policy_name>;

ALTER AUTHENTICATION POLICY <policy_name>
  SET AUTHENTICATION_METHODS = ('OAUTH', 'PASSWORD', 'PROGRAMMATIC_ACCESS_TOKEN');

ALTER AUTHENTICATION POLICY <policy_name>
  SET PAT_POLICY = (MAX_EXPIRY_IN_DAYS=365, DEFAULT_EXPIRY_IN_DAYS=365);

ALTER USER IF EXISTS <username> ADD PROGRAMMATIC ACCESS TOKEN <token_name>
  DAYS_TO_EXPIRY = 365
  COMMENT = 'PAT for StreamNative Ursa';
```

Record the generated PAT.

<img alt="Generate PAT" />

### 4.2 Verify the Endpoint

Generate an OAuth access token using the PAT and verify the Horizon REST endpoint:

```bash theme={null}
curl -i --fail -X POST \
  "https://<account_identifier>.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens" \
  --header 'Content-Type: application/x-www-form-urlencoded' \
  --data-urlencode 'grant_type=client_credentials' \
  --data-urlencode 'scope=session:role:<role>' \
  --data-urlencode 'client_secret=<PAT>'

curl -i --fail -X GET \
  "https://<account_identifier>.snowflakecomputing.com/polaris/api/catalog/v1/config?warehouse=<database_name>" \
  -H "Authorization: Bearer <access_token>" \
  -H "Content-Type: application/json"
```

### 4.3 Grant Permissions

```sql theme={null}
GRANT ROLE PUBLIC TO USER <username>;
GRANT USAGE ON DATABASE <db_name> TO ROLE PUBLIC;
GRANT USAGE ON EXTERNAL VOLUME <volume_name> TO ROLE PUBLIC;
```

## Catalog Information Summary

When the steps above are complete, collect the following values for the StreamNative Ursa compaction service:

```sql theme={null}
-- Determine the URI prefix
SELECT CURRENT_ORGANIZATION_NAME() || '-' || CURRENT_ACCOUNT_NAME();
```

| Value                | Description                                                                   |
| -------------------- | ----------------------------------------------------------------------------- |
| `iceberg.uri`        | `https://<organization>-<account>.snowflakecomputing.com/polaris/api/catalog` |
| `iceberg.warehouse`  | The catalog database name created in step 3                                   |
| `iceberg.credential` | The PAT generated in step 4.1                                                 |
| `iceberg.scope`      | `session:role:<role>` (e.g., `session:role:PUBLIC`)                           |

For the next steps, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog).


# Open Catalog for Iceberg on AWS
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/open-catalog/iceberg/aws



This guide describes how to prepare a Snowflake Open Catalog (Polaris) for use with StreamNative Ursa as an Iceberg catalog on AWS.

> **Important:** Polaris does not support reading buckets from a different region. The StreamNative Ursa cluster, the storage bucket, and the Polaris catalog must all reside in the **same AWS region**.

## Prerequisites

* A Snowflake standard account
* An AWS account with permissions to create S3 buckets and IAM roles
* Access to the Snowflake Open Catalog feature (request via your Snowflake account team if not yet enabled)

## 1. Create a Snowflake Open Catalog Account

The Snowflake Open Catalog console requires a dedicated **Open Catalog** account. From the standard Snowflake console, navigate to **Admin -> Accounts** and use the toggle to **Create Snowflake Open Catalog Account**.

<img alt="Snowflake console" />

<img alt="Create Open Catalog account" />

Configure the account with:

* **Cloud:** AWS
* **Region:** the region in which your S3 bucket resides (for example, `US East (Ohio)`)
* **Edition:** any

<img alt="Account configuration" />

Provide an admin username and password.

<img alt="Account credentials" />

After creation, click the **Account URL** to sign in to the Open Catalog console.

<img alt="Account created" />

<img alt="Open Catalog console" />

## 2. Create an S3 Bucket

Create an S3 bucket in the same region as the Open Catalog account.

<img alt="Create bucket" />

## 3. Create an IAM Policy

Navigate to **AWS IAM -> Policies -> Create policy**.

<img alt="Create policy" />

Paste the following policy, replacing the bucket name and subpath with your values:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:DeleteObject",
        "s3:DeleteObjectVersion"
      ],
      "Resource": "arn:aws:s3:::<your-bucket>/<your-subpath>/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": "arn:aws:s3:::<your-bucket>/<your-subpath>",
      "Condition": {
        "StringLike": {
          "s3:prefix": ["*"]
        }
      }
    }
  ]
}
```

<img alt="Policy JSON" />

<img alt="Policy next step" />

## 4. Create an IAM Role

Navigate to **AWS IAM -> Roles -> Create role** and configure:

* **Trusted entity type:** AWS account
* **An AWS account:** This account
* **Enable External ID** with a unique value (you will reference this when creating the Polaris catalog)

<img alt="Create role" />

<img alt="Trust settings" />

Attach the policy created in step 4.

<img alt="Attach policy" />

Provide a role name and create the role.

<img alt="Save role" />

Record the role ARN (for example, `arn:aws:iam::<account-id>:role/<role-name>`).

<img alt="Role ARN" />

## 5. Create the Polaris Catalog

In the Snowflake Open Catalog console, create a new catalog.

<img alt="Create catalog" />

Configure the catalog with:

* **External:** disabled
* **Storage provider:** S3
* **Default base location:** `s3://<your-bucket>/<your-subpath>` (the path from step 3)
* **S3 role ARN:** the role ARN recorded in step 5
* **External ID:** the External ID configured in step 5

<img alt="Catalog configuration" />

<img alt="Catalog created" />

Open the catalog details and record the **IAM user ARN** that Polaris uses to access AWS. You will use this in step 7 to update the trust policy of the IAM role.

<img alt="Catalog IAM user ARN" />

## 6. Update the IAM Role Trust Policy

Return to the AWS IAM console, open the role created in step 5, and edit the trust relationship.

<img alt="Find role" />

<img alt="Edit trust policy" />

Update `Principal.AWS` to the Polaris IAM user ARN recorded in step 6.

<img alt="Update trust policy" />

Click **Update policy**.

## 7. Create a Connection (Service Principal)

In the Open Catalog console, create a new connection that StreamNative Ursa will use to authenticate.

<img alt="Create connection" />

Configure with:

* **Name:** any name
* **Create new principal role:** enabled
* **Principal Role Name:** any name

<img alt="Connection configuration" />

After creation, record the **Client ID** and **Client Secret** -- the secret cannot be retrieved later.

<img alt="Connection credentials" />

## 8. Create a Catalog Role and Grant Privileges

Navigate to **Catalogs -> \[your catalog] -> Roles -> + Catalog Role** and create a role with the following privileges:

* `NAMESPACE_CREATE`
* `NAMESPACE_LIST`
* `NAMESPACE_READ_PROPERTIES`
* `NAMESPACE_WRITE_PROPERTIES`
* `TABLE_LIST`
* `TABLE_CREATE`
* `TABLE_WRITE_DATA`
* `TABLE_READ_DATA`
* `TABLE_READ_PROPERTIES`
* `TABLE_WRITE_PROPERTIES`

<img alt="Create catalog role" />

Click **Grant to Principals Role** and grant the catalog role to the principal role created in step 8.

<img alt="Grant to principal role" />

<img alt="Grant configuration" />

<img alt="Role bindings" />

For background on the relationship between catalogs, catalog roles, principal roles, and principals, see the [Polaris Quick Start](https://polaris.io/#section/Quick-Start/Defining-a-Catalog).

## Catalog Information Summary

When the steps above are complete, collect the following values for the StreamNative Ursa compaction service:

| Value                | Description                                                                                                                                                    |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `iceberg.uri`        | Polaris REST endpoint (e.g., `https://<account>.<region>.aws.snowflakecomputing.com/polaris/api/catalog`). The format follows the URL of your Polaris console. |
| `iceberg.warehouse`  | The Polaris catalog name created in step 6                                                                                                                     |
| `iceberg.credential` | `<client-id>:<client-secret>` from step 8                                                                                                                      |
| `iceberg.scope`      | `PRINCIPAL_ROLE:ALL`                                                                                                                                           |

## Table Maintenance

Snowflake Open Catalog (Polaris) and the Hadoop catalog do **not** run table maintenance on your behalf. Streaming writes from the StreamNative Ursa compaction service produce many small Parquet files and accumulate snapshot history over time, which degrades query performance and inflates storage costs. You are responsible for scheduling and running maintenance against every Iceberg table written by Ursa.

Run the maintenance operations below on a regular schedule. They are provided as [Apache Iceberg Spark stored procedures](https://iceberg.apache.org/docs/latest/spark-procedures/) and can be triggered from any Spark cluster (Databricks, AWS EMR, AWS Glue, GCP Dataproc, or self-managed Spark) that has the Iceberg Spark runtime, catalog credentials, and IAM access to the warehouse bucket.

**Maintenance operations**

| Operation             | Purpose                                                                                                                               | Suggested cadence                                                                         |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| `rewrite_data_files`  | Compact small Parquet files into fewer, larger files. Reduces file-listing overhead and improves scan performance.                    | Hourly to daily, depending on ingestion rate                                              |
| `expire_snapshots`    | Drop snapshots older than the retention window so their data and manifest files can be cleaned up.                                    | Daily; retain at least 1–7 days so in-flight readers and time-travel queries keep working |
| `remove_orphan_files` | Delete files in the table location that are no longer referenced by any snapshot (typically left behind by failed or partial writes). | Weekly                                                                                    |
| `rewrite_manifests`   | Rewrite manifest files so they align with the current partition layout. Improves query planning time.                                 | Weekly, or after large schema or partition changes                                        |

**Example: run maintenance from Spark**

The following examples assume the catalog has been registered in Spark as `<catalog>`. Replace `<catalog>`, `<namespace>`, and `<table>` with your values.

```sql theme={null}
-- Compact small files. Iceberg targets files smaller than the default 512 MB.
CALL <catalog>.system.rewrite_data_files(table => '<namespace>.<table>');

-- Expire snapshots older than 3 days; keep the 5 most recent snapshots.
CALL <catalog>.system.expire_snapshots(
  table       => '<namespace>.<table>',
  older_than  => TIMESTAMP '2026-05-20 00:00:00',
  retain_last => 5
);

-- Remove orphan files older than 7 days.
CALL <catalog>.system.remove_orphan_files(
  table      => '<namespace>.<table>',
  older_than => TIMESTAMP '2026-05-20 00:00:00'
);

-- Rewrite manifests to match the current partition layout.
CALL <catalog>.system.rewrite_manifests(table => '<namespace>.<table>');
```

**Operational guidance**

* **Credentials.** The principal that runs maintenance must have catalog privileges to read and write the target table (for example, the same `TABLE_READ_DATA`, `TABLE_WRITE_DATA`, `TABLE_READ_PROPERTIES`, and `TABLE_WRITE_PROPERTIES` privileges configured for the Ursa compaction service) and IAM access to the warehouse bucket so it can read and rewrite the underlying data files. With the Hadoop catalog there is no catalog service to authenticate against — only the bucket IAM access is required.
* **Concurrency.** Iceberg uses optimistic concurrency control. If maintenance commits race with the Ursa compaction writer, one of them retries. Schedule heavy operations (`rewrite_data_files`, `rewrite_manifests`) during low-write windows when possible.
* **Retention vs. time travel.** `expire_snapshots` and `remove_orphan_files` permanently delete files. Choose a retention window that exceeds the longest expected read query and your time-travel SLA.
* **Schedule the workload.** Most teams orchestrate these procedures from Databricks Jobs, AWS EMR steps, Airflow, Dagster, or a Kubernetes `CronJob`. Pick a scheduler that fits your existing operational stack.
* **Reference.** See the [Iceberg Spark procedures](https://iceberg.apache.org/docs/latest/spark-procedures/#metadata-management) documentation for the full parameter list, including options for partial rewrites (`where`), file-size targets, and merge-on-read delete file compaction.

For the next steps, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog).


# Open Catalog for Iceberg on Azure
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/open-catalog/iceberg/azure



This guide describes how to prepare a Snowflake Open Catalog (Polaris) for use with StreamNative Ursa as an Iceberg catalog on Microsoft Azure.

## Prerequisites

* A Snowflake standard account
* An Azure subscription with permissions to create storage accounts and configure trusted apps
* Access to the Snowflake Open Catalog feature

## 1. Create a Snowflake Open Catalog Account

The Snowflake Open Catalog console requires a dedicated **Open Catalog** account. From the standard Snowflake console, navigate to **Admin -> Accounts** and use the toggle to **Create Snowflake Open Catalog Account**.

<img alt="Snowflake console" />

<img alt="Create Open Catalog account" />

Configure the account with:

* **Cloud:** AWS or Azure (per Polaris availability)
* **Region:** the region in which your storage container resides
* **Edition:** any

<img alt="Account configuration" />

Provide an admin username and password.

<img alt="Account credentials" />

After creation, click the **Account URL** to sign in to the Open Catalog console.

<img alt="Account created" />

<img alt="Open Catalog console" />

## 2. Collect Azure Account Information

### 2.1 Azure Tenant ID

In the Azure portal, search for **Tenant properties** and record the **Tenant ID**.

<img alt="Search Tenant properties" />

<img alt="Tenant ID" />

### 2.2 Storage Service Endpoint

In the Azure portal, navigate to **Storage accounts**, open the target storage account, and click **Settings -> Endpoints**. Record the **Blob service** primary endpoint (for example, `https://<account>.blob.core.windows.net/`).

<img alt="Search Storage accounts" />

<img alt="Storage endpoint" />

### 2.3 Create a Container

In the storage account, navigate to **Data storage -> Containers -> + Container** and create a new container.

<img alt="Create container" />

## 3. Create the Polaris Catalog

In the Snowflake Open Catalog console, create a new catalog.

<img alt="Create catalog" />

> **Important:** The StreamNative compaction service writes data using the AzureDFS protocol, so the Polaris catalog must use the same protocol. Use `abfss://<container>@<account>.dfs.core.windows.net` (note `dfs`, not `blob`).

Configure the catalog with:

* **External:** disabled
* **Storage provider:** AZURE
* **Default base location:** `abfss://<container>@<storage-account>.dfs.core.windows.net`
* **Azure tenant ID:** the value from step 2.1

<img alt="Catalog configuration" />

## 4. Create a Trusted App in Azure

Open the catalog details and record the values of `AZURE_CONSENT_URL` and `AZURE_MULTI_TENANT_APP_NAME`.

<img alt="Catalog Azure values" />

Open the `AZURE_CONSENT_URL` in a browser and click **Accept**. This redirects to Snowflake and creates a trusted app in Azure. The trusted app name is the portion of `AZURE_MULTI_TENANT_APP_NAME` before the underscore.

> **Note:** Provisioning the trusted app in Azure can take several minutes.

## 5. Grant Container Permissions to the Trusted App

In the Azure portal, navigate to the storage account's **Access Control (IAM) -> + Add -> Add role assignment**.

<img alt="Add role assignment" />

Search for **Storage Blob Data** and select **Storage Blob Data Contributor**.

<img alt="Select role" />

Click **Select members**, search for the trusted app name from step 4, select it, and click **Review + assign**.

<img alt="Select members" />

<img alt="Role assigned" />

## 6. Create a Connection (Service Principal)

In the Open Catalog console, create a new connection that StreamNative Ursa will use to authenticate.

<img alt="Create connection" />

Configure with:

* **Name:** any name
* **Create new principal role:** enabled
* **Principal Role Name:** any name

<img alt="Connection configuration" />

After creation, record the **Client ID** and **Client Secret** -- the secret cannot be retrieved later.

<img alt="Connection credentials" />

## 7. Create a Catalog Role and Grant Privileges

Navigate to **Catalogs -> \[your catalog] -> Roles -> + Catalog Role** and create a role with the following privileges:

* `NAMESPACE_CREATE`
* `NAMESPACE_READ_PROPERTIES`
* `TABLE_CREATE`
* `TABLE_WRITE_DATA`
* `TABLE_READ_DATA`

<img alt="Create catalog role" />

Click **Grant to Principals Role** and grant the catalog role to the principal role created in step 6.

<img alt="Grant to principal role" />

<img alt="Grant configuration" />

## Catalog Information Summary

When the steps above are complete, collect the following values for the StreamNative Ursa compaction service:

| Value                | Description                                                                                                                                                |
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `iceberg.uri`        | Polaris REST endpoint (e.g., `https://<account>.<region>.snowflakecomputing.com/polaris/api/catalog`). The format follows the URL of your Polaris console. |
| `iceberg.warehouse`  | The Polaris catalog name created in step 3                                                                                                                 |
| `iceberg.credential` | `<client-id>:<client-secret>` from step 6                                                                                                                  |
| `iceberg.scope`      | `PRINCIPAL_ROLE:ALL`                                                                                                                                       |

## Table Maintenance

Snowflake Open Catalog (Polaris) and the Hadoop catalog do **not** run table maintenance on your behalf. Streaming writes from the StreamNative Ursa compaction service produce many small Parquet files and accumulate snapshot history over time, which degrades query performance and inflates storage costs. You are responsible for scheduling and running maintenance against every Iceberg table written by Ursa.

Run the maintenance operations below on a regular schedule. They are provided as [Apache Iceberg Spark stored procedures](https://iceberg.apache.org/docs/latest/spark-procedures/) and can be triggered from any Spark cluster (Databricks, AWS EMR, AWS Glue, GCP Dataproc, or self-managed Spark) that has the Iceberg Spark runtime, catalog credentials, and IAM access to the warehouse bucket.

**Maintenance operations**

| Operation             | Purpose                                                                                                                               | Suggested cadence                                                                         |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| `rewrite_data_files`  | Compact small Parquet files into fewer, larger files. Reduces file-listing overhead and improves scan performance.                    | Hourly to daily, depending on ingestion rate                                              |
| `expire_snapshots`    | Drop snapshots older than the retention window so their data and manifest files can be cleaned up.                                    | Daily; retain at least 1–7 days so in-flight readers and time-travel queries keep working |
| `remove_orphan_files` | Delete files in the table location that are no longer referenced by any snapshot (typically left behind by failed or partial writes). | Weekly                                                                                    |
| `rewrite_manifests`   | Rewrite manifest files so they align with the current partition layout. Improves query planning time.                                 | Weekly, or after large schema or partition changes                                        |

**Example: run maintenance from Spark**

The following examples assume the catalog has been registered in Spark as `<catalog>`. Replace `<catalog>`, `<namespace>`, and `<table>` with your values.

```sql theme={null}
-- Compact small files. Iceberg targets files smaller than the default 512 MB.
CALL <catalog>.system.rewrite_data_files(table => '<namespace>.<table>');

-- Expire snapshots older than 3 days; keep the 5 most recent snapshots.
CALL <catalog>.system.expire_snapshots(
  table       => '<namespace>.<table>',
  older_than  => TIMESTAMP '2026-05-20 00:00:00',
  retain_last => 5
);

-- Remove orphan files older than 7 days.
CALL <catalog>.system.remove_orphan_files(
  table      => '<namespace>.<table>',
  older_than => TIMESTAMP '2026-05-20 00:00:00'
);

-- Rewrite manifests to match the current partition layout.
CALL <catalog>.system.rewrite_manifests(table => '<namespace>.<table>');
```

**Operational guidance**

* **Credentials.** The principal that runs maintenance must have catalog privileges to read and write the target table (for example, the same `TABLE_READ_DATA`, `TABLE_WRITE_DATA`, `TABLE_READ_PROPERTIES`, and `TABLE_WRITE_PROPERTIES` privileges configured for the Ursa compaction service) and IAM access to the warehouse bucket so it can read and rewrite the underlying data files. With the Hadoop catalog there is no catalog service to authenticate against — only the bucket IAM access is required.
* **Concurrency.** Iceberg uses optimistic concurrency control. If maintenance commits race with the Ursa compaction writer, one of them retries. Schedule heavy operations (`rewrite_data_files`, `rewrite_manifests`) during low-write windows when possible.
* **Retention vs. time travel.** `expire_snapshots` and `remove_orphan_files` permanently delete files. Choose a retention window that exceeds the longest expected read query and your time-travel SLA.
* **Schedule the workload.** Most teams orchestrate these procedures from Databricks Jobs, AWS EMR steps, Airflow, Dagster, or a Kubernetes `CronJob`. Pick a scheduler that fits your existing operational stack.
* **Reference.** See the [Iceberg Spark procedures](https://iceberg.apache.org/docs/latest/spark-procedures/#metadata-management) documentation for the full parameter list, including options for partial rewrites (`where`), file-size targets, and merge-on-read delete file compaction.

For the next steps, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog).


# Open Catalog for Iceberg on GCP
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/open-catalog/iceberg/gcp



This guide describes how to prepare a Snowflake Open Catalog (Polaris) for use with StreamNative Ursa as an Iceberg catalog on Google Cloud Platform (GCP).

> **Important:** Polaris does not support reading buckets from a different region. The StreamNative Ursa cluster, the GCS bucket, and the Polaris catalog must all reside in the **same region**.

## Prerequisites

* A Snowflake standard account
* A GCP project with permissions to create GCS buckets and IAM roles
* Access to the Snowflake Open Catalog feature

## 1. Create a Snowflake Open Catalog Account

The Snowflake Open Catalog console requires a dedicated **Open Catalog** account. From the standard Snowflake console, navigate to **Admin -> Accounts** and use the toggle to **Create Snowflake Open Catalog Account**.

<img alt="Snowflake console" />

<img alt="Create Open Catalog account" />

Configure the account with:

* **Cloud:** GCP
* **Region:** the region in which your GCS bucket resides
* **Edition:** any

<img alt="Account configuration" />

Provide an admin username and password.

<img alt="Account credentials" />

After creation, click the **Account URL** to sign in to the Open Catalog console.

<img alt="Account created" />

<img alt="Open Catalog console" />

## 2. Create the Polaris Catalog

In the Snowflake Open Catalog console, create a new catalog.

<img alt="Create catalog" />

Configure the catalog with:

* **External:** disabled
* **Storage provider:** GCS
* **Default base location:** the GCS path used by the Ursa cluster (`gs://<bucket>/<subpath>`)

<img alt="Catalog configuration" />

<img alt="Catalog created" />

Open the catalog details and record the **GCP\_SERVICE\_ACCOUNT** value. Polaris uses this service account to access GCS, so it must be granted permission on the bucket.

<img alt="Catalog GCP service account" />

## 3. Grant Bucket Permissions to the Polaris Service Account

### 3.1 Create a Custom IAM Role

In the GCP console, navigate to **IAM & Admin -> Roles -> Create role** and add the following permissions:

* `storage.buckets.get`
* `storage.objects.create`
* `storage.objects.delete`
* `storage.objects.get`
* `storage.objects.list`

<img alt="Create role" />

<img alt="Role setup" />

<img alt="Permissions" />

<img alt="Permissions selected" />

### 3.2 Assign the Role to the Polaris Service Account

Open the bucket, navigate to **PERMISSIONS -> View BY PRINCIPALS -> GRANT ACCESS**.

<img alt="Grant bucket access" />

Add the **GCP\_SERVICE\_ACCOUNT** from step 3, choose the role created in step 4.1, and click **SAVE**.

<img alt="Save access" />

## 4. Create a Connection (Service Principal)

In the Open Catalog console, create a new connection that StreamNative Ursa will use to authenticate.

<img alt="Create connection" />

Configure with:

* **Name:** any name
* **Create new principal role:** enabled
* **Principal Role Name:** any name

<img alt="Connection configuration" />

After creation, record the **Client ID** and **Client Secret** -- the secret cannot be retrieved later.

<img alt="Connection credentials" />

## 5. Create a Catalog Role and Grant Privileges

Navigate to **Catalogs -> \[your catalog] -> Roles -> + Catalog Role** and create a role with the following privileges:

* `NAMESPACE_CREATE`
* `NAMESPACE_LIST`
* `NAMESPACE_READ_PROPERTIES`
* `NAMESPACE_WRITE_PROPERTIES`
* `TABLE_LIST`
* `TABLE_CREATE`
* `TABLE_WRITE_DATA`
* `TABLE_READ_DATA`
* `TABLE_READ_PROPERTIES`
* `TABLE_WRITE_PROPERTIES`

<img alt="Create catalog role" />

Click **Grant to Principals Role** and grant the catalog role to the principal role created in step 5.

<img alt="Grant to principal role" />

<img alt="Grant configuration" />

<img alt="Role bindings" />

For background on the relationship between catalogs, catalog roles, principal roles, and principals, see the [Polaris Quick Start](https://polaris.io/#section/Quick-Start/Defining-a-Catalog).

## Catalog Information Summary

When the steps above are complete, collect the following values for the StreamNative Ursa compaction service:

| Value                | Description                                                                                                                                                    |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `iceberg.uri`        | Polaris REST endpoint (e.g., `https://<account>.<region>.gcp.snowflakecomputing.com/polaris/api/catalog`). The format follows the URL of your Polaris console. |
| `iceberg.warehouse`  | The Polaris catalog name created in step 3                                                                                                                     |
| `iceberg.credential` | `<client-id>:<client-secret>` from step 5                                                                                                                      |
| `iceberg.scope`      | `PRINCIPAL_ROLE:ALL`                                                                                                                                           |

## Table Maintenance

Snowflake Open Catalog (Polaris) and the Hadoop catalog do **not** run table maintenance on your behalf. Streaming writes from the StreamNative Ursa compaction service produce many small Parquet files and accumulate snapshot history over time, which degrades query performance and inflates storage costs. You are responsible for scheduling and running maintenance against every Iceberg table written by Ursa.

Run the maintenance operations below on a regular schedule. They are provided as [Apache Iceberg Spark stored procedures](https://iceberg.apache.org/docs/latest/spark-procedures/) and can be triggered from any Spark cluster (Databricks, AWS EMR, AWS Glue, GCP Dataproc, or self-managed Spark) that has the Iceberg Spark runtime, catalog credentials, and IAM access to the warehouse bucket.

**Maintenance operations**

| Operation             | Purpose                                                                                                                               | Suggested cadence                                                                         |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| `rewrite_data_files`  | Compact small Parquet files into fewer, larger files. Reduces file-listing overhead and improves scan performance.                    | Hourly to daily, depending on ingestion rate                                              |
| `expire_snapshots`    | Drop snapshots older than the retention window so their data and manifest files can be cleaned up.                                    | Daily; retain at least 1–7 days so in-flight readers and time-travel queries keep working |
| `remove_orphan_files` | Delete files in the table location that are no longer referenced by any snapshot (typically left behind by failed or partial writes). | Weekly                                                                                    |
| `rewrite_manifests`   | Rewrite manifest files so they align with the current partition layout. Improves query planning time.                                 | Weekly, or after large schema or partition changes                                        |

**Example: run maintenance from Spark**

The following examples assume the catalog has been registered in Spark as `<catalog>`. Replace `<catalog>`, `<namespace>`, and `<table>` with your values.

```sql theme={null}
-- Compact small files. Iceberg targets files smaller than the default 512 MB.
CALL <catalog>.system.rewrite_data_files(table => '<namespace>.<table>');

-- Expire snapshots older than 3 days; keep the 5 most recent snapshots.
CALL <catalog>.system.expire_snapshots(
  table       => '<namespace>.<table>',
  older_than  => TIMESTAMP '2026-05-20 00:00:00',
  retain_last => 5
);

-- Remove orphan files older than 7 days.
CALL <catalog>.system.remove_orphan_files(
  table      => '<namespace>.<table>',
  older_than => TIMESTAMP '2026-05-20 00:00:00'
);

-- Rewrite manifests to match the current partition layout.
CALL <catalog>.system.rewrite_manifests(table => '<namespace>.<table>');
```

**Operational guidance**

* **Credentials.** The principal that runs maintenance must have catalog privileges to read and write the target table (for example, the same `TABLE_READ_DATA`, `TABLE_WRITE_DATA`, `TABLE_READ_PROPERTIES`, and `TABLE_WRITE_PROPERTIES` privileges configured for the Ursa compaction service) and IAM access to the warehouse bucket so it can read and rewrite the underlying data files. With the Hadoop catalog there is no catalog service to authenticate against — only the bucket IAM access is required.
* **Concurrency.** Iceberg uses optimistic concurrency control. If maintenance commits race with the Ursa compaction writer, one of them retries. Schedule heavy operations (`rewrite_data_files`, `rewrite_manifests`) during low-write windows when possible.
* **Retention vs. time travel.** `expire_snapshots` and `remove_orphan_files` permanently delete files. Choose a retention window that exceeds the longest expected read query and your time-travel SLA.
* **Schedule the workload.** Most teams orchestrate these procedures from Databricks Jobs, AWS EMR steps, Airflow, Dagster, or a Kubernetes `CronJob`. Pick a scheduler that fits your existing operational stack.
* **Reference.** See the [Iceberg Spark procedures](https://iceberg.apache.org/docs/latest/spark-procedures/#metadata-management) documentation for the full parameter list, including options for partial rewrites (`where`), file-size targets, and merge-on-read delete file compaction.

For the next steps, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog).


# S3Table for Iceberg
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/s3table/iceberg



This guide describes how to prepare an AWS S3Table catalog for use with StreamNative Ursa as an Iceberg catalog.

> **Important:** The StreamNative Ursa cluster must run in the **same region** as the S3Table bucket. Cross-region access is not supported and will fail with an AWS client error.

## Prerequisites

* An AWS account with permissions to create S3Table buckets and modify IAM roles
* A StreamNative Ursa cluster (the cluster's IAM role will be granted access to the S3Table bucket)

## 1. Create an S3Table Bucket

In the AWS S3 console, create an **S3Table bucket** in the same region as the Ursa cluster. Record the bucket ARN, which has the form:

```
arn:aws:s3tables:<region>:<account-id>:bucket/<bucket-name>
```

## 2. Grant S3Table Permissions via a Table Bucket Policy

Apply a [table bucket policy](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-bucket-policy.html) on the S3Table bucket so that the StreamNative Ursa cluster's IAM role can read and write tables in the bucket.

### 2.1 Find the StreamNative cluster role ARN

The role ARN to grant access to is the IAM role bound to the broker pods of the StreamNative cluster. When you enable Lakehouse Table for **Amazon S3 Tables** at the cluster, namespace, or topic level in the StreamNative Cloud Console, the dialog displays the exact IAM role ARN that needs access to your S3Table bucket. Copy that ARN.

For details on enabling S3 Tables, see [Enable Lakehouse Table](/cloud/lakehouse/enable-lakehouse-integration).

### 2.2 Apply the table bucket policy

In the AWS S3 console, open your S3Table bucket and go to the **Permissions** tab. Choose **Edit** under **Table bucket policy** and paste the following JSON, replacing `<streamnative-cluster-role-arn>` with the ARN you copied in the previous step and `<your-region>`, `<your-aws-account>`, and `<your-bucket-name>` with the values for your bucket.

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3ListTableBucket",
      "Effect": "Allow",
      "Principal": {
        "AWS": "<streamnative-cluster-role-arn>"
      },
      "Action": [
        "s3tables:ListTableBuckets"
      ],
      "Resource": ["*"]
    },
    {
      "Sid": "DataAccessPermissionsForS3TableBucket",
      "Effect": "Allow",
      "Principal": {
        "AWS": "<streamnative-cluster-role-arn>"
      },
      "Action": [
        "s3tables:GetTableBucket",
        "s3tables:CreateNamespace",
        "s3tables:GetNamespace",
        "s3tables:ListNamespaces",
        "s3tables:CreateTable",
        "s3tables:GetTable",
        "s3tables:ListTables",
        "s3tables:UpdateTableMetadataLocation",
        "s3tables:GetTableMetadataLocation",
        "s3tables:GetTableData",
        "s3tables:PutTableData"
      ],
      "Resource": [
        "arn:aws:s3tables:<your-region>:<your-aws-account>:bucket/<your-bucket-name>",
        "arn:aws:s3tables:<your-region>:<your-aws-account>:bucket/<your-bucket-name>/*"
      ]
    }
  ]
}
```

The policy panel is on the **Permissions** tab of the table bucket, as shown below:

<img alt="Set the S3 Table bucket policy" />

<img alt="Configuration example" />

<img alt="Configuration example" />

<img alt="IAM permissions example" />

## 3. Adjust S3Table Maintenance Settings (Recommended)

By default, S3 Tables retains snapshots for 120 hours (5 days). For production workloads, StreamNative recommends shortening this window to **6 hours**.

| Setting              | Default            | Recommended |
| -------------------- | ------------------ | ----------- |
| `MaximumSnapshotAge` | 120 hours (5 days) | 6 hours     |

**Why this matters.** S3 Tables enforces a fixed 5 MB cap on the Iceberg metadata file size. Long retention windows cause many unexpired snapshots to accumulate in the metadata file. When the metadata file grows beyond 5 MB, **subsequent snapshot commits will fail**, which stops new data from being written into the lakehouse table. A shorter `MaximumSnapshotAge` keeps the metadata file under the cap.

You can update the snapshot management settings from the S3 Table bucket console or via the AWS CLI after the bucket is created. See [S3 Tables maintenance -- Snapshot management](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-maintenance.html#s3-tables-maintenance-snapshot) for the full list of maintenance settings.

## 4. (Optional) Configure AWS Athena Access

If you intend to query the S3Table data via AWS Athena, you must integrate it with AWS Lake Formation.

### 4.1 Prerequisites

Refer to [S3 Tables integration prerequisites](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-aws.html#table-integration-prerequisites). In summary:

* Attach `AWSLakeFormationDataAdmin` to your IAM principal
* Add `glue:PassConnection` and `lakeformation:RegisterResource` permissions
* Use the latest version of the AWS CLI

When the S3Table bucket is created with AWS analytics services enabled, AWS automatically provisions a role (for example, `S3TablesRoleForLakeFormation_1`).

<img alt="S3Tables role" />

Add the following inline policy to your IAM principal to grant Lake Formation operations:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AWSLakeFormationDataAdminAllow",
      "Effect": "Allow",
      "Action": [
        "lakeformation:*",
        "cloudtrail:DescribeTrails",
        "cloudtrail:LookupEvents",
        "glue:CreateCatalog", "glue:UpdateCatalog", "glue:DeleteCatalog",
        "glue:GetCatalog", "glue:GetCatalogs",
        "glue:GetDatabase", "glue:GetDatabases",
        "glue:CreateDatabase", "glue:UpdateDatabase", "glue:DeleteDatabase",
        "glue:GetConnections", "glue:SearchTables",
        "glue:GetTable", "glue:CreateTable", "glue:UpdateTable", "glue:DeleteTable",
        "glue:GetTableVersions", "glue:GetPartitions", "glue:GetTables",
        "glue:ListWorkflows", "glue:BatchGetWorkflows", "glue:DeleteWorkflow",
        "glue:GetWorkflowRuns", "glue:StartWorkflowRun", "glue:GetWorkflow",
        "glue:PassConnection",
        "s3:ListBucket", "s3:GetBucketLocation", "s3:ListAllMyBuckets", "s3:GetBucketAcl",
        "iam:ListUsers", "iam:ListRoles", "iam:GetRole", "iam:GetRolePolicy"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AWSLakeFormationDataAdminDeny",
      "Effect": "Deny",
      "Action": ["lakeformation:PutDataLakeSettings"],
      "Resource": "*"
    }
  ]
}
```

### 4.2 Create a Resource Link

A resource link provides a Glue Data Catalog reference to your S3Table namespace. See [Creating a resource link](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-aws.html#database-link-tables).

```bash theme={null}
aws glue create-database --region <region> --catalog-id "<account-id>" --database-input '{
  "Name": "<resource-link-name>",
  "TargetDatabase": {
    "CatalogId": "<account-id>:s3tablescatalog/<table-bucket-name>",
    "DatabaseName": "<namespace>"
  },
  "CreateTableDefaultPermissions": []
}'
```

### 4.3 Grant Lake Formation Permissions

#### On the Table

In the AWS Lake Formation console, navigate to **Data permissions -> Grant** and configure:

1. **Principals**: the IAM user, role, or SAML group that will run queries.
2. **LF-Tags or catalog resources**: choose **Named Data Catalog resources**.
3. **Catalogs**: the Glue Data Catalog created when the table bucket was integrated (`<account-id>:s3tablescatalog/<table-bucket-name>`).
4. **Databases**: the S3Table namespace.
5. **Tables**: the S3Table table.
6. **Table permissions**: **Super**.

<img alt="Grant table permission" />

<img alt="Grant table permission" />

#### On the Resource Link

Resource links require their own grants in addition to the underlying namespace/table grants:

1. **Principals**: the IAM user/role/group.
2. **Catalogs**: your account's default catalog.
3. **Databases**: the resource link from step 4.2.
4. **Resource link permissions**: **Describe**.

<img alt="Grant resource link permission" />

<img alt="Athena query" />

## Catalog Information Summary

When the steps above are complete, collect the following values for the StreamNative Ursa compaction service:

| Value                         | Description                                                                                                                                                                                                             |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `iceberg.catalog-backend`     | `s3table`                                                                                                                                                                                                               |
| `iceberg.uri`                 | `https://s3tables.<region>.amazonaws.com/iceberg` (region-specific endpoint -- see the [S3 Tables regions list](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-regions-quotas.html#s3-tables-regions)) |
| `iceberg.warehouse`           | The S3Table bucket ARN: `arn:aws:s3tables:<region>:<account>:bucket/<bucket-name>`                                                                                                                                      |
| `iceberg.rest.signing-region` | The region of the S3Table bucket                                                                                                                                                                                        |

For the next steps, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog).


# Unity Catalog for Delta Lake on AWS
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/unity-catalog/delta-lake/aws



This guide describes how to prepare a Databricks Unity Catalog for use with StreamNative Ursa as a Delta Lake catalog on AWS.

## Prerequisites

* An AWS account with permissions to create S3 buckets and IAM roles
* A Databricks account with permissions to create workspaces

## 1. Create a Databricks Workspace

> Skip this step if you already have the Databricks Workspace

In the Databricks account console, create a new workspace. The workspace creation flow uses an AWS CloudFormation stack, so you must be logged into AWS in the same browser session.

<img alt="Workspace list" />

Click **Create workspace**.

<img alt="Create workspace" />

Choose **Quickstart**.

<img alt="Quickstart option" />

Enter a workspace name and select the AWS region in which your S3 bucket resides (for example, `us-east-2`). Click **Start Quickstart**.

<img alt="Workspace settings" />

In the AWS console, acknowledge the IAM resource creation and click **Create Stack**.

<img alt="Create CloudFormation stack" />

<img alt="Stack creating" />

When the stack reaches `CREATE_COMPLETE`, return to the Databricks console and open the workspace.

<img alt="Stack complete" />

<img alt="Workspace ready" />

<img alt="Unity Catalog console" />

## 2. (Recommend) Generate an OAuth2 Service Principal

If you prefer OAuth2 over a personal access token, create a service principal:

Navigate to **Developer -> Identity and access -> Service principals -> Manage**.

<img alt="Service principals menu" />

Click **Add service principal -> Add new**, give it a name, and click **Add**.

<img alt="Add service principal" />

Open the service principal, click **Secrets -> Generate secret**, choose an expiration period, and **Generate**.

<img alt="Generate secret" />

Record the **Client ID** and **Client Secret** -- the secret cannot be retrieved later.

<img alt="Generated credentials" />

## 3. (Alternative) Generate a User Token

A Databricks user token can be used by StreamNative Ursa to authenticate against Unity Catalog.

Open **User Settings**.

<img alt="User settings" />

Navigate to **Developer -> Access tokens -> Manage** and generate a new token. Record the token value -- it cannot be retrieved later.

<img alt="Developer settings" />

<img alt="Access tokens management" />

<img alt="Create token" />

## 4. Configure Unity Catalog Access

Navigate to **Catalog -> Settings -> Metastore**.

<img alt="Catalog settings" />

Enable **External data access** on the metastore.

<img alt="Enable external data access" />

Grant privileges on the catalog with the following settings:

* **Principal:** All accounts (or the specific user/service principal)
* **Privilege presets:** Data Editor (selects related privileges automatically)
* **EXTERNAL USE SCHEMA:** Enabled

<img alt="Grant privileges" />

<img alt="Privilege settings" />

If you use OAuth2 authentication, set the **Principal** to the service principal name created in step 3.

<img alt="OAuth2 privileges" />

## 5. Create an S3 Bucket

In your AWS account, create an S3 bucket for the Unity Catalog managed location (for example, `delta-unity-catalog-bucket`).

<img alt="S3 bucket" />

## 6. Create an IAM Policy

Navigate to **AWS IAM -> Policies -> Create policy**, choose JSON, and paste the following (replace `<your-bucket>`):

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:DeleteObject",
        "s3:DeleteObjectVersion"
      ],
      "Resource": "arn:aws:s3:::<your-bucket>/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": "arn:aws:s3:::<your-bucket>",
      "Condition": {
        "StringLike": {
          "s3:prefix": ["*"]
        }
      }
    }
  ]
}
```

<img alt="Create policy" />

<img alt="Policy JSON" />

<img alt="Save policy" />

## 7. Create an IAM Role

Navigate to **AWS IAM -> Roles -> Create role** and configure:

* **Trusted entity type:** AWS account
* **An AWS account:** This account
* **Enable External ID** with placeholder value `0000` (will be updated in step 9)

<img alt="Create role" />

<img alt="Trust settings" />

Attach the policy from step 6.

<img alt="Attach policy" />

<img alt="Save role" />

Record the role ARN (for example, `arn:aws:iam::<account-id>:role/<role-name>`).

<img alt="Role ARN" />

## 8. Create a Storage Credential in Unity Catalog

Navigate to **Catalog -> Settings -> Credentials**.

<img alt="Credentials menu" />

<img alt="Create credential" />

Configure with:

* **Credential:** Storage Credential
* **Type:** AWS IAM Role
* **Name:** any name
* **Role ARN:** the ARN recorded in step 7

<img alt="Credential form" />

Databricks generates a trust relationship policy. Copy it.

<img alt="Trust policy generated" />

## 9. Update the IAM Role Trust Policy

Return to the AWS IAM console, open the role created in step 7, and replace the trust policy with the one generated by Databricks.

<img alt="Update trust policy" />

Click **Validate** in the Unity Catalog console to verify the credential.

<img alt="Validate credential" />

## 10. Create an External Location

Navigate to **Catalog -> Settings -> External Locations**.

<img alt="External locations" />

<img alt="Create external location" />

Choose **Manual** (the AWS Quickstart creates a new bucket).

<img alt="Manual external location" />

Configure:

* **External location name:** any name
* **URL:** `s3://<your-bucket>`
* **Storage credential:** the credential from step 8

<img alt="External location form" />

After creation, click **Test connection** to verify access.

<img alt="Test external location" />

If you use OAuth2, grant **ALL PRIVILEGES** on the external location to the service principal:

<img alt="External location details" />

<img alt="Grant OAuth2 permissions" />

## 11. Create the Catalog

In Databricks, create a new catalog and bind it to the external location created in step 10.

<img alt="Create catalog" />

## Catalog Information Summary

When the steps above are complete, collect the following values for the StreamNative Ursa compaction service:

| Value                                               | Description                                                              |
| --------------------------------------------------- | ------------------------------------------------------------------------ |
| `unityCatalogUri`                                   | Databricks workspace URL (e.g., `https://dbc-xxxx.cloud.databricks.com`) |
| `unityCatalogName`                                  | The Unity Catalog name created in step 11                                |
| `unityCatalogToken`                                 | Personal access token from step 2, **or**                                |
| `unityCatalogClientId` / `unityCatalogClientSecret` | OAuth2 credentials from step 3                                           |

For the next steps, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog).


# Unity Catalog for Delta Lake on Azure
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/unity-catalog/delta-lake/azure



This guide describes how to prepare a Databricks Unity Catalog for use with StreamNative Ursa as a Delta Lake catalog on Microsoft Azure.

## Prerequisites

* An Azure subscription with permissions to create storage accounts and Access Connectors
* A Databricks workspace on Azure

## 1. Create an Access Connector for Azure Databricks

In the Azure Marketplace, search for **Access Connector for Azure Databricks** and click **Create**.

<img alt="Search Access Connector" />

Choose the resource group, provide a connector name (for example, `unity-catalog-access-connector`), and click **Next**.

<img alt="Connector configuration" />

In the **Managed Identity** panel, enable **System assigned identity**, then click **Next** -> **Create**.

<img alt="Enable managed identity" />

<img alt="Connector created" />

Record the connector **Resource ID**:

<img alt="Connector Resource ID" />

## 2. Grant `Storage Blob Data Contributor` to the Connector

Open the storage account that will hold the Delta tables, navigate to **Access Control (IAM) -> Add -> Add role assignment**.

<img alt="Access control" />

Search for and select **Storage Blob Data Contributor**, then click **Next**.

<img alt="Select Blob Data Contributor" />

Choose **Managed identity** and select the Access Connector created in step 1.

<img alt="Select members" />

Click **Next -> Review + assign**.

<img alt="Role assigned" />

## 3. Grant `Storage Queue Data Contributor` to the Connector

Repeat the process from step 2 with the **Storage Queue Data Contributor** role.

<img alt="Queue Data Contributor" />

Both roles are now assigned to the Access Connector.

<img alt="Both roles assigned" />

## 4. Create a Storage Credential in Unity Catalog

In the Databricks Catalog console, navigate to **Catalog -> Settings -> Credentials**.

<img alt="Credentials menu" />

Click **Create Credential**, provide a name, and paste the Access Connector **Resource ID** from step 1.

<img alt="Create credential" />

<img alt="Credential created" />

## 5. Create an External Location

In the Databricks Catalog console, create a new external location.

<img alt="External locations" />

Configure with:

* **Storage type:** Azure Data Lake Storage
* **URL:** `abfss://<container>@<storage-account>.dfs.core.windows.net`
* **Storage credential:** the credential created in step 4

<img alt="External location form" />

<img alt="External location created" />

Click **Test Connection** to verify the credential.

<img alt="Test connection" />

> **Troubleshooting:** If the test fails with a `Hierarchical Namespace Enabled` error, ensure that **Hierarchical namespace** is enabled on the storage account.

<img alt="Hierarchical namespace" />

<img alt="Hierarchical namespace" />

## 6. Create a Service Principal

Navigate to **User -> Settings -> Identity and access -> Service principals -> Manage**.

<img alt="Service principals" />

Click **Add service principal -> Add new**.

<img alt="Add service principal" />

Choose **Databricks managed** and provide a name.

<img alt="Name service principal" />

Open the service principal, click **Secrets**, choose an expiration period, and **Generate**.

<img alt="Generate secret" />

Record both the **Client ID** and **Client Secret** -- the secret cannot be retrieved later.

<img alt="Secret and Client ID" />

## 7. Create the Catalog

Create a new Catalog with **Type: Standard** and select the **storage location** created in step 5.

<img alt="Create catalog" />

<img alt="Catalog form" />

## 8. Grant Permissions to the Service Principal

### 8.1 Catalog Permissions

Navigate to the new catalog and click **Permissions -> Grant**.

<img alt="Catalog permissions" />

Configure:

* **Principals:** the service principal from step 6
* **Privilege presets:** Data Editor
* **EXTERNAL USE SCHEMA:** Enabled

<img alt="Grant catalog permissions" />

<img alt="Permissions granted" />

### 8.2 External Location Permissions

Open the external location from step 5.

<img alt="External location details" />

<img alt="External location details" />

Click **Grant**, choose the service principal, select **ALL PRIVILEGES**, and click **Confirm**.

<img alt="Grant external location permission" />

<img alt="Permission granted" />

## Catalog Information Summary

When the steps above are complete, collect the following values for the StreamNative Ursa compaction service:

| Value                                               | Description                                                                       |
| --------------------------------------------------- | --------------------------------------------------------------------------------- |
| `unityCatalogUri`                                   | Databricks workspace URL (e.g., `https://adb-<workspace-id>.azuredatabricks.net`) |
| `unityCatalogName`                                  | The Unity Catalog name created in step 7                                          |
| `unityCatalogClientId` / `unityCatalogClientSecret` | OAuth2 credentials from step 6                                                    |

For the next steps, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog).


# Unity Catalog for Delta Lake on GCP
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/unity-catalog/delta-lake/gcp



This guide describes how to prepare a Databricks Unity Catalog for use with StreamNative Ursa as a Delta Lake catalog on Google Cloud Platform (GCP).

## Prerequisites

* A GCP project with permissions to create GCS buckets and IAM roles
* A Databricks account with permissions to create workspaces

## 1. Create a Databricks Workspace

> Skip this step if you already have the Databricks Workspace in GCP

In the GCP Databricks account console, click **Create workspace**.

<img alt="Create workspace" />

Enter the workspace name, choose the region, and provide your GCP project ID.

<img alt="Workspace configuration" />

Click **Save**. The workspace status shows **Provisioning** while initialization is in progress.

<img alt="Workspace provisioning" />

When the status changes to **Running**, the workspace is ready.

<img alt="Workspace running" />

Open the workspace to enter the Unity Catalog console.

<img alt="Unity Catalog console" />

## 2. (Recommend) Generate an OAuth2 Service Principal

For OAuth2 authentication, navigate to **Identity and access -> Service principals -> Manage**.

<img alt="Service principals" />

Click **Add service principal -> Add new** and provide a name.

<img alt="Add service principal" />

<img alt="Create service principal" />

<img alt="Service principal created" />

Open the service principal, click **Secrets -> Generate secret**, choose an expiration period, and **Generate**.

<img alt="Generate secret" />

Record both the **Client ID** and **Client Secret** -- the secret cannot be retrieved later.

<img alt="Client ID and Secret" />

## 3. (Alternative) Generate a User Token

A Databricks user token can be used by StreamNative Ursa to authenticate against Unity Catalog.

Open **User Settings**.

<img alt="User settings" />

Navigate to **Developer -> Access tokens -> Manage** and generate a new token. Record the token value -- it cannot be retrieved later.

<img alt="Developer menu" />

<img alt="Access tokens" />

<img alt="Generate token" />

## 4. Configure Unity Catalog Access

Navigate to **Catalog -> Settings -> Metastore**.

<img alt="Catalog settings" />

Enable **External data access** on the metastore.

<img alt="External data access" />

Grant catalog privileges with the following settings:

* **Principal:** All accounts (or the specific user/service principal)
* **Privilege presets:** Data Editor (selects related privileges automatically)
* **EXTERNAL USE SCHEMA:** Enabled

<img alt="Grant privileges" />

<img alt="Privilege configuration" />

## 5. Grant Bucket Permissions to the Databricks Service Account

When the Databricks workspace is initialized, a service account is created for Unity Catalog.

Navigate to **Catalog -> Settings -> Credentials** to find the service account.

<img alt="Credentials menu" />

Example service account name:

```
db-uc-credential-<id>@uc-uswest1.iam.gserviceaccount.com
```

<img alt="Databricks service account" />

### 5.1 Create a Custom IAM Role

In the GCP console, navigate to **IAM & Admin -> Roles -> Create role** and add the following permissions:

* `storage.buckets.get`
* `storage.objects.create`
* `storage.objects.delete`
* `storage.objects.get`
* `storage.objects.list`

<img alt="Create role" />

<img alt="Role setup" />

<img alt="Permissions" />

<img alt="Permissions selected" />

### 5.2 Assign the Role to the Databricks Service Account

Open your bucket, click **PERMISSIONS -> View BY PRINCIPALS -> GRANT ACCESS**.

<img alt="Grant bucket access" />

Add the Databricks service account, select the role created in step 6.1, and click **SAVE**.

<img alt="Save access" />

## 6. Create an External Location in Unity Catalog

Navigate to **Catalog -> Settings -> External Locations** and create a new external location.

<img alt="External locations" />

<img alt="Create external location" />

Configure with:

* **External location name:** any name
* **URL:** the GCS bucket path
* **Storage credential:** the Unity Catalog credential

<img alt="External location form" />

Click **Test connection** to verify access.

<img alt="External location created" />

Grant **ALL PRIVILEGES** on the external location to the service principal.

<img alt="Grant OAuth2 permissions" />

## Catalog Information Summary

When the steps above are complete, collect the following values for the StreamNative Ursa compaction service:

| Value                                               | Description                                                               |
| --------------------------------------------------- | ------------------------------------------------------------------------- |
| `unityCatalogUri`                                   | Databricks workspace URL (e.g., `https://<workspace>.gcp.databricks.com`) |
| `unityCatalogName`                                  | The Unity Catalog name                                                    |
| `unityCatalogToken`                                 | Personal access token from step 2, **or**                                 |
| `unityCatalogClientId` / `unityCatalogClientSecret` | OAuth2 credentials from step 3                                            |

For the next steps, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog).


# Unity Catalog Managed Iceberg Table on AWS
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/unity-catalog/iceberg/aws



This guide describes how to prepare a Databricks Unity Catalog for use with StreamNative Ursa as a managed Iceberg table catalog on AWS.

## Prerequisites

* A Databricks account with Unity Catalog and Iceberg Managed Table enabled
* An AWS account with permissions to create S3 buckets and IAM roles

## 1. Create an S3 Bucket

In your AWS account, create an S3 bucket to use as the Unity Catalog storage location (for example, `aws-unitycatalog-iceberg-bucket`).

<img alt="Create S3 bucket" />

## 2. Create the IAM Role

### 2.1 Create the Role with a Placeholder Trust Policy

Create an IAM role that allows the Unity Catalog master role to assume it. Use the following trust policy with a placeholder `External ID` of `0000` (you will replace it with the value generated by Databricks in step 3):

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": ["arn:aws:iam::414351767826:role/unity-catalog-prod-UCMasterRole-14S5ZJVKOTYTL"]
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "0000"
        }
      }
    }
  ]
}
```

Skip the permissions policy on this screen -- it will be added in the next steps.

<img alt="Create IAM role" />

<img alt="Save IAM role" />

### 2.2 Attach the S3 Access Policy

Create the following policy and attach it to the role. Replace `<your-bucket>` and `<your-account-id>` with your values.

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads",
        "s3:ListMultipartUploadParts",
        "s3:AbortMultipartUpload"
      ],
      "Resource": [
        "arn:aws:s3:::<your-bucket>/*",
        "arn:aws:s3:::<your-bucket>"
      ]
    },
    {
      "Effect": "Allow",
      "Action": ["sts:AssumeRole"],
      "Resource": ["arn:aws:iam::<your-account-id>:role/<your-role-name>"]
    }
  ]
}
```

<img alt="S3 access policy" />

### 2.3 Attach the File Events Policy

Create a second policy for managed file events (S3 notifications, SNS, SQS) and attach it to the same role:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ManagedFileEventsSetupStatement",
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketNotification",
        "s3:PutBucketNotification",
        "sns:ListSubscriptionsByTopic",
        "sns:GetTopicAttributes",
        "sns:SetTopicAttributes",
        "sns:CreateTopic",
        "sns:TagResource",
        "sns:Publish",
        "sns:Subscribe",
        "sqs:CreateQueue",
        "sqs:DeleteMessage",
        "sqs:ReceiveMessage",
        "sqs:SendMessage",
        "sqs:GetQueueUrl",
        "sqs:GetQueueAttributes",
        "sqs:SetQueueAttributes",
        "sqs:TagQueue",
        "sqs:ChangeMessageVisibility",
        "sqs:PurgeQueue"
      ],
      "Resource": [
        "arn:aws:s3:::<your-bucket>",
        "arn:aws:sqs:*:*:csms-*",
        "arn:aws:sns:*:*:csms-*"
      ]
    },
    {
      "Sid": "ManagedFileEventsListStatement",
      "Effect": "Allow",
      "Action": [
        "sqs:ListQueues",
        "sqs:ListQueueTags",
        "sns:ListTopics"
      ],
      "Resource": [
        "arn:aws:sqs:*:*:csms-*",
        "arn:aws:sns:*:*:csms-*"
      ]
    },
    {
      "Sid": "ManagedFileEventsTeardownStatement",
      "Effect": "Allow",
      "Action": [
        "sns:Unsubscribe",
        "sns:DeleteTopic",
        "sqs:DeleteQueue"
      ],
      "Resource": [
        "arn:aws:sqs:*:*:csms-*",
        "arn:aws:sns:*:*:csms-*"
      ]
    }
  ]
}
```

<img alt="File events policy" />

Verify that both policies are attached to the role.

<img alt="Attach policies to role" />

## 3. Create an External Location in Unity Catalog

In the Databricks Catalog console, create a new external location pointing to the S3 bucket and the IAM role created above.

<img alt="Create external location" />

<img alt="External location settings" />

<img alt="External location summary" />

When you submit the form, Databricks generates an **External ID** and a trust policy. Copy these values.

<img alt="Generated External ID" />

## 4. Update the IAM Role Trust Policy

Return to the AWS IAM console and replace the role's trust policy with the one generated by Databricks in step 3, using the new External ID.

<img alt="Update trust policy" />

<img alt="Trust policy applied" />

After saving the trust policy, click **IAM role configured** in the Databricks catalog console and then **Test connection** to verify the credential.

<img alt="Test connection" />

## 5. Create the Unity Catalog

Create a new catalog in Databricks bound to the external location created in step 3:

* **Type:** Standard
* **Storage location:** the external location created above

<img alt="Create catalog" />

<img alt="Select Standard type" />

<img alt="Select external location" />

## 6. Grant Catalog Permissions

Grant permissions on the catalog. The `EXTERNAL_USE_SCHEMA` permission is **required** for Iceberg Managed Tables in Unity Catalog.

<img alt="Grant permissions" />

<img alt="EXTERNAL_USE_SCHEMA permission" />

## 7. Create OAuth2 Credentials

Create an OAuth2 service principal that StreamNative Ursa will use to authenticate against Unity Catalog.

<img alt="OAuth2 setup" />

<img alt="OAuth2 setup" />

<img alt="OAuth2 setup" />

Generate a secret for the principal and record both the **Client ID** and **Client Secret**.

<img alt="Generate secret" />

## 8. Enable External Data Access on the Metastore

This step is **required** for Unity Catalog Iceberg Managed Tables. Enable **External data access** on the metastore in Databricks.

<img alt="Enable external data access" />

<img alt="External data access enabled" />

## Catalog Information Summary

When the steps above are complete, collect the following values for the StreamNative Ursa compaction service:

| Value      | Description                                                                                                 |
| ---------- | ----------------------------------------------------------------------------------------------------------- |
| URI        | Databricks workspace URL (e.g., `https://dbc-xxxx.cloud.databricks.com/api/2.1/unity-catalog/iceberg-rest`) |
| Warehouse  | The Unity Catalog name created in step 5                                                                    |
| Credential | `<client-id>:<client-secret>` from step 7                                                                   |

For the next steps, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog).


# Unity Catalog Managed Iceberg Table on Azure
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/unity-catalog/iceberg/azure



This guide describes how to prepare a Databricks Unity Catalog for use with StreamNative Ursa as a managed Iceberg table catalog on Microsoft Azure.

## Prerequisites

* A Databricks workspace on Azure with Unity Catalog and Iceberg Managed Table enabled
* An Azure subscription with permissions to create storage accounts and Access Connectors

## 1. Create an Azure Storage Container

Create a storage container in your Azure Storage Account (for example, `unity-catalog-iceberg`). The container path will follow the format:

```
abfss://<container>@<storage-account>.dfs.core.windows.net
```

<img alt="Create storage container" />

<img alt="Storage container" />

## 2. Create an Access Connector for Azure Databricks

Refer to the [Azure Databricks Managed Identities documentation](https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/cloud-storage/azure-managed-identities) for the canonical procedure.

In the Azure Portal, create an **Access Connector for Azure Databricks**.

<img alt="Access Connector" />

<img alt="Access Connector settings" />

<img alt="Access Connector created" />

Record the connector **Resource ID**, which has the form:

```
/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Databricks/accessConnectors/<connector-name>
```

## 3. Grant Storage Permissions to the Access Connector

The Access Connector identity requires the following roles:

| Scope           | Role                                      |
| --------------- | ----------------------------------------- |
| Storage Account | `Storage Blob Data Contributor`           |
| Storage Account | `Storage Queue Data Contributor`          |
| Resource Group  | `EventGrid EventSubscription Contributor` |

### 3.1 Grant `Storage Blob Data Contributor`

<img alt="Grant Blob Data Contributor" />

<img alt="Grant Blob Data Contributor" />

<img alt="Grant Blob Data Contributor" />

<img alt="Grant Blob Data Contributor" />

### 3.2 Grant `Storage Queue Data Contributor`

<img alt="Grant Queue Data Contributor" />

<img alt="Grant Queue Data Contributor" />

<img alt="Grant Queue Data Contributor" />

### 3.3 Grant `EventGrid EventSubscription Contributor`

<img alt="Grant EventGrid Contributor" />

<img alt="Grant EventGrid Contributor" />

<img alt="Grant EventGrid Contributor" />

<img alt="Grant EventGrid Contributor" />

## 4. Create the Unity Catalog Metastore

Create the Unity Catalog metastore in Databricks.

<img alt="Create metastore" />

<img alt="Metastore configuration" />

<img alt="Metastore created" />

## 5. Create a Storage Credential

In the Databricks Catalog console, create a storage credential linked to the Access Connector created in step 2.

<img alt="Create credential" />

<img alt="Credential form" />

## 6. Create the External Location

Create an external location pointing to the Azure storage container:

* **URL:** `abfss://<container>@<storage-account>.dfs.core.windows.net`
* **Storage credential:** the credential created in step 5

<img alt="Create external location" />

<img alt="External location settings" />

## 7. Create the Unity Catalog

Create a new Catalog and bind it to the external location created in step 6.

<img alt="Create catalog" />

<img alt="Catalog form" />

## 8. Grant Catalog Permissions

Grant permissions on the catalog. The `EXTERNAL_USE_SCHEMA` permission is **required** for Iceberg Managed Tables in Unity Catalog.

<img alt="Grant permissions" />

<img alt="EXTERNAL_USE_SCHEMA permission" />

## 9. Enable External Data Access on the Metastore

> **Note:** This action requires **Azure Account Admin** privileges; without them, the **Metastore** entry is not visible.

Enable **External data access** on the metastore. This step is **required** for Unity Catalog Iceberg Managed Tables.

<img alt="Enable external data access" />

<img alt="External data access enabled" />

## 10. Create OAuth2 Credentials

Create an OAuth2 service principal that StreamNative Ursa will use to authenticate.

<img alt="OAuth2 setup" />

<img alt="OAuth2 setup" />

<img alt="OAuth2 setup" />

Generate a secret for the principal and record both the **Client ID** and **Client Secret**.

<img alt="Generate secret" />

## Catalog Information Summary

When the steps above are complete, collect the following values for the StreamNative Ursa compaction service:

| Value      | Description                                                                                                          |
| ---------- | -------------------------------------------------------------------------------------------------------------------- |
| URI        | Databricks workspace URL (e.g., `https://adb-<workspace-id>.azuredatabricks.net/api/2.1/unity-catalog/iceberg-rest`) |
| Warehouse  | The Unity Catalog name created in step 7                                                                             |
| Credential | `<client-id>:<client-secret>` from step 10                                                                           |

For the next steps, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog).


# Unity Catalog Managed Iceberg Table on GCP
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-catalogs/unity-catalog/iceberg/gcp



This guide describes how to prepare a Databricks Unity Catalog for use with StreamNative Ursa as a managed Iceberg table catalog on Google Cloud Platform (GCP).

## Prerequisites

* A Databricks workspace on GCP with Unity Catalog and Iceberg Managed Table enabled
* A GCP project with permissions to create GCS buckets

## 1. Create a GCS Bucket

Create a GCS bucket to use as the Unity Catalog storage location (for example, `unity-catalog-iceberg-bucket`).

> **Important:** The bucket must be located in the **same region** as your Databricks workspace and your StreamNative Ursa cluster. Cross-region access introduces additional network traffic and latency.

<img alt="Create GCS bucket" />

For additional details, see the [Databricks GCP Unity Catalog documentation](https://docs.databricks.com/gcp/en/data-governance/unity-catalog/create-metastore).

## 2. Create a Storage Credential in Unity Catalog

In the Databricks Catalog console, create a new storage credential. Databricks generates a service account that needs permissions on the bucket.

<img alt="Create credential" />

<img alt="Credential form" />

After creation, record the generated service account name. Example:

```
db-uc-credential-<identifier>@uc-uswest1.iam.gserviceaccount.com
```

<img alt="Generated service account" />

## 3. Grant GCS Permissions to the Service Account

In the GCP console, navigate to the bucket's **Permissions** tab and click **Grant access**.

<img alt="Grant access" />

Grant the following roles to the service account from step 2:

* **Storage Legacy Bucket Reader**
* **Storage Object Admin**

<img alt="Assign storage roles" />

## 4. Create the External Location

In the Databricks Catalog console, create an external location with the following settings:

* **External location name:** any name
* **URL:** the GCS bucket path created in step 1
* **Storage credential:** the credential created in step 2

<img alt="Create external location" />

<img alt="External location settings" />

Use **Test connection** to verify the credential has sufficient permissions.

<img alt="Test connection" />

## 5. Create the Unity Catalog

Create a new Catalog with:

* **Type:** Standard
* **Storage location:** the external location created in step 4 (a sub-path within this location may be specified)

<img alt="Create catalog" />

<img alt="Catalog form" />

<img alt="Catalog created" />

## 6. Grant Catalog Permissions

Grant permissions on the catalog. The `EXTERNAL_USE_SCHEMA` permission is **required** for Iceberg Managed Tables in Unity Catalog.

<img alt="Grant catalog permissions" />

<img alt="EXTERNAL_USE_SCHEMA permission" />

## 7. Enable External Data Access on the Metastore

> **Note:** This action requires **Databricks Account Admin** privileges; without them, the **Metastore** entry is not visible.

Enable **External data access** on the metastore. This step is **required** for Unity Catalog Iceberg Managed Tables.

<img alt="External data access" />

<img alt="External data access enabled" />

## 8. Create OAuth2 Credentials

Create an OAuth2 service principal that StreamNative Ursa will use to authenticate.

<img alt="OAuth2 setup" />

<img alt="OAuth2 setup" />

<img alt="OAuth2 setup" />

Generate a secret for the principal and record both the **Client ID** and **Client Secret**.

<img alt="Generate secret" />

## Catalog Information Summary

When the steps above are complete, collect the following values for the StreamNative Ursa compaction service:

| Value      | Description                                                                                                  |
| ---------- | ------------------------------------------------------------------------------------------------------------ |
| URI        | Databricks workspace URL (e.g., `https://<workspace>.gcp.databricks.com/api/2.1/unity-catalog/iceberg-rest`) |
| Warehouse  | The Unity Catalog name created in step 5                                                                     |
| Credential | `<client-id>:<client-secret>` from step 8                                                                    |

For the next steps, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog).


# Prepare Lakehouse Catalogs
Source: https://docs.streamnative.io/cloud/lakehouse/prepare-lakehouse-catalogs



Preparing an external catalog is **optional**. StreamNative Ursa supports the following catalog modes:

* **Iceberg with the Hadoop catalog (no external catalog).** Iceberg metadata and data files are written directly to the configured object storage path. No external catalog service is required, and downstream engines can read the table by pointing at the storage location.
* **Delta Lake without a catalog.** Delta tables are written directly to the configured storage path; readers point at the path (`s3://...`, `gs://...`, or `abfss://...`) to query them.
* **External catalog.** Use a managed catalog service (Databricks Unity Catalog, Snowflake Open Catalog, Snowflake Horizon Catalog, AWS S3Table, or Google BigLake) to register the tables. This is required if you want governance, discoverability, or integration with managed query engines (Databricks SQL, Snowflake, Athena, BigQuery, etc.).

If you choose to use an external catalog, this page is the index for the catalog preparation guides. Each guide focuses solely on provisioning the catalog itself and its supporting cloud resources (storage bucket, IAM role or service principal, and credentials). Once the catalog is ready, see [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog) for how to wire it into the compaction service.

> **Skip this page if you are using the Hadoop catalog (Iceberg) or no catalog (Delta).** Go directly to [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog), which describes the no-catalog configuration.

## Catalog Support Matrix

| Catalog                                    | Table Format | AWS                                                                   | GCP                                                                   | Azure                                                                     |
| ------------------------------------------ | ------------ | --------------------------------------------------------------------- | --------------------------------------------------------------------- | ------------------------------------------------------------------------- |
| Databricks Unity Catalog (Managed Iceberg) | Iceberg      | [AWS](/cloud/lakehouse/prepare-catalogs/unity-catalog/iceberg/aws)    | [GCP](/cloud/lakehouse/prepare-catalogs/unity-catalog/iceberg/gcp)    | [Azure](/cloud/lakehouse/prepare-catalogs/unity-catalog/iceberg/azure)    |
| Databricks Unity Catalog (Delta Lake)      | Delta Lake   | [AWS](/cloud/lakehouse/prepare-catalogs/unity-catalog/delta-lake/aws) | [GCP](/cloud/lakehouse/prepare-catalogs/unity-catalog/delta-lake/gcp) | [Azure](/cloud/lakehouse/prepare-catalogs/unity-catalog/delta-lake/azure) |
| Snowflake Open Catalog (Polaris)           | Iceberg      | [AWS](/cloud/lakehouse/prepare-catalogs/open-catalog/iceberg/aws)     | [GCP](/cloud/lakehouse/prepare-catalogs/open-catalog/iceberg/gcp)     | [Azure](/cloud/lakehouse/prepare-catalogs/open-catalog/iceberg/azure)     |
| Snowflake Horizon Catalog                  | Iceberg      | [AWS](/cloud/lakehouse/prepare-catalogs/horizon-catalog/iceberg/aws)  | --                                                                    | --                                                                        |
| AWS S3Table                                | Iceberg      | [Iceberg](/cloud/lakehouse/prepare-catalogs/s3table/iceberg)          | --                                                                    | --                                                                        |
| Google BigLake                             | Iceberg      | --                                                                    | [Iceberg](/cloud/lakehouse/prepare-catalogs/biglake/iceberg)          | --                                                                        |

## Choosing a Catalog

| Use case                                                            | Recommended catalog                                                        |
| ------------------------------------------------------------------- | -------------------------------------------------------------------------- |
| Manage Iceberg tables alongside Databricks workloads                | [Unity Catalog (Iceberg)](#databricks-unity-catalog-managed-iceberg-table) |
| Manage Delta Lake tables alongside Databricks workloads             | [Unity Catalog (Delta Lake)](#databricks-unity-catalog-delta-lake)         |
| Cloud-portable Iceberg REST catalog from Snowflake                  | [Snowflake Open Catalog (Polaris)](#snowflake-open-catalog-polaris)        |
| Governed Iceberg tables managed by Snowflake Horizon                | [Snowflake Horizon Catalog](#snowflake-horizon-catalog)                    |
| AWS-native Iceberg tables with built-in Athena/Redshift integration | [AWS S3Table](#aws-s3table)                                                |
| GCP-native Iceberg with BigQuery/BigLake integration                | [Google BigLake](#google-biglake)                                          |

## Catalog Preparation Guides

### Databricks Unity Catalog (Managed Iceberg Table)

Use Unity Catalog to manage Iceberg tables governed by Databricks. The compaction service writes to Iceberg Managed Tables; downstream readers query them via the Unity Catalog Iceberg REST endpoint.

| Cloud | Guide                                                                                                         |
| ----- | ------------------------------------------------------------------------------------------------------------- |
| AWS   | [Unity Catalog Managed Iceberg Table on AWS](/cloud/lakehouse/prepare-catalogs/unity-catalog/iceberg/aws)     |
| GCP   | [Unity Catalog Managed Iceberg Table on GCP](/cloud/lakehouse/prepare-catalogs/unity-catalog/iceberg/gcp)     |
| Azure | [Unity Catalog Managed Iceberg Table on Azure](/cloud/lakehouse/prepare-catalogs/unity-catalog/iceberg/azure) |

### Databricks Unity Catalog (Delta Lake)

Use Unity Catalog to manage Delta Lake tables. The compaction service writes Delta Lake files governed by Unity Catalog and queryable from Databricks SQL or external Spark sessions.

| Cloud | Guide                                                                                                     |
| ----- | --------------------------------------------------------------------------------------------------------- |
| AWS   | [Unity Catalog for Delta Lake on AWS](/cloud/lakehouse/prepare-catalogs/unity-catalog/delta-lake/aws)     |
| GCP   | [Unity Catalog for Delta Lake on GCP](/cloud/lakehouse/prepare-catalogs/unity-catalog/delta-lake/gcp)     |
| Azure | [Unity Catalog for Delta Lake on Azure](/cloud/lakehouse/prepare-catalogs/unity-catalog/delta-lake/azure) |

### Snowflake Open Catalog (Polaris)

Snowflake Open Catalog (Polaris) is a cloud-agnostic Iceberg REST catalog operated by Snowflake. It can be used as the catalog for Iceberg tables hosted on AWS, GCP, or Azure object storage.

| Cloud | Guide                                                                                             |
| ----- | ------------------------------------------------------------------------------------------------- |
| AWS   | [Open Catalog for Iceberg on AWS](/cloud/lakehouse/prepare-catalogs/open-catalog/iceberg/aws)     |
| GCP   | [Open Catalog for Iceberg on GCP](/cloud/lakehouse/prepare-catalogs/open-catalog/iceberg/gcp)     |
| Azure | [Open Catalog for Iceberg on Azure](/cloud/lakehouse/prepare-catalogs/open-catalog/iceberg/azure) |

### Snowflake Horizon Catalog

Snowflake Horizon provides governed Iceberg tables with native Snowflake integration. The catalog uses a Snowflake **External Volume** backed by an S3 bucket and authenticates via a Programmatic Access Token (PAT).

| Cloud | Guide                                                                                               |
| ----- | --------------------------------------------------------------------------------------------------- |
| AWS   | [Horizon Catalog for Iceberg on AWS](/cloud/lakehouse/prepare-catalogs/horizon-catalog/iceberg/aws) |

### AWS S3Table

AWS S3Table is the AWS-native Iceberg catalog with first-class integration into AWS analytics services such as Athena, Redshift, and EMR.

| Cloud | Guide                                                                    |
| ----- | ------------------------------------------------------------------------ |
| AWS   | [S3Table for Iceberg](/cloud/lakehouse/prepare-catalogs/s3table/iceberg) |

> **Important:** The Ursa cluster must run in the **same region** as the S3Table bucket. Cross-region access is not supported.

### Google BigLake

Google BigLake provides an Iceberg REST catalog tightly integrated with BigQuery and Google Cloud Storage.

| Cloud | Guide                                                                    |
| ----- | ------------------------------------------------------------------------ |
| GCP   | [BigLake for Iceberg](/cloud/lakehouse/prepare-catalogs/biglake/iceberg) |

> **Important:** The Ursa cluster, GCS bucket, and BigLake catalog must all be in the same region. Each BigLake catalog maps to exactly one GCS bucket (1:1 mapping; sub-paths are not supported).

## Next Steps

After preparing your catalog, proceed to:

1. [Dynamic Configuration Guide](/cloud/lakehouse/dynamic-configuration) -- Reference for all dynamic configuration keys and the cluster-name prefix requirement.
2. [Register Lakehouse Catalogs](/cloud/lakehouse/catalogs/register-catalog) -- Connect the prepared catalog to the StreamNative Ursa compaction service.
3. [Enable Lakehouse Integration](/cloud/lakehouse/enable-lakehouse-integration) -- Enable SDT (External Table) at the cluster, namespace, or topic level.


# Migrate to StreamNative Kafka Service
Source: https://docs.streamnative.io/kafka/kafka-migration-guide

Migrate from Amazon MSK, Confluent Cloud, or self-managed Apache Kafka to StreamNative Kafka Service with zero code changes.

StreamNative Kafka Service runs **native Apache Kafka** — it is not a compatibility layer. Migration requires changing your bootstrap servers endpoint. Your client code, Kafka Connect connectors, and Kafka Streams applications work unchanged.

This guide covers the general approach for migrating from any existing Kafka deployment -- including Amazon MSK, Confluent Cloud or Platform, and self-managed Apache Kafka -- to StreamNative Kafka Service.

## What changes vs. what stays the same

Understanding what changes during migration helps you plan with confidence. The short answer: very little changes.

### What changes

* **Bootstrap servers endpoint** -- You point your clients to the StreamNative Kafka Service endpoint instead of your current Kafka broker addresses.
* **Authentication method** -- StreamNative uses OAuth 2.0 or API keys for authentication. You update your client configuration to use one of these methods. See [Authentication overview](/cloud/security/authentication/authentication-overview) for details.

### What stays the same

* **Client code** -- Your producers, consumers, and admin clients require no code changes. Only configuration properties change.
* **Kafka Connect connectors** -- Your existing connector configurations work as-is after pointing to the new endpoint.
* **Kafka Streams applications** -- Your stream processing topologies run unchanged.
* **Topic names** -- Your topic naming scheme carries over directly.
* **Consumer group IDs** -- Your consumer groups and their offsets can be preserved during migration.

<Tip>
  Because StreamNative Kafka Service runs native Apache Kafka, you can validate your migration with a single test client before committing to a full cutover. Change the bootstrap servers, update authentication, and confirm your application works.
</Tip>

## Migration steps

Follow these steps to migrate your Kafka workloads to StreamNative Kafka Service.

### Step 1: Assess your current Kafka deployment

Before migrating, document your existing deployment:

* **Topics and partitions** -- List all topics, their partition counts, and replication factors.
* **Throughput** -- Measure your peak produce and consume rates (MB/s and messages/s).
* **Retention policies** -- Record retention times and sizes for each topic or namespace.
* **Consumer groups** -- Identify all active consumer groups and their current offsets.
* **Connectors** -- Inventory your Kafka Connect source and sink connectors.
* **Security configuration** -- Note your current authentication and authorization setup (SASL, TLS, ACLs).

<Tip>
  StreamNative Kafka Service runs native Apache Kafka, so the default data retention policies are the same as open-source Apache Kafka (7-day retention by default). If you are using [Kafka compatibility on Pulsar Clusters (KSN)](/cloud/build/kafka-clients/kafka-on-cloud) instead, review the [data retention differences](/cloud/build/kafka-clients/migrating-to-streamnative) as Pulsar uses different default retention behavior.
</Tip>

### Step 2: Create a StreamNative Kafka cluster

Create a Kafka cluster in StreamNative Cloud. Choose your cloud provider, region, and cluster size based on the throughput requirements you identified in Step 1.

See [Get Started with Kafka Service](/kafka/kafka-getting-started) for step-by-step instructions on creating a cluster.

### Step 3: Configure authentication

Set up authentication for your clients. StreamNative supports two authentication methods:

* **OAuth 2.0** -- Recommended for production workloads. Provides token-based authentication with automatic rotation.
* **API keys** -- Suitable for development and testing, or when OAuth 2.0 is not practical.

See [Authentication overview](/cloud/security/authentication/authentication-overview) for configuration details.

### Step 4: Set up Universal Linking for zero-downtime migration

Universal Linking mirrors data between your source Kafka deployment and StreamNative, enabling zero-downtime migration. It replicates topics, consumer group offsets, and schemas from your existing cluster to StreamNative Kafka Service through object storage.

Key capabilities of Universal Linking:

* **Offset preservation** -- Maintains consumer group offsets so consumers can resume from where they left off.
* **Schema migration** -- Replicates schemas from your source schema registry.
* **No cross-zone traffic** -- Transfers data through object storage, avoiding expensive cross-zone networking costs.

See [Universal Linking overview](/cloud/universal-linking/unilink-overview) for setup instructions.

<Tip>
  Universal Linking is the recommended approach for production migrations. It allows you to run both clusters in parallel and validate your migration before cutting over traffic.
</Tip>

### Step 5: Validate with test consumers

Before migrating production traffic, validate the migration:

1. Connect a test consumer to the StreamNative cluster and verify it can read mirrored data.
2. Connect a test producer to StreamNative and confirm messages are written and readable.
3. Verify that your Kafka Connect connectors work with the new endpoint.
4. Confirm Kafka Streams applications process data correctly.
5. Compare message counts and latencies between the source and StreamNative clusters.

### Step 6: Cut over production traffic

Once validation is complete, migrate production traffic:

1. Update producer configurations to point to the StreamNative bootstrap servers.
2. Wait for consumers to process any remaining messages from the source cluster.
3. Update consumer configurations to point to the StreamNative bootstrap servers.
4. Monitor consumer lag and throughput to confirm the cutover is successful.
5. Decommission the source cluster after a stabilization period.

<Note>
  Keep your source cluster running for a stabilization period (typically 24-72 hours) after cutover. This gives you a rollback path if unexpected issues arise.
</Note>

## Coming from a specific platform

<Accordion title="From Amazon MSK">
  If you are migrating from Amazon MSK, consider the following advantages of StreamNative Kafka Service:

  * **Cost savings** -- StreamNative's Ursa Engine uses tiered storage with object storage (S3), eliminating the need for expensive EBS volumes and reducing storage costs significantly.
  * **No AZ replication costs** -- Traditional MSK clusters replicate data across availability zones, incurring cross-AZ data transfer charges. StreamNative's architecture avoids these costs.
  * **Lakehouse-native** -- Built-in support for Apache Iceberg and lakehouse formats allows you to query streaming data directly with analytics engines, without building separate ETL pipelines.
  * **Simplified operations** -- No need to manage broker instances, patch Kafka versions, or tune JVM settings. StreamNative handles infrastructure management.
  * **Elastic scaling** -- Scale throughput up or down without the manual broker rebalancing required in MSK.
</Accordion>

<Accordion title="From Confluent Cloud or Platform">
  If you are migrating from Confluent Cloud or Confluent Platform, consider the following advantages of StreamNative Kafka Service:

  * **Open formats** -- StreamNative stores data in open formats (Apache Iceberg) rather than proprietary storage layers, giving you full control over your data.
  * **No vendor lock-in** -- Standard Kafka API compatibility means your applications are portable. No proprietary client libraries or APIs required.
  * **Cost efficiency** -- StreamNative's Ursa Engine provides significant cost savings through efficient tiered storage and compute-storage separation.
  * **Multi-protocol support** -- Access your data through both Kafka and Pulsar protocols, giving you flexibility in how you build applications.
  * **Transparent pricing** -- Predictable pricing without hidden costs for features like Schema Registry, connectors, or cluster linking.
</Accordion>

<Accordion title="From self-managed Apache Kafka">
  If you are migrating from a self-managed Apache Kafka deployment, consider the following advantages of StreamNative Kafka Service:

  * **Zero operational overhead** -- Eliminate the need to manage ZooKeeper or KRaft controllers, broker instances, and operating system patches.
  * **Auto-scaling** -- StreamNative automatically scales compute and storage based on your workload, removing the need for manual capacity planning.
  * **Managed infrastructure** -- Automated upgrades, security patches, and monitoring are handled for you, freeing your team to focus on application development.
  * **Built-in observability** -- Pre-configured metrics, dashboards, and alerting replace the need to build and maintain your own monitoring stack.
  * **Enterprise security** -- OAuth 2.0, RBAC, and encryption are built in and ready to use, without manual configuration of SASL, ACLs, and TLS certificates.
</Accordion>

## Existing resources

<CardGroup>
  <Card title="Migrating to StreamNative" icon="arrow-right" href="/cloud/build/kafka-clients/migrating-to-streamnative">
    Detailed guide on Kafka-specific differences and data retention policies to consider when migrating.
  </Card>

  <Card title="Universal Linking" icon="link" href="/cloud/universal-linking/unilink-overview">
    Set up data replication between your existing Kafka cluster and StreamNative for zero-downtime migration.
  </Card>

  <Card title="Kafka compatibility" icon="check" href="/cloud/build/kafka-clients/compatibility/kafka-compatibility">
    Full reference of supported Kafka APIs, protocol versions, and feature compatibility.
  </Card>

  <Card title="Get started with Kafka Service" icon="rocket" href="/kafka/kafka-getting-started">
    Create your first Kafka cluster and produce a message in under 5 minutes.
  </Card>
</CardGroup>


# Agent Context Reference
Source: https://docs.streamnative.io/agent-engine/agents-context

Understand the Orca Engine AgentContext APIs for accessing runtime data, state, and session controls.

The Orca runtime injects an `AgentContext` object into every invocation so your agent code can read message metadata, publish responses, and manage runtime state. This reference summarizes the key APIs that become available after importing `AgentContext` in your project:

```python theme={null}
from orca.functions.agent_context import AgentContext
```

## Access the current context

* `AgentContext.current()` returns the context bound to the active asynchronous task or thread. It raises `RuntimeError` if your code runs outside an initialized request scope.
* `AgentContext.enter(ctx)` explicitly associates a context with the current task. Use it when you spawn your own threads or background jobs so downstream code can call `AgentContext.current()` safely.
* The context implements synchronous and asynchronous context managers. Wrap long-running sections in `with AgentContext.current() as ctx:` (or `async with` inside asynchronous code) to ensure the previous context is restored automatically.

## Message metadata and acknowledgement

The runtime records properties of the active message before your agent executes. Use these helpers to inspect request details:

* `get_message_id()` exposes the origin of the current message.
* `get_message_key()`, `get_partition_key()`, `get_message_eventtime()`, and `get_message_properties()` surface custom metadata that producers may include.
* `get_current_message_topic_name()` returns the input topic name for the current invocation.
* `get_function_name()`, `get_function_tenant()`, `get_function_namespace()`, `get_function_id()`, `get_function_version()`, and `get_instance_id()` identify the deployed agent and running instance.

To emit outputs or acknowledge work:

* `publish(topic_name, message, serde_class_name="serde.IdentitySerDe", properties=None, compression_type=None, callback=None, message_conf=None)` sends asynchronous responses. The context reuses producers under the hood and accepts extra message configuration through the optional `message_conf` dictionary.
* `ack(msgid=None, topic=None)` acknowledges the active message. Provide a `msgid` or `topic` to override the defaults when you manage acknowledgements manually.

## Configuration, secrets, and logging

* `get_user_config_value(key)` and `get_user_config_map()` expose agent-level configuration supplied at deployment time.
* `get_secret(secret_key)` retrieves secrets from the configured provider. Expect a `None` return value when a key is missing and handle defaults accordingly.
* `get_logger()` returns the runtime logger so you can emit structured logs that align with platform observability settings.

## Metrics, state, and counters

`AgentContext` provides utilities for custom telemetry and lightweight state management:

* `record_metric(metric_name, metric_value)` records Prometheus summary metrics with the current instance labels.
* `incr_counter(key, amount)`, `get_counter(key)`, and `del_counter(key)` maintain numeric counters.
* `put_state(key, value)` and `get_state(key)` persist arbitrary state using the runtime-backed store.

## Topic configuration

* `get_input_topics()` lists all input topics connected to the agent.
* `get_output_topic()` and `get_output_serde_class_name()` show where default outputs go and which serialization class the runtime expects.

## Session management

* `get_session_mode()` returns the configured session strategy:
  * `SessionMode.SHARED` reuses one conversation for every invocation.
  * `SessionMode.SESSION_PER_MESSAGE` isolates each request.
  * `SessionMode.SESSION_PER_USER` partitions sessions by user identity when the incoming messages include user identifiers.
* For tool discovery and Model Context Protocol integrations, see [Managed agent tools](/agent-engine/agents-tools).

## Example usage

The snippet below shows how an agent can inspect message metadata, record metrics, and publish follow-up events while acknowledging work through the context:

```python theme={null}
from orca.functions.agent_context import AgentContext

async def handle_request(payload: dict) -> str:
    ctx = AgentContext.current()
    logger = ctx.get_logger()

    message_id = ctx.get_message_id()
    logger.info("processing message %s", message_id)

    ctx.record_metric("agent_requests_total", 1)
    routing_mode = ctx.get_user_config_value("routing-mode") or "direct"

    if routing_mode == "fanout":
        ctx.publish(ctx.get_output_topic(), {"status": "received", "payload": payload})

    ctx.ack()
    return f"handled message {message_id}"
```

## Best practices

* Always call `AgentContext.current()` inside request handlers so you stay within the active invocation scope.
* When launching background tasks, copy the active context with `AgentContext.enter(context_instance)` before executing work.
* Handle `None` returns from `get_user_config_value` and `get_secret` gracefully to keep agents resilient to missing settings.
* Use the metrics helpers sparingly to avoid high-cardinality metric names, and reset counters when they no longer apply.
* Publish follow-up events and acknowledge messages through the context so the runtime can reuse producers and maintain consistent delivery guarantees.

With these APIs in mind, you can build agents that combine messaging, durable state, and session-aware behavior across the Orca Engine platform.


# Manage Agents
Source: https://docs.streamnative.io/agent-engine/agents-manage

Use snctl to update, control, and inspect Orca Agents running on StreamNative Cloud.

Use the `snctl agents` command group to manage Orca Agents after they are deployed. The commands cover day-two operations such as listing agents, rolling out new artifacts, restarting runtimes, and deleting resources when they are no longer needed.

## Prerequisites

* Complete the [environment setup guide](/agent-engine/agents-setup) so `snctl` is authenticated against the correct organization, cluster, tenant, and namespace.
* Ensure your service account has permissions to read, update, and delete agents in the target namespace.
* Package the agent artifact (ZIP archive) you plan to deploy, as described in the framework-specific guides under **Develop Agents**.

## List agents in a namespace

Run `snctl agents list` with the tenant and namespace that scope your deployment. The command prints a table of agents.

```bash theme={null}
snctl agents list \
  --tenant public \
  --namespace operations
```

## Inspect an agent configuration

Fetch the stored configuration to confirm topics, framework, and runtime options. Use either the tenant/namespace pair or the fully qualified agent name.

```bash theme={null}
snctl agents get \
  --tenant public \
  --namespace operations \
  --name support-agent
```

Example output (trimmed for brevity):

```json theme={null}
{
  "name": "support-agent",
  "framework": "openai",
  "inputs": [
    "persistent://public/operations/support-requests"
  ],
  "output": "persistent://public/operations/support-responses",
  "parallelism": 1,
  "secrets": {
    "OPENAI_API_KEY": {
      "path": "llm-secrets",
      "key": "openai"
    }
  }
}
```

## Update an agent

Use `snctl agents update` to apply new code artifacts or adjust runtime settings. Reuse the same flags you passed when creating the agent.

### Deploy a new artifact

Provide the new ZIP archive with `--agent-file`. Include any other fields that changed (for example, updated topic bindings or secrets).

```bash theme={null}
snctl agents update \
  --tenant public \
  --namespace operations \
  --name support-agent \
  --agent-framework openai \
  --directory openai_multi_tool \
  --agent-file openai_multi_tool.zip \
  --inputs persistent://public/operations/support-requests \
  --output persistent://public/operations/support-responses \
  --secrets '{"OPENAI_API_KEY":{"path":"llm-secrets","key":"openai"}}'
```

### Adjust runtime settings

Pass the setting you want to change. The example below increases parallelism to scale out processing.

```bash theme={null}
snctl agents update \
  --tenant public \
  --namespace operations \
  --name support-agent \
  --parallelism 2
```

If the update command reports **Update contains no change**, verify that at least one flag carries a new value.

## Control agent lifecycle

Pause or resume the agent when you need to stop processing without deleting the deployment.

```bash theme={null}
# Gracefully stop all running instances
snctl agents stop \
  --tenant public \
  --namespace operations \
  --name support-agent

# Restart the agent after applying fixes
snctl agents restart \
  --tenant public \
  --namespace operations \
  --name support-agent

# Start the agent if it is currently stopped
snctl agents start \
  --tenant public \
  --namespace operations \
  --name support-agent
```

## Check runtime status

Retrieve the live status to confirm instance health, restart counts, and recent processing activity.

```bash theme={null}
snctl agents status \
  --tenant public \
  --namespace operations \
  --name support-agent
```

The output includes metrics for each instance. See [Monitor agents](/agent-engine/agents-monitoring) for guidance on interpreting the fields.

## Trigger a test request

If the agent deployed and running healthy, you can deliver an ad-hoc payload with `snctl agents trigger`. Supply the input topic name along with the payload that should be delivered to the agent.

```bash theme={null}
snctl agents trigger \
  --tenant public \
  --namespace operations \
  --name support-agent \
  --topic persistent://public/operations/support-requests \
  --payload 'how to reset the API token'
```

## Delete an agent

When the deployment is no longer required, remove it with `snctl agents delete`. The command supports both tenant/namespace pairs and fully qualified agent names.

```bash theme={null}
snctl agents delete \
  --tenant public \
  --namespace operations \
  --name support-agent
```

Run `snctl agents list` again to confirm the agent no longer appears. If the delete command reports that the agent does not exist, double-check the tenant, namespace, and name values.


# Configure Model Access
Source: https://docs.streamnative.io/agent-engine/agents-model

Manage large language model (LLM) provider credentials with StreamNative cloud Secrets and bind them to Orca Engine agents.

Agents rely on provider-specific API keys to call large language model (LLM) endpoints and other foundation models. StreamNative cloud Secrets let you manage those credentials centrally and expose them to running agents through the `--secrets` flag. This guide explains how to map secrets to environment variables for supported agent frameworks and how to retrieve those values at runtime.

## Before you start

1. [Create the required secrets](/cloud/security/secret) in your organization. Each secret maps a `path` (the secret name) to one or more `key` entries.
2. Ensure `snctl` is configured for the target tenant and namespace.
3. Decide which environment variable names your agent framework expects (see the sections below).

### Secret mapping syntax

When you submit or update an agent, pass provider credentials with the `--secrets` flag. The JSON payload follows this structure:

```bash theme={null}
snctl agents create \
  --name <agent-name> \
  --directory <package-root> \
  --agent-framework <framework> \
  --secrets '{
    "ENV_NAME": {"path": "secret-name", "key": "entry"}
  }'
```

* `ENV_NAME` becomes an environment variable inside the agent runtime. You can also access the same value through `AgentContext.current().get_secret("ENV_NAME")`.
* `path` references the StreamNative secret name, and `key` selects the field within that secret.
* Provide multiple entries in the JSON object to surface several credentials at once.

## Google agent development kit secrets

Google’s Agent Development Kit (ADK) uses the `google-genai` SDK, which respects the following environment variables:

* `GOOGLE_API_KEY`—API key for the Gemini Developer API.
* `GEMINI_API_KEY`—legacy alias; the runtime prefers `GOOGLE_API_KEY` when both are present.
* `GOOGLE_GENAI_USE_VERTEXAI`—set to `true` when you want to call Vertex AI endpoints instead of the public Gemini API.
* `GOOGLE_CLOUD_PROJECT`—required for Vertex AI requests.
* `GOOGLE_CLOUD_LOCATION`—the Vertex AI region (for example, `us-central1`).

Example deployment snippet:

```bash theme={null}
snctl agents create \
  --tenant <tenant> \
  --namespace <namespace> \
  --name gemini-agent \
  --directory multi_tool_agent \
  --agent-framework adk \
  --session-mode SHARED \
  --inputs <input-topic> \
  --output <output-topic> \
  --agent-file multi_tool_agent.zip \
  --secrets '{
    "GOOGLE_API_KEY": {"path": "gemini-secret", "key": "api_key"},
    "GOOGLE_GENAI_USE_VERTEXAI": {"path": "gemini-secret", "key": "use_vertex"},
    "GOOGLE_CLOUD_PROJECT": {"path": "gemini-secret", "key": "project"},
    "GOOGLE_CLOUD_LOCATION": {"path": "gemini-secret", "key": "location"}
  }'
```

Inside your agent module, read the values with the ADK SDK or directly from the environment:

```python theme={null}
import os
from google import genai

client = genai.Client()
vertex_enabled = os.environ.get("GOOGLE_GENAI_USE_VERTEXAI", "false").lower() in ("true", "1")
```

Because `client = genai.Client()` automatically checks these environment variables, no additional configuration is required once the secrets are mapped.

## Manage secrets for OpenAI agents

The OpenAI Agents SDK expects the standard OpenAI environment variables:

* `OPENAI_API_KEY`—required for all requests.
* `OPENAI_PROJECT`—optional project scoping, used when you organize keys by project.
* `OPENAI_ORG_ID`—optional organization identifier.
* `OPENAI_BASE_URL`—override for custom endpoints such as Azure OpenAI or on-prem gateways.

Submit an agent with the necessary secrets:

```bash theme={null}
snctl agents create \
  --tenant <tenant> \
  --namespace <namespace> \
  --name openai-agent \
  --directory openai_multi_tool \
  --agent-framework openai \
  --session-mode SHARED \
  --inputs <input-topic> \
  --output <output-topic> \
  --agent-file openai_multi_tool.zip \
  --secrets '{
    "OPENAI_API_KEY": {"path": "openai-secret", "key": "api_key"},
    "OPENAI_PROJECT": {"path": "openai-secret", "key": "project"},
    "OPENAI_BASE_URL": {"path": "openai-secret", "key": "base_url"}
  }'
```

In your agent code, rely on the SDK’s default environment handling or fetch values manually:

```python theme={null}
import os
from agents import Agent

api_key = os.environ.get("OPENAI_API_KEY")
base_url = os.environ.get("OPENAI_BASE_URL")

root_agent = Agent(
    name="Assistant",
    instructions="Use the configured OpenAI model to respond.",
)
```

If you prefer not to expose certain variables broadly, call `AgentContext.current().get_secret("OPENAI_API_KEY")` inside request handlers instead of reading from `os.environ`.

## Operational tips

* Store non-string data (for example, JSON configs) as base64-encoded strings inside the secret value.
* Rotate provider keys by updating the StreamNative secret; re-run `snctl agents update` with the same `--secrets` payload to refresh running agents.
* Share secrets across multiple agents by reusing the same `path` while pointing each `--secrets` entry to the appropriate `key`.
* Document required environment variables alongside your agent code so collaborators know which secret entries to maintain.

With this setup, StreamNative cloud Secrets act as the single source of truth for model credentials while Orca Engine handles secure distribution to ADK and OpenAI agents.


# Monitor and Troubleshoot Agents
Source: https://docs.streamnative.io/agent-engine/agents-monitoring

Learn how to inspect Orca agent health, review logs, and stream diagnostics to messaging topics.

StreamNative Cloud surfaces health and diagnostics for every Orca agent so you can debug issues quickly. Use the CLI for live status checks, review logs through the console, and stream log data to topics when you need deeper analysis.

## View agent status

<Note title="Before you start">
  Install and configure `snctl` as described in [Set up client tools](/agent-engine/agents-setup#set-up-client-tools) so the commands below can authenticate to your organization.
</Note>

Run `snctl agents status` with the agent's tenant, namespace, and name to fetch its current state:

```bash theme={null}
snctl agents status \
  --tenant public \
  --namespace support \
  --name concierge
```

Example output:

```json theme={null}
{
  "numInstances": 1,
  "numRunning": 1,
  "instances": [
    {
      "instanceId": 0,
      "status": {
        "running": true,
        "error": "",
        "numRestarts": 0,
        "numReceived": 128,
        "numSuccessfullyProcessed": 128,
        "numUserExceptions": 0,
        "latestUserExceptions": [],
        "numSystemExceptions": 0,
        "latestSystemExceptions": [],
        "averageLatency": 135,
        "lastInvocationTime": 1709855312000,
        "workerId": "orcaworker-1"
      }
    }
  ]
}
```

Key fields to watch:

* `running`: Confirms the runtime loop is healthy. A `false` value indicates the instance stopped and requires attention.
* `numRestarts`: Counts automatic restarts. Frequent restarts suggest initialization errors or crashes.
* `numUserExceptions` and `latestUserExceptions`: Surface issues thrown by your agent code.
* `numSystemExceptions` and `latestSystemExceptions`: Capture platform-level errors such as connectivity or serialization problems.
* `averageLatency` and `lastInvocationTime`: Help validate that requests are being processed on schedule.

Combine the status output with `snctl agents restart` or `snctl agents trigger` (when supported) to validate fixes before returning agents to production traffic.

## Inspect logs and exceptions in StreamNative Cloud Console

The StreamNative Cloud Console provides a real-time view of agent activity.

1. [Log in to StreamNative Cloud Console](https://console.streamnative.cloud/?defaultMethod=signup).
2. In the left navigation, open **Agents** panel.
3. Select the agent you deployed to open its detail page. Review the status metrics for quick health insight.
4. Switch to the **Logs** tab for streaming log output.

Console views combine status, logs, and configuration so you can verify rollouts, confirm tool configs, and observe runtime behavior without leaving the browser.

## View the agent logs using `snctl`

This section describes how to view agent logs using `snctl`.

<Note title="Note">
  This example assumes you have [installed snctl](/tools/cli/snctl/snctl-overview#install-snctl) and [initialized snctl configurations](/tools/cli/snctl/snctl-overview#initialize-snctl-configurationn).
</Note>

You can run the `snctl logs` command to view logs for a specific agent. This table outlines the configuration options that are used for viewing agent logs. For details about all supported fields, you can use the `snctl logs -h` command to list more information.

| Option                   | Descriptions                                                                                                                |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------- |
| `-c` or `--cluster`      | The name of your Pulsar cluster where the agent is created.                                                                 |
| `-p` or `--component`    | The type of component to monitor. Available options are `function`, `sink`, `source`, 'kafka-connect' and 'agent-function'. |
| `-f` or `--follow`       | Continuously list the agent log history.                                                                                    |
| `-h` or `--help`         | Show usage information about the `snctl logs` command.                                                                      |
| `-i` or `--instance`     | The name of your Pulsar instance where the agent is created.                                                                |
| `--name`                 | The name of your agent.                                                                                                     |
| `-o` or `--organization` | The name of your organization where the agent is created.                                                                   |
| `--previous`             | Print the logs that are generated before the configured timestamp.                                                          |
| `--pulsar-tenant`        | The name of your Pulsar tenant where the agent is created.                                                                  |
| `--pulsar-namespace`     | The name of your Pulsar namespace where the agent is created.                                                               |
| `--since`                | List logs more recent than the specific time. Available units are `second`, `minute`, and `hour`, such as `24h`.            |
| `-s` or `--size`         | Specify how many lines of recent logs to display.                                                                           |
| `--timestamp`            | Include timestamps on each line in the log output.                                                                          |

The following command example shows how to view up to 60 lines of the `agent-func` agent's logs within the last 5 hours.

```bash theme={null}
snctl logs --since 5h --organization sndev --instance aws --cluster aws --name agent-func --pulsar-tenant public --pulsar-namespace default -p agent-function -f -s 60
```

You should see the agent logs output.


# StreamNative Orca Agent Platform (Private Preview)
Source: https://docs.streamnative.io/agent-engine/agents-overview

Deploy and manage AI agents processing real-time data at scale with the StreamNative Orca Agent Platform private preview.

# StreamNative Orca Agent Platform (Private Preview)

Deploy and manage AI agents that process real-time data at scale using industry-standard frameworks and models.

## Prerequisites

* StreamNative Cloud account with Orca Private Preview access enabled
* Basic understanding of event streaming concepts
* Familiarity with Python and your chosen agent framework (Google ADK, OpenAI Agents SDK)

## What is StreamNative Orca Agent Platform

StreamNative Orca Agent enables developers to deploy and operate advanced AI agents handling realtime events securely and at scale. It provides streaming infrastructure purpose-built for event-driven agent workloads, robust tools to enhance agent capabilities, and enterprise-grade controls for real-world production deployments. Orca Agent services can be used individually or in combination and integrate seamlessly with popular agent frameworks, including Google Agent Development Kit, OpenAI Agents SDK, and Langchain (coming soon), as well as any foundation model, giving you maximum flexibility. By removing the undifferentiated heavy lifting of building and managing specialized streaming infrastructure, Orca Agent accelerates your path from development to production for tackling realtime events.

Orca Agent Engine brings autonomous agent workloads to StreamNative Cloud by combining AI-first capabilities with the platform's event-driven infrastructure. It lets you deploy reasoning agents without abandoning topic-centric pipelines, so stream processing, tool calls, and LLM orchestration share the same operational surface.

## Orca Agent Platform Capabilities

The Orca Agent Platform delivers a comprehensive set of capabilities designed to simplify the deployment, operation, and scaling of real-time AI agents. From seamless integration with streaming data systems to dynamic workflow orchestration, advanced context management, and enterprise-grade monitoring, Orca Agent equips teams with the tools needed to build, run, and govern production-ready agents with confidence. These capabilities work together to reduce complexity, accelerate time-to-market, and ensure reliability across diverse environments.

**Platform highlights**

* Runs inside the managed Orca agent runtime, inheriting StreamNative Cloud scaling, tenancy, and topic-based connectivity.
* Supports both [Google Agent Development Kit (ADK)](https://google.github.io/adk-docs/) agents and [OpenAI Agents](https://openai.github.io/openai-agents-python/) runners today, with a shared agent runtime interface for future frameworks.
* Normalizes tool access through a managed context layer, allowing Model Context Protocol (MCP) toolchains to be injected at runtime alongside user-defined tools.

### Real-Time Data Integration for Agents

Orca Agent Platform natively integrates with high-throughput messaging systems such as Apache Pulsar and Kafka, enabling agents to consume, process, and act on live event streams with ultra-low latency. This seamless connection allows agents to respond to changing conditions instantly, enrich decision-making with the latest information, and power real-time applications at any scale. Because data ingestion, routing, and back-pressure management are built into the platform, teams avoid the complexity of building custom pipelines or maintaining additional middleware.

Native Pulsar and Kafka adapters integrate with Schema Registry across StreamNative Cloud clusters. Schema handling is unified so agent functions can ingest or emit different schema payloads without custom serialization code.

### Workflow Orchestration and Agent Coordination

The platform provides flexible orchestration support that allows you to define, schedule, and coordinate agents and workflows around your business logic. You can chain agents together, run them in parallel, or dynamically adjust execution paths based on rules or events, enabling complex, event-driven architectures. This orchestration capability reduces operational overhead and allows organizations to adapt quickly to new requirements or scale up specific functions on demand.

### Runtime Architecture

* **Runtime interface** provides async-friendly initialization and processing hooks, so agent loops can await external LLM calls without blocking the platform runtime.
* **Context layer** wraps the execution environment, exposing messaging clients, tool registries, and session mode so agent code can remain framework-agnostic.
* **Base runtime** loads agent packages, wires Google ADK or OpenAI runners, and reuses the built-in artifact service for intermediate outputs.

### Session and State Management

* The runtime session service persists agent state in the managed store and streams events to per-session topics.
* Multiple session modes (`shared`, `session_per_message`, `session_per_user`) let you decide whether conversations share context or stay isolated; select the mode that fits your workload during agent configuration.
* Producer instances are cached so the platform avoids recreating Pulsar or Kafka producers on every turn, keeping latency low for conversational workloads.

### Enhanced Agent Functionality with MCP and Context Management

With built-in support for MCP tool calling, agents can securely invoke external services, APIs, or data sources to extend their functionality far beyond the core model. Coupled with first-class memory and context management, agents gain the ability to maintain state, retain knowledge across sessions, and personalize responses or actions based on historical interactions. This combination transforms agents from stateless responders into context-aware, adaptive actors capable of executing sophisticated workflows and reasoning over time.

The [StreamNative MCP Server](/agent-engine/mcp/mcp-overview) is available as a [local binary](/agent-engine/mcp/local-mcp-server) for individual developers or as a [managed remote service](/agent-engine/sn-remote-mcp/remote-mcp-overview) with centralized [governance and access controls](/agent-engine/sn-remote-mcp/remote-mcp-governance). Both options expose Pulsar and Kafka clusters as MCP tools that agents and IDE copilots can call directly.

* **Google ADK agents** can load Python modules, merge MCP-provided tools with user tools, and run through the ADK runner with optional persistent sessions.
* **OpenAI agents** execute configurations built with the `openai-agents` package, maintain topic-based session caches, and reload MCP tools each turn.
* MCP tool merging prevents duplicate registrations: both adapters filter out tools that already come from the managed tool registry before assembling the final tool list for the framework runtime.
* Support for MCP server-sent events (SSE) and streaming responses keeps agents in sync with external tools through real-time updates.

### Enterprise-Grade Monitoring and Administration

Orca Agent Platform offers a unified observability and management layer, providing enterprise-grade monitoring, logging, and auditing across all deployed agents. Administrators can track performance metrics, resource utilization, and agent activities in real time, ensuring compliance and operational transparency. Fine-grained access control, secure isolation, and automated policy enforcement protect sensitive data and maintain consistent standards across multi-cloud or hybrid environments.

* **Agent admin API** exposes REST endpoints aligned with agent management workflows, including listing, describing, creating, and deleting agents across namespaces and tenants.
* **CLI tooling**: `snctl` ships agent-focused commands that enable day-to-day lifecycle management (deploy, update, status, logs) from the terminal.
* **Cloud Console (UI)** surfaces agent details in the Agents panel so teams can monitor deployments without leaving the browser.

## Orca Agent Platform Benefits

### Simplicity - Faster Development

* Python support as a first-class citizen: Build, deploy, and extend agents using Python with full native support, enabling developers to work with familiar tooling, libraries, and workflows.
* No extra infrastructure to spin up or manage: Orca Agent runs on a fully managed platform, eliminating the need to provision or maintain separate servers, containers, or clusters.
* No additional framework or API to introduce: Seamlessly integrate with your existing stack without learning new APIs or frameworks, reducing development overhead and accelerating time-to-market.
* Minimal code changes to existing agents: Existing agents can be onboarded with little to no refactoring, allowing teams to migrate quickly without disrupting production workloads.

### Flexibility - Build Your Way

* Agent framework at users' choice: Support for Google ADK, OpenAI Agents SDK, and other popular agent frameworks gives teams freedom to build with the tools they know best.
* Messaging protocol at users' choice: Native integration with Pulsar or Kafka lets you connect to your preferred streaming backbone without additional middleware.
* Foundation model at users' choice: Work with Anthropic Claude, Google Gemini, OpenAI GPT, and other foundation models to adapt quickly to new capabilities and use cases.
* Hyperscaler at users' choice: Deploy across AWS, GCP, or Azure to take advantage of your existing cloud footprint and optimize for cost, performance, or compliance.

### Security - Enterprise-Grade Protection

* Fine-grained data access control inherited from StreamNative Enterprise: Enforce enterprise-grade identity, authorization, and data access policies without additional configuration.
* Complete isolation between agents to minimize interference risk: Each agent runs in an isolated environment, preventing data leakage, cross-contamination, and performance impact from other agents.
* Ability to audit and trace agent activities whenever needed: Built-in auditing and observability allow you to track agent behaviors, ensure compliance, and quickly troubleshoot issues.

### Cloud-Native - Operate at Scale

* Native auto-scaling capabilities: Automatically scale agents up or down in response to demand, ensuring high performance and cost efficiency without manual intervention.
* Strong fault tolerance with minimal maintenance overhead: Built-in redundancy and recovery mechanisms ensure continuous availability and reduced operational burden.
* Collaborate with other workflows/agents within seconds: Natively orchestrate or chain multiple agents and workflows together in real time, enabling complex, event-driven applications.

## What's next?

* Review stream processing fundamentals in [Stream processing overview](/cloud/process/pulsar-functions/functions-overview) before deploying agents.
* Learn how to [set up your environment](/agent-engine/agents-setup).


# Set Up Your Environment
Source: https://docs.streamnative.io/agent-engine/agents-setup

Prepare StreamNative Cloud accounts and permissions to deploy Orca Agents.

This guide walks you through preparing an environment that can deploy Orca Agents on StreamNative Cloud. You will create a dedicated service account, assign the minimum required permissions, and configure tooling for both the CLI and the StreamNative Cloud Console.

## Prerequisites

* [Install](/tools/cli/snctl/snctl-overview#install-snctl) and [configure](/tools/cli/snctl/snctl-overview#configure-snctl) the `snctl` CLI tool.
* [Log in to StreamNative Cloud Console](https://console.streamnative.cloud/?defaultMethod=signup).
* Create or select a [cluster](/cloud/clusters/manage-clusters/cluster#create-a-cluster), [tenant](/cloud/manage-data-streams/tenant#create-a-tenant), and [namespace](/cloud/manage-data-streams/namespace#create-a-namespace) where agents will run.
* Ensure your user has administrator access for the target cluster and organization.

## Create a service account for agents

1. In StreamNative Cloud Console, open the left navigation and choose **Service Accounts**.
2. Click **Create Service Account**.
3. Provide a name and optional description, then click **Confirm**.

<Note title="Tip">
  Leave **Super Admin** unchecked so the account only receives the scoped permissions you grant later.
</Note>

## Authorize the service account

To make the service account work, you need to make the service account granted with proper permissions (`functions`, `packages`, `produce`, and `consume`).

To grant the service account permissions on the namespace level, follow these steps:

<Tabs>
  <Tab title="Cloud Console">
    1. Navigate to the **Namespace Dashboard** page by [switching to the namespace workspace](/cloud/get-started/cloud-console#switch-a-namespace).

    2. On the **Namespace Dashboard** page, click **Configuration** on the left navigation pane.

    3. On the **Namespace configuration** page, click **ADD ROLE**, and select the service account that you want to authorize.

    4. On the drop-down menu below the service account, select the proper permissions to assign to the service account. There are six permissions in total:

    * `consume`: allow the service account to consume messages.
    * `produce`: allow the service account to publish messages.
    * `functions`: allow the service account to submit and manage functions.
    * `sinks`: allow the service account to create and manage sink connectors.
    * `sources`: allow the service account to create and manage source connectors.
    * `packages`: allow the service account to upload and manage pulsar packages. If you want to submit a customized function/connector, then you will need to upload the function/connector’s JAR/NAR/Python file first, which requires the `packages` permission.

      <img alt="Authorize Service Account" />
  </Tab>
</Tabs>

## Grant access to the service account

To grant the underlying infrastructure with access to the newly created service account's OAuth2 key file, you need to create a service account binding via UI.

Go to the `Service Accounts` tab and choose the service account you want to use for running the connector. Clicking on the right button and there will be a `Edit service account bindings` option.

<img alt="Binding Service Account step-1" />

Click the `Edit service account bindings`, choose the desired pool member and confirm.

<img alt="Binding Service Account step-2" />

You can also enable the `Enable IAM Role Creation` option to create a separate IAM role for the service account.

Now your connector is ready to use the service account in StreamNative environments.

## (Optional) Create a separate IAM role for the service account

StreamNative's I/O components (Pulsar Functions, Pulsar Connectors, and Kafka Connectors) run as cloud-native workloads on AWS, GCP, and Azure infrastructures.

Use the cloud providers' native IAM (Identity and Access Management) services to control access to infrastructure resources such as S3, GCS, and Azure Blob Storage. This removes the need for password-based authentication for the service accounts that run connectors in StreamNative Cloud.

By default, StreamNative uses a single service account per `PulsarCluster` for all I/O components (Pulsar Functions, Pulsar Connectors, and Kafka Connectors) to access underlying infrastructure resources. This means all I/O Components in the same cluster share one service account and the same permissions.

<Warning>
  Using a single service account for all I/O components means all functions and connectors share the same permissions. If one component is compromised, it could access resources intended for other components.

  So it's better to leave the default service account with no permissions and use it for running IO components that do not require access to external resources.

  And create separate service accounts with the minimum required permissions for each IO component that need to access external resources.
</Warning>

To enhance security and improve isolation, you can create a separate IAM role for each service account used to run connectors. This let you grant only the permissions each service account needs.

Create a separate IAM role for your service account:

<Tabs>
  <Tab title="Snctl">
    <Note title="Note">
      This feature is available in [snctl](/tools/cli/snctl/snctl-overview) v1.3.0 or later.
    </Note>

    1. Get the `PoolMember` name and namespace from the `PulsarCluster`

       ```shell theme={null}
       snctl get pulsarcluster <cluster-id> -o yaml
       ```

       In the output, find the `poolMemberRef` block, which looks like:

       ```yaml theme={null}
       poolMemberRef:
         name: <pool-member-name>
         namespace: <namespace>
       ```

       <Note title="Note">
         Multiple clusters may be located in the same `PoolMember`. You do not need to create separate IAM roles for each cluster within the same `PoolMember`.
       </Note>

    2. Create a new `ServiceAccountBinding` that binds the service account to the `PoolMember`

       ```shell theme={null}
       snctl create serviceaccountbinding <binding-name> \
         --pool-member <pool-member-namespace>/<pool-member-name> \
         --enable-iam-account-creation \
         --service-account-name <service-account-name>
       ```

       **AWS only:** You can specify one or more `AWS Assume Role ARNs` that can be assumed by the IAM role created for the service account (repeat the flag for each ARN):

       ```shell theme={null}
       snctl create serviceaccountbinding <binding-name> \
         --pool-member <pool-member-namespace>/<pool-member-name> \
         --enable-iam-account-creation \
         --service-account-name <service-account-name> \
         --aws-assume-role-arns <arn1> \
         --aws-assume-role-arns <arn2>
       ```

       <Note title="Note">
         The IAM role created by StreamNative will include permissions to call `sts:AssumeRole` on `arn1` and `arn2`. You must still add a trust policy on `arn1` and `arn2` to allow the newly created role to assume them, an example likes below:

         ```json theme={null}
         {
             "Version": "2012-10-17",
             "Statement": [
                 {
                     "Effect": "Allow",
                     "Principal": {
                         "AWS": "arn:aws:iam::[aws-account-id]:role/StreamNative/sncloud-role/authorization.streamnative.io/iamaccounts/IamAccount-[org-id]-sab-[binding-name]"
                     },
                     "Action": "sts:AssumeRole"
                 }
             ]
         }
         ```
       </Note>

    3. (Optional) Update an existing `ServiceAccountBinding` to create the IAM role

       ```shell theme={null}
       snctl edit serviceaccountbinding <binding-name>
       ```

    4. Verify the IAM role was created successfully

       ```shell theme={null}
       snctl get serviceaccountbinding <binding-name> -o yaml
       ```

       Expected output (example):

       ```yaml theme={null}
       apiVersion: cloud.streamnative.io/v1alpha1
       kind: ServiceAccountBinding
       metadata:
         creationTimestamp: "2025-06-25T08:45:35Z"
         finalizers:
         - serviceaccountbinding.finalizers.cloud.streamnative.io
         generation: 1
         name: test-admin
         namespace: o-lftqu
         ownerReferences:
         - apiVersion: cloud.streamnative.io/v1alpha1
           kind: ServiceAccount
           name: admin
           uid: 4ef639aa-6278-4863-9a23-f1da50cea448
         resourceVersion: "54707713"
         uid: 918d40ad-551d-416b-a2ae-d41548d6608e
       spec:
         enableIamRoleCreation: true
         poolMemberRef:
           name: azure-eastus-zephyr
           namespace: streamnative
         serviceAccountName: admin
       status:
         conditions:
         - lastTransitionTime: "2025-06-25T08:45:35Z"
           status: "True"
           type: IAMAccountReady
         - lastTransitionTime: "2025-06-25T08:45:35Z"
           status: "True"
           type: ServiceAccountReady
         - lastTransitionTime: "2025-07-16T13:56:17Z"
           status: "True"
           type: ResourceExists
         - lastTransitionTime: "2025-07-16T13:56:17Z"
           status: "True"
           type: PoolMemberReady
         - lastTransitionTime: "2025-07-16T13:56:17Z"
           reason: AllConditionStatusTrue
           status: "True"
           type: Ready
       ```

       In the output, the `status.conditions` array should include a condition with `type: IAMAccountReady` and `status: "True"`, indicating the IAM role was created successfully.<br /><br />

    5. Use the service account when creating I/O components

       <br /><br />You can now select this service account in Console (or use its API key with the CLI) when creating I/O components (Pulsar Functions, Pulsar Connectors, and Kafka Connectors). The components will inherit the permissions granted to the IAM role created in the previous step.
  </Tab>

  <Tab title="Console">
    You can also create a separate IAM role for your service account via StreamNative Cloud Console. Just enable the `Enable IAM Role Creation` option when creating or editing a service account binding.

    <img alt="Binding Service Account step-2" />

    **AWS only:** You can specify one or more `AWS Assume Role ARNs` that can be assumed by the IAM role created for the service account (use one line for each ARN)

    <img alt="Binding Service Account step-3" />

    <Note title="Note">
      The IAM role created by StreamNative will include permissions to call `sts:AssumeRole` on `arn1` and `arn2`. You must still add a trust policy on `arn1` and `arn2` to allow the newly created role to assume them, an example likes below:

      ```json theme={null}
      {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Principal": {
                      "AWS": "arn:aws:iam::[aws-account-id]:role/StreamNative/sncloud-role/authorization.streamnative.io/iamaccounts/IamAccount-[org-id]-sab-[binding-name]"
                  },
                  "Action": "sts:AssumeRole"
              }
          ]
      }
      ```
    </Note>
  </Tab>
</Tabs>

<Note title="Note">
  In Azure, a Managed Identity `sab-[binding-name]-[org-id]` is created;

  In AWS, an IAM role `role/StreamNative/sncloud-role/authorization.streamnative.io/iamaccounts/IamAccount-[org-id]-sab-[binding-name]` is created;

  In GCP, a service account with display name: `IamAccount/[org-id]/sab-[binding-name]` is created.
</Note>

## Set up client tools

### Use `snctl` to manage agents

StreamNative CLI (`snctl`) provides agent-aware commands that are surfaced under `snctl agents`.

1. Set your organization: `snctl config set --organization <org-name>`.
2. Select a cluster context: `snctl context use` and choose the target cluster.
3. Work with agents by using the following commands (pass `--tenant` and `--namespace` as needed):
   * `snctl agents list` – list agents in the current namespace.
   * `snctl agents create` – deploy a new agent from a declarative spec and package.
   * `snctl agents update` – roll out spec or code changes.
   * `snctl agents status` – view runtime status.
   * `snctl agents start|stop|restart` – control the running agent.
   * `snctl agents delete` – remove an agent.
   * `snctl agents get` – show the stored agent spec.
   * `snctl agents trigger` – deliver an ad-hoc test event when supported by the agent implementation.

### Use StreamNative Cloud Console

The StreamNative Cloud Console offers a guided experience for connecting client applications and reviewing agent activity.

Navigate to **Agents** from the left panel to review existing deployments, logs, and status.

## What's next?

* Review the [Orca Agents overview](/agent-engine/agents-overview) to understand runtime architecture and capabilities.
* Package your first agent and deploy it with `snctl` or the Cloud Console.


# Managed Agent Tools
Source: https://docs.streamnative.io/agent-engine/agents-tools

Learn how Orca Engine discovers agent tools, integrates Model Context Protocol servers, and keeps tool sets current at runtime.

Orca Engine aggregates every tool your agent can invoke and exposes them through `AgentContext`. Use the context API to request the latest tool list on each invocation:

```python theme={null}
from orca.functions.agent_context import AgentContext

available_tools = AgentContext.current().get_tools()
```

## Tool discovery overview

`AgentContext.current().get_tools()` returns an ordered list of tool definitions ready for use by your agent framework. The runtime combines two sources:

* **Static tools** that you register with your agent package or configure through deployment metadata.
* **Managed Model Context Protocol (MCP) tools** supplied by the StreamNative cloud console and connected MCP servers.

Tool entries with duplicate names are skipped automatically so the first matching definition wins. When your organization scopes tools through inclusion rules, only allowed entries appear in the final list.

## Use tools in agent code

Invoke `get_tools()` inside the handler that constructs your agent rather than caching results at import time. This ensures each invocation sees the current configuration and any MCP updates.

```python theme={null}
from agents import Agent, function_tool
from orca.functions.agent_context import AgentContext

@function_tool
async def fetch_weather(city: str) -> str:
    return "Weather data goes here"

managed_tools = AgentContext.current().get_tools()

root_agent = Agent(
    name="operations_assistant",
    instructions="Use managed tools whenever additional context is helpful.",
    tools=[fetch_weather, *managed_tools],
)
```

Agents built with other frameworks follow the same pattern: request the active context, append your static helpers, and pass the combined list to the runtime.

## Dynamic updates with the model context protocol

Orca Engine maintains long-lived Server-Sent Events (SSE) streams to each registered MCP server. These notifications power two capabilities:

* **Dynamic tool discovery**: When an MCP server adds, removes, or updates a tool, the runtime refreshes its catalog and the next call to `get_tools()` reflects the change—no redeployment required.
* **Real-time MCP messages**: SSE messages surface remote events so your agent can react to live signals, such as tool hot reloads or targeted prompts from connected systems.

When the runtime receives a "tools changed" notification and detects a difference from the cached definition set, it reloads the active agent instance before processing additional requests. This refresh ensures subsequent calls to `get_tools()` return the latest tools and that long-lived sessions see the updated capabilities.

Because MCP updates arrive asynchronously, call `get_tools()` within the request scope or immediately before you instantiate the agent. Avoid storing the returned list in module-level variables.

## Best practices

* Request `AgentContext.current().get_tools()` inside the invocation flow to pick up runtime changes.
* Use descriptive names and function comments for static tools so large-language-model-based agents can choose them accurately.
* Coordinate session strategy with your tooling approach—combine this page with [Agent Context](/agent-engine/agents-context) for state, metrics, and session controls.
* When testing locally, inject a mock context that provides representative tool lists before importing modules that expect runtime tools.

With these patterns, your Orca Engine agents can blend built-in helpers with dynamically managed MCP integrations while staying up to date automatically.


# Develop ADK Agents
Source: https://docs.streamnative.io/agent-engine/develop-agents/agents-develop-adk

Build and package Google ADK agents for Orca Engine and deploy them with snctl.

This guide shows how to adapt Google Agent Development Kit (ADK) projects so they can run as Orca Agents on StreamNative Cloud. You will learn how to prepare an ADK project, expose the required `root_agent` entry point, and deploy the artifact with `snctl`.

## Prerequisites

* Python 3.10 or later and a virtual environment for packaging your agent code.
* The [`google-adk-python`](https://github.com/google/adk-python) library and any model SDKs your agent requires.
* The `orca-agent-engine` python SDK for orca engine.
* StreamNative CLI (`snctl`) installed and configured with the target organization, tenant, and namespace.
* Access to StreamNative Cloud topics that will deliver agent input and capture responses.

## Prepare an ADK project

1. Start from an ADK reference implementation. You can reuse examples from the Google ADK samples repository or the Orca engine examples provided below.

2. Copy the project into your own source repository and update the package metadata (for example, rename the module and adjust `pyproject.toml`).

3. Install dependencies into a fresh virtual environment:

   ```bash theme={null}
   python -m venv .venv
   source .venv/bin/activate
   pip install --upgrade pip setuptools
   pip install google-adk
   pip install orca-agent-engine
   ```

4. Add any helper libraries your agent needs (requests, domain SDKs, etc.) to `requirements.txt` or your packaging metadata so the runtime installs them alongside your agent code.

## Export the `root_agent`

The Orca runtime loads ADK projects by importing a module-level variable named `root_agent`. Define that agent with Google ADK's `Agent` class and list the tools or workflows it should expose.

```python theme={null}
from google.adk.agents import Agent
from google.adk.models import Gemini

# Optional: pull managed tools from the Orca execution context when you need MCP integration.
try:
    from orca.functions.agent_context import AgentContext
    managed_tools = AgentContext.current().get_tools()
except ImportError:  # Local unit tests can supply their own tools
    managed_tools = []

root_agent = Agent(
    name="multi_tool_agent",
    model=Gemini(model="gemini-2.5-pro"),
    description="Answer questions with live tool assistance.",
    instruction="Use available tools to fetch facts before responding.",
    tools=managed_tools,
)
```

Tips for adapting existing ADK samples:

* Keep fast feedback loops by writing thin wrapper functions around external systems (weather APIs, knowledge bases, etc.).
* Export the agent symbol through `__all__` in your package's `__init__.py` so the runtime can discover it.

## Package the project

Orca Engine accepts ADK artifacts packaged as ZIP archives.

### Package as a ZIP archive (recommended)

ZIP archives are the quickest way to bundle ADK agents. The structure mirrors the Python packaging pattern linked above.

```
multi_tool_agent/
  __init__.py
  agent.py
requirements.txt
```

```bash theme={null}
python -m pip freeze --exclude-editable > requirements.txt
zip -r multi_tool_agent.zip multi_tool_agent requirements.txt
```

* Run the `pip freeze` command from the same virtual environment you used while developing the agent so dependency versions stay in sync.
* The Orca runtime installs listed dependencies automatically when you deploy the agent.
* Reference the archive with `--agent-file multi_tool_agent.zip` during `snctl agents create` or `snctl agents update`.

## Deploy with `snctl`

Use the agent-aware subcommands to publish the package and configuration to StreamNative Cloud.

### Package as a ZIP archive

```bash theme={null}
snctl agents create \
  --tenant <tenant> \
  --namespace <namespace> \
  --name multi-tool-agent \
  --directory multi_tool_agent \
  --agent-framework adk \
  --session-mode SHARED \
  --inputs <input-topic> \
  --output <output-topic> \
  --agent-file multi_tool_agent.zip
```

* `--directory` matches the importable package path inside your ZIP archive.
* `--agent-file` accepts local files or URLs; point it to the ZIP archive you produced.
* Repeat the command with new artifacts or specs and `snctl agents update` to roll out changes. Use `snctl agents status` to monitor instances and `snctl agents trigger` to inject test messages.

## Next steps

* Configure [service accounts and permissions](/agent-engine/agents-setup) before deploying to production namespaces.
* Review managed tool configuration in the StreamNative Cloud Console so your ADK agent can discover MCP servers at runtime.
* Add automated tests that import the package and call `root_agent` to validate tools and prompts before publishing updates.


# Develop OpenAI Agents
Source: https://docs.streamnative.io/agent-engine/develop-agents/agents-develop-openai

Build and package OpenAI Agents SDK projects for Orca Engine and deploy them with snctl.

This guide shows how to adapt projects built with the [OpenAI Agents Python SDK](https://openai.github.io/openai-agents-python/) so they run as Orca Agents on the StreamNative cloud platform. It walks through preparing a project, exposing the required `root_agent` entry point, packaging the code for upload, and deploying it with `snctl`.

## Prerequisites

* Python 3.10 or later with a virtual environment for isolating dependencies.
* The `openai-agents` library, the `openai` SDK components your agent requires, and any extra model integrations.
* The `orca-agent-engine` Python SDK for interacting with the Orca runtime.
* StreamNative CLI (`snctl`) installed and configured for the organization, tenant, and namespace where you plan to deploy.
* Access to StreamNative cloud topics that deliver agent input and capture responses.
* An OpenAI API key stored as a secret so the runtime injects it into the agent (for example, `OPENAI_API_KEY`).

## Prepare an OpenAI Agents project

1. Start from an OpenAI Agents SDK example, such as the multi-tool or Model Context Protocol (MCP) samples distributed with the Orca Engine examples, or create a new project from scratch.

2. Copy the project into your own source repository and update the package metadata (for example, adjust module names and version numbers in `pyproject.toml`).

3. Install dependencies inside a fresh virtual environment:

   ```bash theme={null}
   python -m venv .venv
   source .venv/bin/activate
   pip install --upgrade pip setuptools
   pip install openai-agents
   pip install openai
   pip install orca-agent-engine
   ```

4. Add any helper libraries (HTTP clients, supporting SDK packages, etc.) to `requirements.txt` or your packaging metadata so they stay bundled with the agent artifact.

## Export the `root_agent`

The Orca runtime imports OpenAI projects by loading a module-level variable named `root_agent`. Define the agent with the OpenAI SDK and keep runtime secrets such as `OPENAI_API_KEY` outside the code.

```python theme={null}
from agents import Agent

root_agent = Agent(
    name="Assistant",
    instructions="You only respond in haikus.",
)
```

* Ensure your package exposes the module through `__all__` in `__init__.py` so the runtime can discover the agent.
* Validate the agent locally by importing the package and sending a sample request through the OpenAI SDK.

## Add tools and orchestrations

Use the `@function_tool` decorator to register functions as callable tools. This allows the agent to execute Python code when it needs to fetch data or perform actions.

```python theme={null}
from agents import Agent, function_tool

@function_tool
async def fetch_weather(city: str) -> str:
    # Replace with a real API call
    return "The weather is sunny."

root_agent = Agent(
    name="Assistant",
    instructions="Use tools when you need more context before answering.",
    tools=[fetch_weather],
)
```

* Leverage type hints and descriptive function comments to provide context that improves tool selection.
* When you need access to the execution context (for example, to stream tokens), annotate parameters with `RunContextWrapper` from the OpenAI SDK.

## Discover managed tools at runtime

When your Orca workspace provides managed tools through the console, fetch them with the Orca runtime context and supply them to the agent.

```python theme={null}
from agents import Agent
from orca.functions.agent_context import AgentContext

managed_tools = AgentContext.current().get_tools()

root_agent = Agent(
    name="operations_assistant",
    instructions="Use the managed tools provided by Orca Engine before you respond.",
    tools=managed_tools,
)
```

Wrap the context lookup in a `try`/`except` block if you run unit tests outside of Orca Engine.

## Package the project

Orca Engine accepts OpenAI agent artifacts packaged as ZIP archives.

### Package as a ZIP archive (recommended)

ZIP archives offer the fastest iteration loop. Organize the files so the top-level folder matches your importable module.

```
openai_multi_tool/
  __init__.py
  agent.py
requirements.txt
```

```bash theme={null}
python -m pip freeze --exclude-editable > requirements.txt
zip -r openai_multi_tool.zip openai_multi_tool requirements.txt
```

* Run `pip freeze` from the virtual environment you used during development so dependency versions stay consistent.
* The runtime installs listed dependencies automatically during deployment.
* Reference the ZIP archive with `--agent-file openai_multi_tool.zip` when running `snctl` commands.

## Deploy with `snctl`

Use the agent-focused commands to publish both the code and its configuration to the StreamNative cloud platform. The CLI captures all required settings, so you don't need to ship an `agent.yaml` file.

### Package as a ZIP archive

```bash theme={null}
snctl agents create \
  --tenant <tenant> \
  --namespace <namespace> \
  --name openai-multi-tool \
  --directory openai_multi_tool \
  --agent-framework openai \
  --inputs <input-topic> \
  --output <output-topic> \
  --agent-file openai_multi_tool.zip
```

* `--directory` must match the importable package path inside your ZIP archive.
* `--agent-framework openai` tells Orca Engine to load the OpenAI runtime adapter.
* Repeat the command with updated artifacts or specifications and `snctl agents update` to roll out changes. Use `snctl agents status` to monitor runtime health and `snctl agents trigger` to submit test payloads.

## Next steps

* Configure [service accounts and permissions](/agent-engine/agents-setup) before deploying to production namespaces.
* Store the OpenAI API key and other credentials in StreamNative cloud secrets so the runtime injects them securely.
* Run local or CI tests that import the package and call `root_agent` to verify tool wiring before publishing updates.
* Use the StreamNative cloud console to review managed tool assignments and ensure your OpenAI agent can discover required Model Context Protocol (MCP) integrations.


# Local MCP Server
Source: https://docs.streamnative.io/agent-engine/mcp/local-mcp-server

Install and run the open-source StreamNative MCP Server on your machine to connect AI agents to StreamNative Cloud, Kafka, or Pulsar clusters.

The local StreamNative MCP Server (`snmcp`) is an open-source binary that runs on your machine and exposes StreamNative Cloud, Apache Kafka, and Apache Pulsar clusters as Model Context Protocol (MCP) tools. Your IDE copilot or agent runtime connects to the local server over stdio or SSE, and the server translates MCP tool calls into cluster operations.

Source code and Helm charts are available on [GitHub](https://github.com/streamnative/streamnative-mcp-server).

## Prerequisites

Before you install, make sure you have:

* **For StreamNative Cloud**: a StreamNative Cloud account, organization ID, and a [service account API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview).
* **For external Kafka**: bootstrap server addresses and credentials (if authentication is enabled).
* **For external Pulsar**: the web service URL, service URL, and an authentication token (if required).

## Install

<Tabs>
  <Tab title="Homebrew (macOS / Linux)">
    ```bash theme={null}
    brew install streamnative/streamnative/snmcp
    ```

    Upgrade to the latest version:

    ```bash theme={null}
    brew upgrade snmcp
    ```
  </Tab>

  <Tab title="Docker">
    ```bash theme={null}
    docker pull streamnative/snmcp:latest
    ```

    Run with stdio (StreamNative Cloud example):

    ```bash theme={null}
    docker run --rm -i \
      -e SNMCP_ORGANIZATION=my-org \
      -e SNMCP_KEY_FILE=/keys/key.json \
      -v /path/to/key-file.json:/keys/key.json \
      streamnative/snmcp:latest \
      stdio
    ```

    Run with SSE:

    ```bash theme={null}
    docker run --rm -i -p 9090:9090 \
      -e SNMCP_ORGANIZATION=my-org \
      -e SNMCP_KEY_FILE=/keys/key.json \
      -v /path/to/key-file.json:/keys/key.json \
      streamnative/snmcp:latest \
      sse --http-addr :9090 --http-path /mcp
    ```
  </Tab>

  <Tab title="Build from source">
    Requires [Go 1.22+](https://go.dev/dl/):

    ```bash theme={null}
    git clone https://github.com/streamnative/streamnative-mcp-server.git
    cd streamnative-mcp-server
    make build
    ```

    The binary is created at `./bin/snmcp`.
  </Tab>
</Tabs>

## Quick start: connect to StreamNative Cloud

1. Create a service account and download the key file. See [Manage service accounts](/cloud/security/authentication/service-accounts/manage-service-accounts) for details.

2. Run the local server in stdio mode:

   ```bash theme={null}
   snmcp stdio --organization my-org --key-file /path/to/key-file.json
   ```

3. The server starts and waits for MCP requests on stdin/stdout. Point your IDE or agent runtime to this process (see [IDE integration](#ide-integration) below).

## Connect to an external Kafka cluster

Use the `--kafka-*` flags to connect to any Kafka cluster, including clusters outside StreamNative Cloud.

```bash theme={null}
snmcp stdio \
  --use-external-kafka \
  --kafka-bootstrap-servers broker1:9092,broker2:9092 \
  --kafka-auth-type SASL_SSL \
  --kafka-auth-mechanism PLAIN \
  --kafka-auth-user alice \
  --kafka-auth-pass "$KAFKA_PASSWORD" \
  --kafka-use-tls
```

**Additional Kafka flags:**

| Flag                                   | Description                                                |
| -------------------------------------- | ---------------------------------------------------------- |
| `--kafka-bootstrap-servers`            | Comma-separated list of broker addresses                   |
| `--kafka-auth-type`                    | Kafka auth type                                            |
| `--kafka-auth-mechanism`               | SASL mechanism (`PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`) |
| `--kafka-auth-user`                    | SASL username                                              |
| `--kafka-auth-pass`                    | SASL password                                              |
| `--kafka-schema-registry-url`          | Schema Registry endpoint                                   |
| `--kafka-schema-registry-auth-user`    | Schema Registry auth username                              |
| `--kafka-schema-registry-auth-pass`    | Schema Registry auth password                              |
| `--kafka-schema-registry-bearer-token` | Schema Registry bearer token                               |
| `--kafka-use-tls`                      | Enable TLS for broker connections                          |
| `--kafka-client-key-file`              | Kafka mTLS client key file                                 |
| `--kafka-client-cert-file`             | Kafka mTLS client certificate file                         |
| `--kafka-ca-file`                      | Kafka CA certificate file                                  |

## Connect to an external Pulsar cluster

Use the `--pulsar-*` flags to connect to any Pulsar cluster.

```bash theme={null}
snmcp stdio \
  --use-external-pulsar \
  --pulsar-web-service-url http://localhost:8080 \
  --pulsar-service-url pulsar://localhost:6650 \
  --pulsar-token "$PULSAR_TOKEN"
```

**Additional Pulsar flags:**

| Flag                                        | Description                            |
| ------------------------------------------- | -------------------------------------- |
| `--pulsar-web-service-url`                  | Pulsar web service URL                 |
| `--pulsar-service-url`                      | Pulsar binary service URL              |
| `--pulsar-token`                            | Authentication token                   |
| `--pulsar-auth-plugin`                      | Pulsar auth plugin                     |
| `--pulsar-auth-params`                      | Pulsar auth parameters                 |
| `--pulsar-tls-allow-insecure-connection`    | Allow insecure TLS connections         |
| `--pulsar-tls-enable-hostname-verification` | Enable TLS host name verification      |
| `--pulsar-tls-trust-certs-file-path`        | Path to trusted TLS certificate bundle |
| `--pulsar-tls-cert-file`                    | Client TLS certificate file            |
| `--pulsar-tls-key-file`                     | Client TLS key file                    |

> In external Kafka and external Pulsar modes, do not pass `--features`. The server enables the matching tool groups automatically.

## IDE integration

### Claude Desktop (stdio)

Add the following to your Claude Desktop MCP configuration file (`claude_desktop_config.json`):

```json theme={null}
{
  "mcpServers": {
    "streamnative": {
      "command": "snmcp",
      "args": [
        "stdio",
        "--organization", "${STREAMNATIVE_CLOUD_ORGANIZATION_ID}",
        "--key-file", "${STREAMNATIVE_CLOUD_KEY_FILE}",
        "--features", "all"
      ]
    }
  }
}
```

### Claude Code

Add the server using the Claude Code CLI:

```bash theme={null}
claude mcp add streamnative -- snmcp stdio --organization my-org --key-file /path/to/key-file.json --features all
```

### VS Code

Create or update `.vscode/mcp.json` in your workspace:

```json theme={null}
{
  "servers": {
    "streamnative": {
      "command": "snmcp",
      "args": [
        "stdio",
        "--organization", "${STREAMNATIVE_CLOUD_ORGANIZATION_ID}",
        "--key-file", "${STREAMNATIVE_CLOUD_KEY_FILE}",
        "--features", "all"
      ]
    }
  }
}
```

### Cursor

Add the following to your Cursor MCP configuration file (`.cursor/mcp.json`):

```json theme={null}
{
  "mcpServers": {
    "streamnative": {
      "command": "snmcp",
      "args": [
        "stdio",
        "--organization", "${STREAMNATIVE_CLOUD_ORGANIZATION_ID}",
        "--key-file", "${STREAMNATIVE_CLOUD_KEY_FILE}",
        "--features", "all"
      ]
    }
  }
}
```

## Feature control

Use the `--features` flag to select which tool groups are available. This controls token usage and limits the tool catalog to what you need.

```bash theme={null}
# Enable only Kafka admin and client tools (Cloud mode)
snmcp stdio --organization my-org --key-file /path/to/key-file.json --features kafka-admin,kafka-client

# Enable all Pulsar tools (Cloud mode)
snmcp stdio --organization my-org --key-file /path/to/key-file.json --features all-pulsar

# Enable everything (Cloud mode)
snmcp stdio --organization my-org --key-file /path/to/key-file.json --features all
```

See the [MCP Tools Reference](/agent-engine/sn-remote-mcp/remote-mcp-tools-reference) for the complete list of tool groups and their descriptions.

Mixed administration feature groups use separate read and write MCP tools. For example, Kafka topic administration uses `kafka_admin_topics_read` and `kafka_admin_topics_write`, and Pulsar topic administration uses `pulsar_admin_topic_read` and `pulsar_admin_topic_write`. Treat older mixed tool names such as `kafka_admin_topics` or `pulsar_admin_topic` as legacy names, not the current default tool surface.

## SSE server mode

Run the local server as an SSE endpoint for multi-client setups:

```bash theme={null}
snmcp sse --http-addr :8080 --http-path /mcp --organization my-org --key-file /path/to/key-file.json --features all
```

Clients can connect to `http://localhost:8080/mcp/sse`.

Additional endpoints:

* Health check: `http://localhost:8080/mcp/healthz`
* Readiness check: `http://localhost:8080/mcp/readyz`

### Multi-session Pulsar mode (SSE only)

For external Pulsar, you can enable per-user session management on the SSE server:

```bash theme={null}
snmcp sse --http-addr :9090 --http-path /mcp \
  --use-external-pulsar \
  --pulsar-web-service-url http://pulsar.example.com:8080 \
  --pulsar-service-url pulsar://pulsar.example.com:6650 \
  --multi-session-pulsar \
  --session-cache-size 100 \
  --session-ttl-minutes 30
```

In this mode, requests to SSE and message endpoints must include `Authorization: Bearer <token>`.

## Read-only mode

Restrict the server to read-only operations:

```bash theme={null}
snmcp stdio --organization my-org --key-file /path/to/key-file.json --features all --read-only
```

In read-only mode, the server exposes only tools that inspect resources, such as listing topics, peeking messages, and retrieving metrics. Write or destructive tools, such as `kafka_admin_topics_write` and `pulsar_admin_topic_write`, are not registered. The server also blocks operations that create, modify, or delete resources.

## Command logging

Enable command request and response logging for troubleshooting:

```bash theme={null}
snmcp stdio --organization my-org --key-file /path/to/key-file.json --features all \
  --enable-command-logging \
  --log-file /tmp/snmcp.log
```

## What's next

* Browse available tool groups in the [MCP Tools Reference](/agent-engine/sn-remote-mcp/remote-mcp-tools-reference).
* Compare with the managed [Remote MCP Server](/agent-engine/sn-remote-mcp/remote-mcp-overview) for a zero-install, team-shared alternative.
* Visit the [GitHub repository](https://github.com/streamnative/streamnative-mcp-server) for the latest releases, Helm charts, and contribution guidelines.


# StreamNative MCP Server
Source: https://docs.streamnative.io/agent-engine/mcp/mcp-overview

Connect AI agents and IDEs to StreamNative Cloud clusters using the Model Context Protocol (MCP).

The StreamNative MCP Server lets AI agents and IDE copilots interact with your StreamNative Cloud clusters through the [Model Context Protocol (MCP)](https://modelcontextprotocol.io/). MCP is an open standard that provides a unified way for AI models to discover and call external tools. With the StreamNative MCP Server, your agents can list topics, produce and consume messages, manage schemas, inspect cluster health, and more - all through natural language prompts in your IDE or agent runtime.

StreamNative offers two deployment options so you can choose the setup that fits your workflow.

## Deployment options

### Local MCP Server

The [local MCP Server](/agent-engine/mcp/local-mcp-server) is an open-source binary (`snmcp`) that runs on your machine. It connects to StreamNative Cloud, standalone Kafka clusters, or standalone Pulsar clusters.

**Best for:**

* Individual developers who want full control over their MCP setup
* Connecting to external Kafka or Pulsar clusters outside StreamNative Cloud
* Air-gapped or restricted environments where outbound connections to managed services are limited
* Quick experimentation and prototyping

### Remote MCP Server

The [Remote MCP Server](/agent-engine/sn-remote-mcp/remote-mcp-overview) is a managed service hosted at `https://mcp.streamnative.cloud`. StreamNative operates the server so you do not need to install or update anything locally.

**Best for:**

* Teams that want a shared, always-current MCP endpoint with no local setup
* Organizations that require centralized [governance and access controls](/agent-engine/sn-remote-mcp/remote-mcp-governance)
* Production environments where administrators control which tools and clusters are available

## Comparison

| Feature                     | Local MCP Server                                                                 | Remote MCP Server                                                                                         |
| --------------------------- | -------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| Installation                | Install binary locally                                                           | No installation required                                                                                  |
| Updates                     | Manual (Homebrew, Docker, or from source)                                        | Automatic - always current                                                                                |
| StreamNative Cloud clusters | Yes                                                                              | Yes                                                                                                       |
| External Kafka clusters     | Yes                                                                              | No                                                                                                        |
| External Pulsar clusters    | Yes                                                                              | No                                                                                                        |
| Organization-level tools    | No                                                                               | Yes - [StreamNative Cloud control-plane tools](/agent-engine/sn-remote-mcp/remote-mcp-organization-tools) |
| Cluster-level governance    | No                                                                               | Yes - [access modes and allowed tools](/agent-engine/sn-remote-mcp/remote-mcp-governance)                 |
| Authentication              | Service account API keys                                                         | OAuth 2.1 or service account API keys                                                                     |
| Multi-user sharing          | Self-managed sharing (for example, SSE deployment with your own access controls) | Shared endpoint across teams                                                                              |
| Agents as tools             | No                                                                               | Yes - [Agents as Tools](/agent-engine/sn-remote-mcp/remote-mcp-agents-as-tools)                           |
| Functions as tools          | Yes (Pulsar Functions)                                                           | Yes - [Functions as Tools](/agent-engine/sn-remote-mcp/remote-mcp-functions-as-tools)                     |

## Get started

<CardGroup>
  <Card title="Local MCP Server" icon="laptop" href="/agent-engine/mcp/local-mcp-server">
    Install the open-source binary and connect to any Pulsar or Kafka cluster from your machine.
  </Card>

  <Card title="Remote MCP Server" icon="cloud" href="/agent-engine/sn-remote-mcp/remote-mcp-overview">
    Connect to the managed MCP endpoint and start using tools with no local setup.
  </Card>
</CardGroup>

## Learn more

* [MCP Tools Reference](/agent-engine/sn-remote-mcp/remote-mcp-tools-reference) - browse all available tool groups for StreamNative Cloud, Pulsar, and Kafka.
* [Organization-Level Tools](/agent-engine/sn-remote-mcp/remote-mcp-organization-tools) - use root and organization sessions to discover clusters and manage StreamNative Cloud resources.
* [Governance & Permissions](/agent-engine/sn-remote-mcp/remote-mcp-governance) - configure cluster-level access modes and allowed tools.
* [Feature Flags](/agent-engine/sn-remote-mcp/remote-mcp-features) - control which tool groups are active at connection time.
* [Local MCP Server setup](/agent-engine/mcp/local-mcp-server) - configure stdio/SSE mode and local feature flags.


# Access the Remote MCP Server
Source: https://docs.streamnative.io/agent-engine/sn-remote-mcp/remote-mcp-access

Learn how to authenticate and connect to the managed StreamNative MCP Server from IDEs and agent runtimes.

This page explains how to reach the managed StreamNative MCP Server during the preview program. Use it when you want to consume StreamNative-managed tools without hosting the server yourself.

## Prerequisites

* A StreamNative Cloud account with access to the organization, tenant, and clusters you plan to explore.
* The MCP Server preview feature enabled for your organization and for any clusters you want to expose. See [Governance & Permissions](/agent-engine/sn-remote-mcp/remote-mcp-governance).
* Either an interactive user login that supports OAuth 2.1 or an [API Key v2](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#api-key-v1-vs-api-key-v2) for a service account with MCP permissions. Root and organization endpoints require an AuthV2 organization audience (`urn:sn:cloud:<organization>`).
* An MCP-compatible client, such as VS Code, an Orca agent, or any runtime that speaks the Model Context Protocol.

## Authenticate with StreamNative Cloud

You can sign in through two supported flows:

* **OAuth 2.1** - ideal for IDEs. Initiate the sign-in from the client, approve the scopes in your browser, and the session token is stored by the IDE. For root endpoint OAuth, send `X-Organization` when the client supports custom headers, or select the organization in the browser sign-in flow.
* **Service account API keys** - send an API Key v2 as a bearer token from clients that run outside the browser (for example, CI/CD automation or headless agents). Rotate keys through the StreamNative Cloud Console.

## Choose the right endpoint

<Warning>
  Do not connect directly to `https://mcp.streamnative.cloud`. Use one of the
  MCP paths shown below.
</Warning>

| Mode          | Endpoint                                                                   | Use when                                                                                                                                                                                                                                                                                                                 |
| ------------- | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Root          | `https://mcp.streamnative.cloud/mcp`                                       | Your client should connect to the shared root path and let the server resolve the organization from the AuthV2 organization audience. For OAuth discovery, send `X-Organization: <organization>` or select the organization during sign-in. For bearer tokens, the token must contain exactly one organization audience. |
| Organization  | `https://mcp.streamnative.cloud/mcp/x/<organization>`                      | You want organization-level tools first, including cluster discovery, cluster switching, and StreamNative Cloud control-plane tools.                                                                                                                                                                                     |
| Fixed cluster | `https://mcp.streamnative.cloud/mcp/x/<organization>/<instance>/<cluster>` | You want one stable Pulsar or Kafka cluster context, a cluster-scoped tool catalog, and support for dynamic Pulsar Function or Orca agent tools.                                                                                                                                                                         |

For example, organization `o-sndev`, instance `instance-1`, and cluster `c-cluster` resolves to `https://mcp.streamnative.cloud/mcp/x/o-sndev/instance-1/c-cluster`.

For root and organization-level workflows, see [Root and Organization-Level Tools](/agent-engine/sn-remote-mcp/remote-mcp-organization-tools).

## Connect from Orca Engine runtimes

* Configure managed tools via `snctl` when submit agents.
* When you deploy with `snctl agents create ... --agent-tools-config`, Orca Engine authenticates to the remote MCP server with the service account already assigned to the agent runtime. This keeps cluster access scoped to the permissions you granted that service account.
* Redeploy or restart agents that depend on remote MCP tools so the runtime attaches the new connection metadata.
* The Orca engine runtime automatically watches for `tools changed` notifications and reloads the agent if the remote catalog diverges from the cached copy.

## Use from other MCP clients

* Set the remote host to the root, organization, or cluster endpoint and follow the client's instructions for either OAuth 2.1 or API key authentication.
* For API key authentication, send `Authorization: Bearer <api-key>`. Use API Key v2 for root and organization endpoints because they require organization-scoped authentication.
* For root endpoint OAuth discovery, send `X-Organization: <organization>` when your client supports custom headers. If your client cannot send this header, the interactive OAuth flow lets you select an organization during sign-in. If both the token and the header provide an organization, they must match.
* Optionally send `X-MCP-Features` to narrow the tool catalog and `X-MCP-Readonly` to request read-only behavior. These headers are evaluated per session and cannot grant tools beyond administrator-configured allow lists. See [Remote MCP Headers and Feature Selection](/agent-engine/sn-remote-mcp/remote-mcp-features).
* Persist refresh tokens or API keys according to your organization's security policies. Delete unused sessions from the StreamNative Cloud console.
* In VS Code, follow the [Use MCP servers in VS Code](https://code.visualstudio.com/docs/copilot/customization/mcp-servers#_use-mcp-tools-in-agent-mode) to enable MCP.

### Example: connect with API Key v2 and custom headers

Use a root endpoint when the client can send `X-Organization`, or use an organization endpoint when you want the organization bound in the URL.

```bash theme={null}
curl -N https://mcp.streamnative.cloud/mcp \
  -H "Authorization: Bearer <api-key-v2>" \
  -H "X-Organization: o-sndev" \
  -H "X-MCP-Features: sncloud_context,sncloud_clusters" \
  -H "X-MCP-Readonly: true" \
  -H "Accept: application/json, text/event-stream" \
  -H "Content-Type: application/json"
```

Use the same headers in MCP clients that support custom HTTP headers. If a client cannot send `X-Organization`, prefer the organization endpoint `https://mcp.streamnative.cloud/mcp/x/<organization>`.

## Troubleshooting

* **401 Unauthorized** - confirm the account or API Key v2 includes the required MCP permissions and that the token has not expired.
* **403 Forbidden on `/mcp`** - confirm the bearer token contains exactly one AuthV2 organization audience and that `X-Organization`, when present, matches that audience.
* **Root OAuth discovery fails** - send `X-Organization: <organization>` when the client supports custom headers, or retry with an interactive client that can complete the organization picker flow.
* **Empty tool list** - ensure MCP is enabled for the organization and target cluster, the selected feature set is not empty, and your identity can access the resources.
* **Connection drops** - the preview service enforces idle timeouts. Reconnect or configure your client to reconnect automatically.


# Expose Orca Agents as MCP Tools
Source: https://docs.streamnative.io/agent-engine/sn-remote-mcp/remote-mcp-agents-as-tools

Publish Orca agents through the Remote StreamNative MCP Server so other clients can invoke them.

The Remote StreamNative MCP Server can publish Orca agents as Model Context Protocol (MCP) tools. This lets IDE copilots, external orchestrators, and other MCP-aware agents call your Orca workloads without building a custom integration layer.

## Automatic discovery and refresh

* Orca Engine registers agents that expose the required `root_agent` entry point and marks them as shareable through MCP.
* The managed server regularly discovers eligible agents and publishes them as MCP tools. Default tool names follow the pattern `agent_<tenant>_<namespace>_<agent>` when you do not supply overrides.
* Server-Sent Events (SSE) signal catalog changes. When the MCP server reports `tools changed`, Orca Engine reloads cached metadata so clients immediately see the latest instructions and tool hints.

## Metadata surfaced to MCP clients

* Each exported tool includes the agent name, description, and supported session mode so callers understand the conversational context before invoking it.
* When an agent declares structured input or output channels, the Remote MCP Server maps that contract into MCP tool parameters so clients know which fields to supply.
* Keep prompt instructions concise and results deterministic where possible so downstream tools can parse responses reliably.

## Customize tool expose metadata

Provide descriptive names and summaries to make the MCP tool easier to be used by LLM agents.

```json theme={null}
{
  "env": {
    "MCP_TOOL_NAME": "IncidentResponder",
    "MCP_TOOL_DESCRIPTION": "Guides engineers through triaging active StreamNative Cloud incidents."
  }
}
```

* Set `MCP_TOOL_NAME` and `MCP_TOOL_DESCRIPTION` as environment variables in the agent configuration through CLI deployment flags.
* When present, the Remote MCP Server uses these values instead of the default derived name and description.
* Administrators can require export metadata before publishing an agent so only well-documented tools appear in Remote StreamNative MCP server.

## Prepare an agent for MCP publication

1. Follow the [Google ADK agents guide](/agent-engine/develop-agents/agents-develop-adk) or [OpenAI agents guide](/agent-engine/develop-agents/agents-develop-openai) to produce a deployable artifact.
2. Store model credentials as StreamNative Cloud secrets and bind them with the `--secrets` flag during deployment.
3. Add `MCP_TOOL_NAME` and `MCP_TOOL_DESCRIPTION` to the agent environment, then redeploy so the discovery cycle can pick up the customized metadata.

## Control discovery with MCP features

* The Remote MCP Server exports shareable Orca agents when the endpoint allows dynamic tools, your identity can reach the associated tenant and namespace, and the current preview runtime supports agent discovery.
* Dynamic agent tools are available on fixed Pulsar cluster endpoints. They are not added after selecting a cluster inside an organization-level session with `sncloud_context_use_cluster`.
* To skip agent discovery from a client, send the `X-MCP-Features` header during connection setup and omit dynamic tool features from the list.
* If an administrator configured an allowed-tools list for the cluster, that list must include the dynamic feature IDs required by the current preview runtime.
* `agents-as-tools` is an accepted feature ID. In the current preview, include `functions-as-tools` or `pulsar-admin-functions` when you want to expose eligible agent tools.
* Provide the feature explicitly - `X-MCP-Features: pulsar-admin-functions,functions-as-tools` - when you want to mix agent tools with other MCP capabilities while keeping control over optional features.

## Consume the agent from MCP clients

* MCP clients invoke the tool using standard `call_tool` requests. The managed server forwards payloads to the corresponding Orca agent and streams responses back.
* The MCP server enforces StreamNative Cloud permissions, so users only see agents they are authorized to access.
* Use StreamNative observability dashboards and `snctl agents status` to monitor traffic routed through MCP.

This pattern lets you share domain-specific assistants - such as troubleshooting copilots or knowledge workers - with any MCP-aware application while keeping configuration centralized in StreamNative Cloud.


# Remote MCP Headers and Feature Selection
Source: https://docs.streamnative.io/agent-engine/sn-remote-mcp/remote-mcp-features

Configure headers and feature selection when connecting to the Remote StreamNative MCP Server.

The Remote StreamNative MCP Server exposes a wide catalog of tools. Clients can use HTTP headers to authenticate, bind root sessions to an organization, request read-only behavior, and narrow the tool catalog. Administrators can use the StreamNative Cloud Console to enable MCP access and set the maximum tool catalog.

<Note>
  Send these headers to a root, organization, or cluster endpoint. See [Remote
  MCP Access](/agent-engine/sn-remote-mcp/remote-mcp-access) for endpoint
  construction details.
</Note>

## Header reference

| Header           | Description                                                                                                                                                                                                                                                                                                                                                                                         |
| ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Authorization`  | Set to `Bearer <token>` for authenticated MCP requests. The token can be an OAuth 2.1 access token or a service account API key. For non-interactive clients, use an [API Key v2](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#api-key-v1-vs-api-key-v2).                                                                                                         |
| `X-Organization` | Optional organization hint for the root `/mcp` endpoint and root OAuth discovery. Use it when the organization is not present in the URL. The value must be a single organization path segment. If both the token and header provide an organization, they must match. If no header is supplied for an interactive root OAuth flow, the sign-in flow can prompt the user to select an organization. |
| `X-MCP-Readonly` | Requests read-only catalog behavior. Accepted true values are `true`, `1`, and `yes`. If the server access mode is Read-Only, mutations are blocked even when this header is omitted. False or omitted values cannot bypass a Read-Only server access mode.                                                                                                                                         |
| `X-MCP-Features` | Comma-separated list of feature identifiers that narrows the tool catalog by tool group. Values are trimmed and case-insensitive. A feature ID can enable multiple MCP tools, including separate read and write tools for the same resource family. The header also controls MCP resources, such as Pulsar resource paths, when the selected features expose resources.                             |

## How feature selection works

* When `X-MCP-Features` is omitted, the server uses the configured default tool set. Pulsar and organization sessions default broadly; Kafka sessions default to Kafka-compatible tools.
* If an administrator configured an allowed-tools list in the Console, the list acts as an allow list. `X-MCP-Features` can request only a subset of that allow list.
* Unsupported feature IDs cause the request to fail when the endpoint has an explicit allow list. Without an allow list, the server falls back to defaults.
* Kafka cluster sessions normalize broad requests such as `all` to Kafka-compatible features. Pulsar-only features requested on Kafka cluster endpoints are ignored; if none of the requested features apply, the request fails.
* Organization-level sessions use StreamNative Cloud features such as `streamnative-cloud`, `sncloud_context`, `sncloud_clusters`, and `sncloud_identity`. Pulsar or Kafka tools are added only after `sncloud_context_use_cluster` selects a cluster.
* Dynamic `functions-as-tools` and `agents-as-tools` run only on fixed Pulsar cluster endpoints. They are not added after a root or organization session selects a cluster with `sncloud_context_use_cluster`.
* Mixed administration feature groups can expose separate read and write tools. For example, `pulsar-admin-topics` can expose `pulsar_admin_topic_read` and `pulsar_admin_topic_write`, and `kafka-admin` can expose `kafka_admin_topics_read` and `kafka_admin_topics_write`.
* `X-MCP-Readonly: true` requests a read-only catalog from a read/write-enabled endpoint. The server omits write or destructive tools where possible and blocks mutation calls server-side. A server configured in Read-Only access mode is the ceiling and cannot be bypassed by omitting the header or setting it to false.
* Start with a minimal feature list to reduce token usage and improve tool selection accuracy.

## Available feature identifiers

Use these values inside the `X-MCP-Features` header. Separate multiple values with commas.

### Combination feature sets

| Feature ID   | Description                                                                      |
| ------------ | -------------------------------------------------------------------------------- |
| `all`        | Enables every available capability across StreamNative Cloud, Pulsar, and Kafka. |
| `all-pulsar` | Includes Pulsar tooling features.                                                |
| `pulsar`     | Alias for `all-pulsar`.                                                          |
| `all-kafka`  | Includes Kafka tooling features.                                                 |
| `kafka`      | Alias for `all-kafka`.                                                           |

### Kafka tooling

| Feature ID                    | Description                                                                                                                                            |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `kafka-admin`                 | Kafka administration tools, including topics, partitions, and consumer groups.                                                                         |
| `kafka-client`                | Kafka produce and consume tools.                                                                                                                       |
| `kafka-admin-schema-registry` | Schema Registry tools.                                                                                                                                 |
| `kafka-admin-kafka-connect`   | Kafka Connect tools. This feature ID is accepted, but Kafka Connect tools are not exposed on Remote MCP Kafka cluster sessions in the current preview. |

### Pulsar tooling

| Feature ID                         | Alias              | Description                       |
| ---------------------------------- | ------------------ | --------------------------------- |
| `pulsar-admin`                     |                    | Pulsar administration tools.      |
| `pulsar-client`                    |                    | Pulsar produce and consume tools. |
| `pulsar-admin-topics`              | `topics`           | Topic administration.             |
| `pulsar-admin-namespaces`          | `namespaces`       | Namespace administration.         |
| `pulsar-admin-tenants`             | `tenants`          | Tenant administration.            |
| `pulsar-admin-brokers`             | `brokers`          | Broker information.               |
| `pulsar-admin-brokers-status`      | `brokers-status`   | Broker status.                    |
| `pulsar-admin-broker-stats`        | `broker-stats`     | Broker statistics.                |
| `pulsar-admin-clusters`            | `clusters`         | Pulsar cluster metadata.          |
| `pulsar-admin-functions`           | `functions`        | Pulsar Functions administration.  |
| `pulsar-admin-functions-worker`    | `functions-worker` | Functions worker status.          |
| `pulsar-admin-sinks`               | `sinks`            | Pulsar IO sinks.                  |
| `pulsar-admin-sources`             | `sources`          | Pulsar IO sources.                |
| `pulsar-admin-namespace-policy`    | `namespace-policy` | Namespace policies.               |
| `pulsar-admin-ns-isolation-policy` | `ns-isolation`     | Namespace isolation policies.     |
| `pulsar-admin-packages`            | `packages`         | Pulsar packages.                  |
| `pulsar-admin-resource-quotas`     | `resource-quotas`  | Resource quotas.                  |
| `pulsar-admin-schemas`             | `schemas`          | Topic schemas.                    |
| `pulsar-admin-subscriptions`       | `subscriptions`    | Topic subscriptions.              |
| `pulsar-admin-topic-policy`        | `topic-policy`     | Topic-level policies.             |

### StreamNative Cloud tooling

| Feature ID           | Alias              | Description                                                                                                                              |
| -------------------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `streamnative-cloud` | `cloud`            | All StreamNative Cloud organization tool families.                                                                                       |
| `sncloud_context`    | `sncloud-context`  | Organization context, cluster discovery, and in-session cluster switching.                                                               |
| `sncloud_byoc`       | `sncloud-byoc`     | BYOC resources, including CloudConnection, CloudEnvironment, Volume, and PoolMember.                                                     |
| `sncloud_identity`   | `sncloud-identity` | Identity and RBAC resources, including User, ServiceAccount, ServiceAccountBinding, Role, RoleBinding, `OIDCProvider`, and IdentityPool. |
| `sncloud_clusters`   | `sncloud-clusters` | Instance and cluster resources, including Instance, PulsarInstance, PulsarCluster, PulsarGateway, and KafkaCluster.                      |

`sncloud_resource_catalog` and `sncloud_resource_schema` are helper tools for the BYOC, identity, and clusters feature families. They are included when you enable a domain feature that needs schema-guided manifest authoring. Domain read tools use names such as `sncloud_clusters_read`; domain write tools use names such as `sncloud_clusters_write`.

For root and organization-level tool workflows, see [Root and Organization-Level Tools](/agent-engine/sn-remote-mcp/remote-mcp-organization-tools).

### Dynamic tooling

| Feature ID           | Alias                       | Description                                                                                                                                                                                             |
| -------------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `functions-as-tools` |                             | Exposes eligible Pulsar Functions as callable MCP tools on fixed Pulsar cluster sessions.                                                                                                               |
| `agents-as-tools`    | `agents`, `agent-functions` | Accepted dynamic agent-tool feature ID. In the current preview, include `functions-as-tools` or `pulsar-admin-functions` when you want to expose eligible Orca agents on fixed Pulsar cluster sessions. |

## Examples

Bind root OAuth discovery to an organization:

```http theme={null}
X-Organization: o-sndev
```

Enable only StreamNative Cloud context tools on a root or organization endpoint:

```http theme={null}
X-MCP-Features: sncloud_context
```

Enable Pulsar topic administration and Pulsar client tools on a fixed Pulsar cluster endpoint:

```http theme={null}
X-MCP-Features: pulsar-admin-topics,pulsar-client
```

Enable dynamic Pulsar Function and Orca agent tools on a fixed Pulsar cluster endpoint:

```http theme={null}
X-MCP-Features: pulsar-admin-functions,functions-as-tools
```

Request read-only behavior from a read/write-enabled endpoint:

```http theme={null}
X-MCP-Readonly: true
```

Use API Key v2 for a non-interactive organization session:

```http theme={null}
Authorization: Bearer <api-key-v2>
X-MCP-Features: sncloud_context,sncloud_clusters
```


# Use Pulsar Functions as MCP Tools
Source: https://docs.streamnative.io/agent-engine/sn-remote-mcp/remote-mcp-functions-as-tools

Publish Pulsar Functions through the Remote StreamNative MCP Server so other clients can invoke them.

The Remote StreamNative MCP Server can publish Apache Pulsar Functions as Model Context Protocol (MCP) tools. This lets IDE copilots, external orchestrators, and other MCP-aware agents call your stream processing workloads without building a custom integration layer.

## Automatic discovery and refresh

* The Remote MCP Server regularly discovers Pulsar Functions that expose the required entry point within namespaces your StreamNative identity can access.
* Eligible functions are published as MCP tools. Default tool names follow the pattern `pulsar_function_<tenant>_<namespace>_<function>` when you do not provide overrides.
* Server-Sent Events (SSE) signal catalog changes. When the MCP server reports `tools changed`, Orca Engine reloads cached metadata so clients immediately see the latest schema and description updates.

## Metadata surfaced to MCP clients

* Exported tools include the function name, description, and tenancy scope so callers understand which workload they are invoking.
* The Remote MCP Server inspects each function's input and output schema and maps supported Pulsar schemas into MCP tool parameters so clients know which fields to supply.
* Use JSON or string output schemas when possible. Functions with missing schemas or unsupported output formats might not be exposed as reliable MCP tools. Capture schema definitions and document expected payload fields to keep downstream calls deterministic.

## Customize tool expose metadata

Provide descriptive names and summaries to make the MCP tool easier for LLM agents to use.

```json theme={null}
{
  "env": {
    "MCP_TOOL_NAME": "InventoryLookup",
    "MCP_TOOL_DESCRIPTION": "Returns stock levels for a given SKU across all warehouses."
  }
}
```

* Set `MCP_TOOL_NAME` and `MCP_TOOL_DESCRIPTION` in the function environment through CLI deployment flags.
* When present, the Remote MCP Server uses these values instead of the default derived name and description.
* Administrators can require export metadata before publishing a function so only well-documented tools appear in the Remote StreamNative MCP Server.

## Prepare functions for publication

1. Confirm the Pulsar Function runs in a tenant and namespace that the Remote MCP Server can reach through your StreamNative Cloud account or service principal.
2. Define clear input and output schemas and update the function description so MCP users recognize the required payload fields.
3. Apply the `MCP_TOOL_NAME` and `MCP_TOOL_DESCRIPTION` variables, then redeploy or restart the function so the discovery cycle picks up the customized metadata.

## Control discovery with MCP features

* The Remote MCP Server exports Pulsar Functions when the endpoint allows dynamic tools and your identity can reach the associated tenant and namespace.
* Dynamic function tools are available on fixed Pulsar cluster endpoints. They are not added after selecting a cluster inside an organization-level session with `sncloud_context_use_cluster`.
* To skip dynamic tool discovery from a client, send the `X-MCP-Features` header during connection setup and omit `functions-as-tools` from the list.
* If an administrator configured an allowed-tools list for the cluster, that list must include `functions-as-tools` for clients to request it.
* Provide the feature explicitly - `X-MCP-Features: pulsar-admin,pulsar-client,functions-as-tools` - when you want to combine function tools with other MCP capabilities while keeping control over optional features.

## Consume the function from MCP clients

* MCP clients invoke the tool using standard `call_tool` requests. The managed server forwards payloads to the corresponding Pulsar Function and streams responses back.
* The MCP server enforces StreamNative Cloud permissions, so users only see functions they are authorized to access.
* Use StreamNative observability dashboards to monitor traffic routed through MCP and track invocation latency and throughput.

This pattern lets you share existing Pulsar Function automation with any MCP-aware application while keeping configuration centralized in StreamNative Cloud.


# Governance & Permissions
Source: https://docs.streamnative.io/agent-engine/sn-remote-mcp/remote-mcp-governance

Configure organization and cluster access modes, allowed tools, and authentication for the Remote StreamNative MCP Server.

The Remote StreamNative MCP Server provides two layers of access control: **server access mode** and **user permissions**. Together, these layers determine what operations an MCP client can perform on StreamNative Cloud organization resources or a selected cluster.

## How access control works

When an MCP client sends a request, the server evaluates three things:

1. **MCP availability**: whether MCP is enabled for the organization and whether the target cluster is enabled for MCP.
2. **Server access mode**: the maximum level of operations the MCP server allows on the organization entry or cluster entry (read-only or read/write). This is configured by an administrator in the Console.
3. **User permissions**: the StreamNative Cloud roles assigned to the authenticated user or service account. These roles determine which resources the user can access.

The server access mode sets the ceiling. User permissions further restrict what is allowed within that ceiling.

<Note>
  Both layers must permit an operation for it to succeed. If either layer denies
  the operation, the request fails.
</Note>

**Examples:**

| Server access mode | User role      | Result                                                      |
| ------------------ | -------------- | ----------------------------------------------------------- |
| Read/write         | Administrator  | Full access: user can read and write                        |
| Read/write         | Read-only role | User can only read because user permissions restrict access |
| Read-only          | Administrator  | User can only read because server mode restricts access     |
| Read-only          | Read-only role | User can only read                                          |

## Enable MCP for your organization

Before you can configure MCP on individual clusters, enable the feature for your organization.

1. In the StreamNative Cloud Console, navigate to **Settings > Preview Features**.
2. Enable **MCP Server**.
3. After you enable the feature, the **Settings > MCP** page displays a table of all clusters in your organization with their MCP status.

Organization-level endpoints, including `https://mcp.streamnative.cloud/mcp` and `https://mcp.streamnative.cloud/mcp/x/<organization>`, require organization-level MCP access and AuthV2 organization authentication. Fixed cluster endpoints also require organization-level MCP access. Use root and organization endpoints for StreamNative Cloud organization tools and in-session cluster discovery. See [Root and Organization-Level Tools](/agent-engine/sn-remote-mcp/remote-mcp-organization-tools).

## Configure organization MCP access

Organization MCP settings control root and organization endpoints. They also set the access mode and maximum tool catalog for StreamNative Cloud organization tools.

Configure these values from the organization MCP settings when available:

* **Enabled** - makes root, organization, and fixed cluster endpoints available for the organization.
* **Access mode** - `Read-Only` blocks organization-level mutation calls. `Read/Write` allows mutation calls when user permissions also allow them.
* **Allowed tools** - sets the maximum organization tool catalog, such as cluster discovery, BYOC resources, identity and RBAC resources, and cluster resources.

Manage organization-level MCP access from the StreamNative Cloud Console.

## Enable MCP per cluster

From the organization-level MCP settings page (**Settings > MCP**):

1. Locate the Pulsar or Kafka cluster in the table.
2. Toggle MCP **on** for the cluster.
3. Click **Configure** to navigate to the cluster-level MCP settings.

You can also disable MCP for a cluster at any time by toggling it off. Disabling MCP disconnects active MCP sessions for that cluster, removes the cluster from organization-level discovery, and prevents new fixed cluster sessions from using the cluster endpoint.

## Configure access mode

The access mode controls the maximum level of operations the MCP server allows. Configure it from the cluster's MCP **Permissions** tab in the Console.

### Read-Only

The MCP server blocks mutation operations. Read-only catalogs omit write or destructive tools where possible. If a write-capable organization tool is still present for compatibility, `apply` and `delete` calls fail while read-only mode is active. Use read-only mode for:

* Viewing cluster information and health metrics
* Listing tenants, namespaces, and topics
* Peeking at messages
* Retrieving schemas and configuration
* Viewing subscription and consumer group status

### Read/Write

The MCP server exposes read tools plus write tools that modify resources. The read/write split helps MCP clients distinguish safe inspection from mutations. For example, clients can use `pulsar_admin_topic_read`, `kafka_admin_topics_read`, or `sncloud_clusters_read` for inspection and reserve `pulsar_admin_topic_write`, `kafka_admin_topics_write`, or `sncloud_clusters_write` for changes.

Write tools can:

* Create and delete topics
* Produce messages
* Manage schemas (create, update, delete)
* Create and manage subscriptions
* Manage connectors and functions

<Warning>
  Set the access mode to **Read-Only** for clusters where MCP users should not
  modify resources. This provides a safety net even if a user has broad
  permissions in StreamNative Cloud.
</Warning>

## Configure allowed tools

From the cluster's MCP **Permissions** tab, you can select which tool groups are available through MCP for this cluster. This provides fine-grained control over what MCP clients can do.

* **Pulsar clusters** display Pulsar, StreamNative Cloud, and compatible dynamic tool groups.
* **Kafka clusters** display Kafka and StreamNative Cloud tool groups. Kafka Connect can be accepted as a feature ID, but Kafka Connect tools are not exposed on Remote MCP Kafka cluster sessions in the current preview.

The selected tools form an allow list. If a client also sends `X-MCP-Features`, the request can only narrow the catalog to a subset of this allow list. It cannot enable tools that an administrator disabled in the Console.

The available tool groups match the tool IDs documented in the [MCP Tools Reference](/agent-engine/sn-remote-mcp/remote-mcp-tools-reference).

### Manage tool groups

1. Navigate to the cluster's MCP settings and open the **Permissions** tab.
2. Use the **Select All** checkbox to enable or disable all tool groups at once, or toggle individual tool groups.
3. Click **Save Changes** to apply your selection.
4. To revert to the default configuration, click **Reset to Defaults**.

<Tip>
  Start with a minimal set of tool groups and add more as needed. Fewer tools
  reduce token usage and improve tool selection accuracy for AI agents.
</Tip>

## Authentication and user permissions

The authenticated identity determines which StreamNative Cloud resources the MCP client can access.

### OAuth 2.1

When a user signs in through OAuth 2.1 (for example, from an IDE), their StreamNative Cloud identity and roles determine access. The MCP server scopes tool discovery and execution to the resources the user is authorized to reach.

### Service account API keys

When a client authenticates with a service account API key, the service account's assigned roles determine access. Use [API Key v2](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#api-key-v1-vs-api-key-v2) for automated clients, CI/CD pipelines, and headless agent runtimes that need organization-scoped access.

For setup details, see [Connect & Authenticate](/agent-engine/sn-remote-mcp/remote-mcp-access).

## Console UI reference

The cluster-level MCP settings page has two tabs:

### Connection tab

Displays the cluster's MCP endpoint URL and provides ready-to-use configuration examples for:

* **Claude Code** - `claude mcp add-json` command
* **Cursor** - JSON configuration for `.cursor/mcp.json`
* **VS Code** - JSON configuration for `.vscode/mcp.json`
* **cURL** - command-line example for testing the endpoint

The connection examples support OAuth 2.1 and API key authentication. API key examples send `Authorization: Bearer <api-key>`. Use API Key v2 for organization-scoped automation.

### Permissions tab

Provides controls for:

* **Access mode** - toggle between Read-Only and Read/Write
* **Allowed tools** - select which tool groups are available, with Select All, individual toggles, Reset to Defaults, and Save Changes

## Manage MCP settings in the Console

Use the StreamNative Cloud Console to manage MCP availability, access mode, and allowed tools for your organization and clusters. Client headers can narrow the configured tool catalog but cannot enable tools that an administrator disabled in the Console.

For the full header reference, see [Remote MCP Headers and Feature Selection](/agent-engine/sn-remote-mcp/remote-mcp-features).


# Use Root and Organization-Level MCP Tools
Source: https://docs.streamnative.io/agent-engine/sn-remote-mcp/remote-mcp-organization-tools

Use root and organization Remote MCP sessions to inspect and manage StreamNative Cloud resources.

The Remote StreamNative MCP Server supports root and organization-level sessions in addition to fixed cluster sessions. Use a root or organization-level session when you want an MCP client to discover MCP-enabled clusters, switch cluster context during a session, or manage StreamNative Cloud control-plane resources.

## Choose a root or organization endpoint

Use one of the following endpoints:

| Mode         | Endpoint                                              | Use case                                                                                                                                                                                                                           |
| ------------ | ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Root         | `https://mcp.streamnative.cloud/mcp`                  | Connect without putting the organization in the URL. The server resolves the organization from the AuthV2 audience in your token. For interactive OAuth flows, provide `X-Organization` or select the organization during sign-in. |
| Organization | `https://mcp.streamnative.cloud/mcp/x/<organization>` | Bind the session to one organization in the URL. Use this when your client or automation can store one endpoint per organization.                                                                                                  |

After connection, the session starts with StreamNative Cloud organization tools. The global identity tool `sncloud_context_whoami` is available for the session. Other organization tools are controlled by the organization feature set. Cluster-specific Pulsar, Kafka, and log tools become available only after you select a cluster with `sncloud_context_use_cluster`.

<Note>
  For a fixed cluster session, use `https://mcp.streamnative.cloud/mcp/x/<organization>/<instance>/<cluster>`. See [Remote MCP Access](/agent-engine/sn-remote-mcp/remote-mcp-access).
</Note>

## Discover and select clusters

Organization-level sessions expose context tools for finding and selecting MCP-enabled clusters.

| Tool                                 | Description                                                                                                                                             |
| ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `sncloud_context_whoami`             | Show the authenticated StreamNative Cloud identity and current organization context. This tool is global and is not controlled by `sncloud_context`.    |
| `sncloud_context_available_clusters` | List MCP-enabled Pulsar and Kafka clusters in the current organization, including instance name, cluster name, cluster type, access mode, and endpoint. |
| `sncloud_context_use_cluster`        | Select a Pulsar or Kafka cluster for the current MCP session. Parameters: `instanceName` and `clusterName`.                                             |
| `sncloud_context_reset`              | Clear the selected cluster and return to organization-only context.                                                                                     |

When you select a cluster, the current MCP session becomes a hybrid session. It keeps organization-level tools and adds static Pulsar or Kafka tools for the selected cluster. The selected cluster overlay also adds `sncloud_logs` for logs from Functions, Sources, Sinks, and Kafka Connect connectors in the selected cluster. Dynamic tools such as `functions-as-tools` and `agents-as-tools` are available on fixed cluster endpoints, but not after in-session cluster switching.

## Use StreamNative Cloud prompts

Organization-level sessions also expose StreamNative Cloud prompts that help clients discover and inspect enabled clusters.

| Prompt                  | Description                                                                                                  |
| ----------------------- | ------------------------------------------------------------------------------------------------------------ |
| `list-sncloud-clusters` | Lists MCP-enabled clusters in the current organization, including the cluster type and endpoint metadata.    |
| `read-sncloud-cluster`  | Reads one cluster resource. Parameters: `name` and `type`. `type` must be `PulsarCluster` or `KafkaCluster`. |

Use the `clusterType` returned by `list-sncloud-clusters` when you call `read-sncloud-cluster`. Disabled or non-exposed clusters aren't returned.

## Manage StreamNative Cloud resources

Organization-level sessions expose domain-scoped tools for StreamNative Cloud resources.

| Feature            | Tools                                             | Resources                                                                                                |
| ------------------ | ------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| `sncloud_byoc`     | `sncloud_byoc_read`, `sncloud_byoc_write`         | `CloudConnection`, `CloudEnvironment`, `Volume`, `PoolMember`                                            |
| `sncloud_identity` | `sncloud_identity_read`, `sncloud_identity_write` | `User`, `ServiceAccount`, `ServiceAccountBinding`, `Role`, `RoleBinding`, `OIDCProvider`, `IdentityPool` |
| `sncloud_clusters` | `sncloud_clusters_read`, `sncloud_clusters_write` | `Instance`, `PulsarInstance`, `PulsarCluster`, `PulsarGateway`, `KafkaCluster`                           |

Read tools support `list` and `get` operations. Write tools support `apply` and `delete` operations. `get` and `delete` require `name`. `apply` requires `manifest`, a JSON string that contains one StreamNative Cloud resource.

After `sncloud_context_use_cluster` selects a cluster, the session also exposes `sncloud_logs` for Functions, Sources, Sinks, and Kafka Connect connectors in the selected cluster.

All domain tool families can use these schema helpers:

| Tool                       | Description                                                                                                                                                                                                                                                                                                                                                         |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `sncloud_resource_catalog` | List supported resource kinds for a domain: `byoc`, `clusters`, or `identity`.                                                                                                                                                                                                                                                                                      |
| `sncloud_resource_schema`  | Return schema guidance for one resource kind. The default `summary` format is optimized for authoring manifests; `example` returns an example manifest; `jsonschema` returns runtime Kubernetes schema when available; `all` returns summary plus runtime metadata. Use `paths` such as `spec.broker` or `spec.clusterRefs` to keep runtime schema responses small. |

<Warning>
  Write operations can change organization-level resources. Use `dry_run=true`
  before applying changes, and keep the Remote MCP access mode set to
  **Read-Only** unless mutation tools are required.
</Warning>

## Use the recommended mutation workflow

When you create or update a StreamNative Cloud resource through MCP, use this workflow:

1. Call `sncloud_resource_catalog` for the resource domain, such as `clusters` or `identity`.
2. Call `sncloud_resource_schema` for the resource type, such as `PulsarCluster`, `KafkaCluster`, or `ServiceAccount`.
3. Call the matching read tool to inspect existing resources.
4. Compose the resource manifest as a JSON string.
5. Call the matching write tool with `operation=apply` and `dry_run=true`.
6. Review the validation result.
7. Repeat the apply operation with `dry_run=false` only when the dry-run result is acceptable.

The `manifest` argument must be a JSON string. Do not pass YAML or a structured object. Omit `status`, `metadata.managedFields`, and other read-only fields copied from read responses. The schema summary is guidance, not a replacement for server-side validation. Runtime OpenAPI schemas do not encode every cross-field or cross-resource constraint.

## Control organization tools with features

Use `X-MCP-Features` to narrow the organization tool catalog by tool group. For example, `X-MCP-Features: sncloud_context,sncloud_clusters` exposes context tools plus cluster-domain resources. A feature ID can enable multiple MCP tools, including read/write pairs such as `sncloud_clusters_read` and `sncloud_clusters_write`. If an administrator configured allowed tools for the organization MCP entry, the header can only request a subset of that allow list. If you omit `X-MCP-Features`, the server uses the configured organization feature set.

| Feature ID                               | Enables                                                                                                                                       |
| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `streamnative-cloud` or `cloud`          | All StreamNative Cloud organization tool families.                                                                                            |
| `sncloud_context` or `sncloud-context`   | Cluster discovery and context switch tools: `sncloud_context_available_clusters`, `sncloud_context_use_cluster`, and `sncloud_context_reset`. |
| `sncloud_byoc` or `sncloud-byoc`         | BYOC control-plane tools.                                                                                                                     |
| `sncloud_identity` or `sncloud-identity` | Identity and RBAC resource tools.                                                                                                             |
| `sncloud_clusters` or `sncloud-clusters` | Instance and cluster resource tools.                                                                                                          |

For the full header reference, see [Remote MCP Headers and Feature Selection](/agent-engine/sn-remote-mcp/remote-mcp-features).

## Manage MCP settings in the Console

Use the StreamNative Cloud Console to manage MCP availability, access mode, and allowed tools for your organization and clusters. The organization settings control root and organization endpoints. Cluster settings control fixed cluster endpoints and cluster tools added after `sncloud_context_use_cluster` selects a cluster.

## Related StreamNative Cloud documentation

* [Use API Keys to Authenticate to StreamNative Cloud](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview) for API Key v2 and organization-scoped service account credentials.
* [Authentication overview](/cloud/security/authentication/authentication-overview) for StreamNative Cloud authentication concepts, including AuthV2 behavior.
* [RBAC overview](/cloud/security/access/rbac/rbac-overview) and [predefined roles](/cloud/security/access/rbac/manage-rbac-roles) for permissions applied to users and service accounts.
* [StreamNative Cloud objects](/cloud/references/cloud-object) for Kubernetes-style resource manifests.
* [Manage StreamNative clusters](/cloud/clusters/manage-clusters/cluster), [cluster configuration](/cloud/clusters/configure-clusters/cluster-configuration-overview), and [Kafka Cluster Guide](/kafka/kafka-cluster-guide) for cluster concepts and service URLs.
* [PulsarCluster API reference](/api-references/cloudapi/pulsarcluster/cloudstreamnativeio_v1alpha1/get-apiscloudstreamnativeiov1alpha1namespaces-pulsarclusters), [PulsarInstance API reference](/api-references/cloudapi/pulsarinstance/cloudstreamnativeio_v1alpha1/get-apiscloudstreamnativeiov1alpha1namespaces-pulsarinstances), and [PulsarGateway API reference](/api-references/cloudapi/pulsargateway/cloudstreamnativeio_v1alpha1/get-apiscloudstreamnativeiov1alpha1namespaces-pulsargateways) for cluster-domain resources. For `KafkaCluster`, use [Kafka Cluster Guide](/kafka/kafka-cluster-guide) for user-facing concepts and `sncloud_resource_schema` for the live manifest schema when available.
* [ServiceAccount API reference](/api-references/cloudapi/serviceaccount/cloudstreamnativeio_v1alpha1/get-apiscloudstreamnativeiov1alpha1namespaces-serviceaccounts) and [API key API reference](/api-references/cloudapi/apikey/cloudstreamnativeio_v1alpha1/get-apiscloudstreamnativeiov1alpha1namespaces-apikeys) for identity-domain resources.


# Remote StreamNative MCP Server
Source: https://docs.streamnative.io/agent-engine/sn-remote-mcp/remote-mcp-overview

Understand the managed StreamNative MCP Server and how it extends Orca Engine tooling.

The Remote StreamNative MCP Server is the managed deployment option for the [StreamNative MCP Server](/agent-engine/mcp/mcp-overview). It provides a globally available, organization-aware Model Context Protocol (MCP) service at `https://mcp.streamnative.cloud` so you can consume managed tools without running infrastructure. You can connect through a root endpoint, an organization endpoint, or a fixed cluster endpoint. It is built on the open-source [StreamNative MCP Server](https://github.com/streamnative/streamnative-mcp-server).

The remote server uses MCP over HTTP. Configure clients with the full MCP path, such as `https://mcp.streamnative.cloud/mcp/x/<organization>/<instance>/<cluster>`, not only the host name. Root and organization endpoints start with StreamNative Cloud organization tools. Fixed cluster endpoints start with tools for one Pulsar or Kafka cluster.

<Tip>
  If you prefer to run the MCP server on your own machine - or need to connect
  to external Kafka or Pulsar clusters outside StreamNative Cloud - see the
  [Local MCP Server](/agent-engine/mcp/local-mcp-server) instead.
</Tip>

## Instant access from your IDE

* Launch tooling with a single click in supported editors such as VS Code - no local binaries or runtimes required.
* Construct an organization or cluster endpoint and paste it into any MCP-compatible client to reuse the same catalog across teams and environments.

## Why use the remote server

* Skip local installation and upgrades - the managed endpoint stays current with the latest StreamNative MCP capabilities.
* Connect from IDEs such as VS Code with a single click or point any MCP-compatible runtime to your cluster endpoint.
* Authenticate with the same StreamNative Cloud credentials you already use for agent deployments.

## StreamNative Cloud integration

The managed server authenticates against StreamNative Cloud and understands your organization context. After you sign in, you can:

* Browse MCP-enabled clusters that belong to your organization.
* Select a Pulsar or Kafka cluster during an organization-level session and add static cluster tools to the current session.
* Inspect and manage organization-level StreamNative Cloud resources such as service accounts, role bindings, instances, Pulsar clusters, and Kafka clusters.
* Read schemas for supported StreamNative Cloud resources before composing JSON manifests.
* Validate organization-level changes with dry-run apply operations before making changes.
* Inspect the status of cluster resources through natural language prompts.
* Interact with StreamNative services using your existing service accounts and permission model.

## Platform support

The Remote MCP Server supports all StreamNative Cloud cluster types, including Serverless, Dedicated, BYOC, and BYOC Pro. For details on each type, see [Cluster types](/cloud/clusters/cluster-types).

The Remote StreamNative MCP Server exposes tools for both Apache Kafka and Apache Pulsar workloads.

* **Apache Kafka** - manage topics, partitions, consumer groups, schema registry artifacts, and run client operations such as producing and consuming messages. The `kafka-admin-kafka-connect` feature ID is accepted, but Kafka Connect tools are not exposed on Remote MCP Kafka cluster sessions in the current preview.
* **Apache Pulsar** - administer tenants, namespaces, topics, and schemas; run client operations; and manage Functions, Sources, and Sinks in read-only mode.
* **Orca integrations** - surface Pulsar Functions and Orca agents as MCP tools so downstream agents can invoke them directly.

## Always current and managed

* StreamNative operates the service so you automatically receive server updates, new tools, and security patches.
* The remote environment mirrors the latest StreamNative Cloud features, keeping documentation and runtime behavior aligned.
* Preview status means feedback can shape the roadmap - share requests through support channels to influence future releases.

## Security and authorization

* OAuth 2.1 authorization flows align with the evolving MCP authorization specification, allowing interactive sign-in from IDEs and browsers.
* StreamNative Cloud [API Key v2](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#api-key-v1-vs-api-key-v2) offers organization-scoped, non-interactive access for automated clients. Send it as `Authorization: Bearer <api-key>`.
* The server is multi-tenant aware and scopes tool discovery to the resources your identity can access through [StreamNative Cloud RBAC](/cloud/security/access/rbac/rbac-overview).

## Endpoint modes

| Mode          | Endpoint                                                                   | Behavior                                                                                                                                                                                                                                                                     |
| ------------- | -------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Root          | `https://mcp.streamnative.cloud/mcp`                                       | Resolves the organization from the AuthV2 organization audience in your token. Use `X-Organization` for root OAuth discovery, or omit it during interactive OAuth and select an organization during sign-in. Bearer-token requests must resolve to exactly one organization. |
| Organization  | `https://mcp.streamnative.cloud/mcp/x/<organization>`                      | Starts with organization-level StreamNative Cloud tools. Use this mode to discover MCP-enabled clusters, switch cluster context, and manage organization resources.                                                                                                          |
| Fixed cluster | `https://mcp.streamnative.cloud/mcp/x/<organization>/<instance>/<cluster>` | Starts directly in one Pulsar or Kafka cluster. Use this mode when you want a stable tool catalog scoped to one cluster and dynamic Pulsar Function or Orca agent tools.                                                                                                     |

For root and organization-level workflows, see [Root and Organization-Level Tools](/agent-engine/sn-remote-mcp/remote-mcp-organization-tools).

## Getting started

1. In the StreamNative Cloud Console, enable MCP for your organization and each cluster that you want to expose. See [Governance & Permissions](/agent-engine/sn-remote-mcp/remote-mcp-governance).
2. Choose an endpoint mode. For example, organization `o-sndev`, instance `instance-1`, and cluster `c-cluster` resolves to `https://mcp.streamnative.cloud/mcp/x/o-sndev/instance-1/c-cluster`.
3. Paste the endpoint URL into your MCP client configuration and complete OAuth 2.1 sign-in, or supply a service account API Key v2 in the `Authorization` header.
4. Optionally set `X-Organization`, `X-MCP-Features`, and `X-MCP-Readonly` headers to bind root sessions, narrow the tool catalog, or request read-only behavior. See [Remote MCP Headers and Feature Selection](/agent-engine/sn-remote-mcp/remote-mcp-features).
5. Use `sncloud_context_available_clusters` and `sncloud_context_use_cluster` when you start from a root or organization endpoint and need cluster tools in the same session.

Once connected, your IDE or agent runtime receives Server-Sent Events (SSE) notifications whenever StreamNative adds or updates tools, keeping the catalog in sync without manual refreshes.


# MCP Tools Reference
Source: https://docs.streamnative.io/agent-engine/sn-remote-mcp/remote-mcp-tools-reference

Reference for available MCP tool groups for StreamNative Cloud, Pulsar, and Kafka.

The StreamNative MCP Server organizes tools into **tool groups**. Each feature ID enables a group of related tools, not necessarily one MCP tool. Pulsar, Kafka, and dynamic feature IDs apply to both the [Remote MCP Server](/agent-engine/sn-remote-mcp/remote-mcp-overview) and the [local MCP Server](/agent-engine/mcp/local-mcp-server), unless a note states otherwise. StreamNative Cloud organization feature IDs apply to Remote MCP organization-level sessions.

Mixed administration groups use separate read and write tools. For example, `kafka-admin` can expose `kafka_admin_topics_read` for inspection and `kafka_admin_topics_write` for mutations. Read tools are marked as read-only and support safe inspection operations. Write tools are marked as mutating or destructive and can create, update, delete, apply, trigger, produce, or otherwise change resources. In read-only mode, the server omits write or destructive tools where possible and also blocks mutation calls server-side.

## StreamNative Cloud tool groups

Organization-level sessions expose StreamNative Cloud tools for cluster discovery and control-plane resources. For workflow details, see [Organization-Level Tools](/agent-engine/sn-remote-mcp/remote-mcp-organization-tools).

| Feature ID                               | Tools                                                                                                    | Description                                                                                                                                                      |
| ---------------------------------------- | -------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `streamnative-cloud` or `cloud`          | All `sncloud_*` tools                                                                                    | Enable all StreamNative Cloud organization tool families.                                                                                                        |
| `sncloud_context` or `sncloud-context`   | `sncloud_context_available_clusters`, `sncloud_context_use_cluster`, `sncloud_context_reset`             | List MCP-enabled clusters, select a cluster, or reset cluster context. The global identity tool `sncloud_context_whoami` is always available.                    |
| `sncloud_byoc` or `sncloud-byoc`         | `sncloud_resource_catalog`, `sncloud_resource_schema`, `sncloud_byoc_read`, `sncloud_byoc_write`         | Read or manage BYOC resources such as `CloudConnection`, `CloudEnvironment`, `Volume`, and `PoolMember`.                                                         |
| `sncloud_identity` or `sncloud-identity` | `sncloud_resource_catalog`, `sncloud_resource_schema`, `sncloud_identity_read`, `sncloud_identity_write` | Read or manage identity and RBAC resources such as `User`, `ServiceAccount`, `ServiceAccountBinding`, `Role`, `RoleBinding`, `OIDCProvider`, and `IdentityPool`. |
| `sncloud_clusters` or `sncloud-clusters` | `sncloud_resource_catalog`, `sncloud_resource_schema`, `sncloud_clusters_read`, `sncloud_clusters_write` | Read or manage cluster resources such as `Instance`, `PulsarInstance`, `PulsarCluster`, `PulsarGateway`, and `KafkaCluster`.                                     |

Read tools support `list` and `get`. Write tools support `apply` and `delete`. Write tools accept manifests as JSON strings and support `dry_run=true` for validation. Use `sncloud_resource_catalog` to discover supported kinds, then use `sncloud_resource_schema` to get summary, example, runtime JSON schema, or selected schema paths before writing manifests.

After `sncloud_context_use_cluster` selects a cluster in a root or organization-level session, `sncloud_logs` is also available for logs from Functions, Sources, Sinks, and Kafka Connect connectors in the selected cluster. `sncloud_logs` is cluster-scoped and appears only after cluster selection.

## Pulsar tool groups

Pulsar clusters support the following tool groups.

| Feature ID                         | Alias              | MCP tools                                                                                                                                                                      | Description                                                                                              |
| ---------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------- |
| `pulsar-admin`                     |                    | All Pulsar administration tools                                                                                                                                                | All Pulsar administration tool groups.                                                                   |
| `pulsar-admin-topics`              | `topics`           | `pulsar_admin_topic_read`, `pulsar_admin_topic_write`                                                                                                                          | List, create, delete, and inspect Pulsar topics.                                                         |
| `pulsar-admin-namespaces`          | `namespaces`       | `pulsar_admin_namespace_read`, `pulsar_admin_namespace_write`                                                                                                                  | List and manage Pulsar namespaces.                                                                       |
| `pulsar-admin-namespace-policy`    | `namespace-policy` | `pulsar_admin_namespace_policy_get`, `pulsar_admin_namespace_policy_set`, `pulsar_admin_namespace_policy_remove`, `pulsar_admin_namespace_policy_get_anti_affinity_namespaces` | Get and set namespace-level policies such as retention, TTL, backlog, and anti-affinity namespace lists. |
| `pulsar-admin-tenants`             | `tenants`          | `pulsar_admin_tenant_read`, `pulsar_admin_tenant_write`                                                                                                                        | List and manage Pulsar tenants.                                                                          |
| `pulsar-admin-schemas`             | `schemas`          | `pulsar_admin_schema_read`, `pulsar_admin_schema_write`                                                                                                                        | View and manage topic schemas.                                                                           |
| `pulsar-admin-subscriptions`       | `subscriptions`    | `pulsar_admin_subscription_read`, `pulsar_admin_subscription_write`                                                                                                            | List and manage topic subscriptions.                                                                     |
| `pulsar-admin-brokers`             | `brokers`          | `pulsar_admin_brokers_read`, `pulsar_admin_brokers_write`                                                                                                                      | List and inspect broker nodes.                                                                           |
| `pulsar-admin-brokers-status`      | `brokers-status`   | `pulsar_admin_status`                                                                                                                                                          | Retrieve broker status.                                                                                  |
| `pulsar-admin-broker-stats`        | `broker-stats`     | `pulsar_admin_broker_stats`                                                                                                                                                    | Retrieve broker-level statistics and metrics.                                                            |
| `pulsar-admin-clusters`            | `clusters`         | `pulsar_admin_cluster_read`, `pulsar_admin_cluster_write`                                                                                                                      | List and inspect Pulsar clusters.                                                                        |
| `pulsar-admin-functions`           | `functions`        | `pulsar_admin_functions_read`, `pulsar_admin_functions_write`                                                                                                                  | List, inspect, and manage Pulsar Functions.                                                              |
| `pulsar-admin-functions-worker`    | `functions-worker` | `pulsar_admin_functions_worker`                                                                                                                                                | Inspect Functions worker status.                                                                         |
| `pulsar-admin-ns-isolation-policy` | `ns-isolation`     | `pulsar_admin_nsisolationpolicy_read`, `pulsar_admin_nsisolationpolicy_write`                                                                                                  | Manage namespace isolation policies.                                                                     |
| `pulsar-admin-packages`            | `packages`         | `pulsar_admin_package_read`, `pulsar_admin_package_write`                                                                                                                      | Manage Pulsar packages.                                                                                  |
| `pulsar-admin-resource-quotas`     | `resource-quotas`  | `pulsar_admin_resourcequota_read`, `pulsar_admin_resourcequota_write`                                                                                                          | View and manage resource quotas.                                                                         |
| `pulsar-admin-sinks`               | `sinks`            | `pulsar_admin_sinks_read`, `pulsar_admin_sinks_write`                                                                                                                          | List and manage Pulsar IO sinks.                                                                         |
| `pulsar-admin-sources`             | `sources`          | `pulsar_admin_sources_read`, `pulsar_admin_sources_write`                                                                                                                      | List and manage Pulsar IO sources.                                                                       |
| `pulsar-admin-topic-policy`        | `topic-policy`     | `pulsar_admin_topic_policy_read`, `pulsar_admin_topic_policy_write`                                                                                                            | Get and set topic-level policies.                                                                        |
| `pulsar-client`                    |                    | `pulsar_client_consume`, `pulsar_client_produce`                                                                                                                               | Consume and produce messages on Pulsar topics.                                                           |

## Kafka tool groups

Kafka clusters support the following tool groups.

| Feature ID                    | MCP tools                                                                                                                                    | Description                                                                                                                                            |
| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `kafka-admin`                 | `kafka_admin_topics_read`, `kafka_admin_topics_write`, `kafka_admin_groups_read`, `kafka_admin_groups_write`, `kafka_admin_partitions_write` | Manage Kafka topics, partitions, and consumer groups.                                                                                                  |
| `kafka-admin-schema-registry` | `kafka_admin_sr_read`, `kafka_admin_sr_write`                                                                                                | Manage Schema Registry subjects and schemas.                                                                                                           |
| `kafka-client`                | `kafka_client_consume`, `kafka_client_produce`                                                                                               | Consume and produce messages on Kafka topics.                                                                                                          |
| `kafka-admin-kafka-connect`   | `kafka_admin_connect_read`, `kafka_admin_connect_write`                                                                                      | Kafka Connect tools. This feature ID is accepted, but Kafka Connect tools are not exposed on Remote MCP Kafka cluster sessions in the current preview. |

<Note>
  Kafka topic, partition, and consumer-group tools are enabled by `kafka-admin`.
  There are no separate Remote MCP feature IDs named `kafka-admin-topics`,
  `kafka-admin-partitions`, or `kafka-admin-groups`.
</Note>

## Combination shortcuts

Use these shortcut IDs to enable multiple tool groups at once.

| Shortcut             | Includes                                                                                |
| -------------------- | --------------------------------------------------------------------------------------- |
| `all`                | Every available tool group across Pulsar, Kafka, StreamNative Cloud, and dynamic tools. |
| `all-pulsar`         | Pulsar tooling features.                                                                |
| `pulsar`             | Alias for `all-pulsar`.                                                                 |
| `all-kafka`          | Kafka tooling features.                                                                 |
| `kafka`              | Alias for `all-kafka`.                                                                  |
| `pulsar-admin`       | All `pulsar-admin-*` tool groups.                                                       |
| `kafka-admin`        | Kafka topic, partition, consumer group, and compatible admin tools.                     |
| `cloud`              | Alias for `streamnative-cloud`.                                                         |
| `streamnative-cloud` | StreamNative Cloud organization tool families.                                          |

## Dynamic tools

In addition to static tool groups, the Remote MCP Server supports dynamic tools that expose your own workloads as MCP tools.

| Feature ID           | Description                                                                                                                                                                                                                                                                            |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `functions-as-tools` | Expose Pulsar Functions as callable MCP tools on fixed Pulsar cluster sessions. See [Functions as Tools](/agent-engine/sn-remote-mcp/remote-mcp-functions-as-tools).                                                                                                                   |
| `agents-as-tools`    | Accepted dynamic agent-tool feature ID. In the current preview, include `functions-as-tools` or `pulsar-admin-functions` when you want to expose eligible Orca agents on fixed Pulsar cluster sessions. See [Agents as Tools](/agent-engine/sn-remote-mcp/remote-mcp-agents-as-tools). |

<Note>
  Dynamic tools are available on fixed cluster endpoints. They are not added
  after selecting a cluster inside an organization-level session with
  `sncloud_context_use_cluster`.
</Note>

## Client tool parameter notes

* Kafka consume tools use `group` for the Kafka consumer group ID. When `group` is omitted, the server uses an ephemeral group and cannot read committed offsets.
* Pulsar consume tools use `subscription-name` for the subscription. This is the Pulsar equivalent of a durable cursor; do not pass a Kafka `group` parameter to Pulsar tools.
* StreamNative Cloud domain read tools use `operation=list|get`, `resource`, and optional `name`. `name` is required for `get`.
* StreamNative Cloud domain write tools use `operation=apply|delete`, `resource`, and either `manifest` for `apply` or `name` for `delete`.
* StreamNative Cloud write tools use `manifest` as a JSON string. Do not pass YAML or an object. Use `dry_run=true` for `apply` before applying changes.
* `sncloud_logs` requires a selected cluster in a root or organization-level session. Use it for logs from Functions, Sources, Sinks, and Kafka Connect connectors.
* StreamNative Cloud read tools omit `status` and `metadata.managedFields` unless you set `includeStatus=true` or `includeManagedFields=true`.
* `sncloud_resource_schema` defaults to `format=summary`. Use `format=example` for an example manifest, `format=jsonschema` for runtime Kubernetes schema when available, and `paths` such as `spec.broker` or `spec.clusterRefs` to reduce schema size.

## Usage tips

* **Start minimal.** Enable only the tool groups you need. Fewer tools reduce token usage and help AI agents select the right tool more reliably.
* **Use shortcuts for broad access.** If you need all Pulsar tools, use `all-pulsar` instead of listing each group individually.
* **Match groups to your cluster type.** Pulsar tool groups only work on Pulsar clusters, and Kafka tool groups only work on Kafka clusters. Enabling mismatched groups has no effect.
* **Combine groups freely.** You can mix StreamNative Cloud, Pulsar, Kafka, and dynamic tool groups. For example, `sncloud_context,pulsar-admin-topics,pulsar-client` enables cluster discovery, Pulsar topic administration, and Pulsar message operations.
* **Use StreamNative Cloud schema helpers before writes.** Call `sncloud_resource_catalog`, then `sncloud_resource_schema`, then a read tool, then a write tool with `dry_run=true` before applying changes.


# Build Consumer
Source: https://docs.streamnative.io/clients/kafka-clients/go/tutorial/kafka-go-build-consumer



Next, create the consumer application by pasting the following Go code into a file named `consumer.go`.

```go theme={null}
package main

import (
    "fmt"
    "os"
    "os/signal"
    "syscall"
    "time"

    "github.com/confluentinc/confluent-kafka-go/kafka"
)

func main() {

    c, err := kafka.NewConsumer(&kafka.ConfigMap{
        // User-specific properties that you must set
        "bootstrap.servers": "<BOOTSTRAP SERVERS>",
        "sasl.username":     "unused",
        "sasl.password":     "token:<API KEY>",

        // Fixed properties
        "security.protocol": "SASL_SSL",
        "sasl.mechanisms":   "PLAIN",
        "group.id":          "kafka-go-getting-started",
        "auto.offset.reset": "earliest"})

    if err != nil {
        fmt.Printf("Failed to create consumer: %s", err)
        os.Exit(1)
    }

    topic := "purchases"
    err = c.SubscribeTopics([]string{topic}, nil)
    // Set up a channel for handling Ctrl-C, etc
    sigchan := make(chan os.Signal, 1)
    signal.Notify(sigchan, syscall.SIGINT, syscall.SIGTERM)

    // Process messages
    run := true
    for run {
        select {
        case sig := <-sigchan:
            fmt.Printf("Caught signal %v: terminating\n", sig)
            run = false
        default:
            ev, err := c.ReadMessage(100 * time.Millisecond)
            if err != nil {
                // Errors are informational and automatically handled by the consumer
                continue
            }
            fmt.Printf("Consumed event from topic %s: key = %-10s value = %s\n",
                *ev.TopicPartition.Topic, string(ev.Key), string(ev.Value))
        }
    }

    c.Close()

}
```

Fill in the appropriate `<BOOTSTRAP SERVERS>` endpoint and `<API KEY>` in the `bootstrap.servers` and `sasl.password` properties where the consumer is instantiated via the `kafka.NewConsumer` method. Compile the consumer as follows:

```bash theme={null}
go build -o out/consumer consumer.go
```


# Build Producer
Source: https://docs.streamnative.io/clients/kafka-clients/go/tutorial/kafka-go-build-producer



Let's create the producer application by pasting the following Go code into a file named `producer.go`.

```go theme={null}
package main

import (
    "fmt"
    "math/rand"
    "os"

    "github.com/confluentinc/confluent-kafka-go/kafka"
)

func main() {

    p, err := kafka.NewProducer(&kafka.ConfigMap{
        // User-specific properties that you must set
        "bootstrap.servers": "<BOOTSTRAP SERVERS>",
        "sasl.username":     "unused",
        "sasl.password":     "token:<API KEY>",

        // Fixed properties
        "security.protocol": "SASL_SSL",
        "sasl.mechanisms":   "PLAIN",
        "acks":              "all"})

    if err != nil {
        fmt.Printf("Failed to create producer: %s", err)
        os.Exit(1)
    }

    // Go-routine to handle message delivery reports and
    // possibly other event types (errors, stats, etc)
    go func() {
        for e := range p.Events() {
            switch ev := e.(type) {
            case *kafka.Message:
                if ev.TopicPartition.Error != nil {
                    fmt.Printf("Failed to deliver message: %v\n", ev.TopicPartition)
                } else {
                    fmt.Printf("Produced event to topic %s: key = %-10s value = %s\n",
                        *ev.TopicPartition.Topic, string(ev.Key), string(ev.Value))
                }
            }
        }
    }()

    users := [...]string{"eabara", "jsmith", "sgarcia", "jbernard", "htanaka", "awalther"}
    items := [...]string{"book", "alarm clock", "t-shirts", "gift card", "batteries"}
    topic := "purchases"

    for n := 0; n < 10; n++ {
        key := users[rand.Intn(len(users))]
        data := items[rand.Intn(len(items))]
        p.Produce(&kafka.Message{
            TopicPartition: kafka.TopicPartition{Topic: &topic, Partition: kafka.PartitionAny},
            Key:            []byte(key),
            Value:          []byte(data),
        }, nil)
    }

    // Wait for all messages to be delivered
    p.Flush(15 * 1000)
    p.Close()
}
```

Fill in the appropriate `<BOOTSTRAP SERVERS>` endpoint and `<API KEY>` in the `bootstrap.servers` and `sasl.password` properties where the producer is instantiated using the `kafka.NewProducer` method.

Compile the producer with the following:

```bash theme={null}
go build -o out/producer producer.go
```

If you get any errors during the build make sure that you initialized the module correctly per the instructions in the [previous step](/clients/kafka-clients/go/tutorial/kafka-go-create-project).


# Cluster Setup
Source: https://docs.streamnative.io/clients/kafka-clients/go/tutorial/kafka-go-cluster-setup



We are going to need a StreamNative Cloud cluster for our client application to operate with. This guide can help you create your first StreamNative Cloud cluster.

## Create a StreamNative Cloud cluster

From within the StreamNative Cloud console, creating a new cluster is just a few clicks.

<Tabs>
  <Tab title="Cloud Console">
    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Organizations** to list your created organizations.

    2. Click the name of your first organization. In the figure below, the organization name is `Demos`.

       <img alt="screenshot of organization section" />

    3. On the **Select an Instance** card of the **Organization Dashboard** page, click the `+` button.

    4. On the **Choose the deployment type for your instance**, click **Deploy Serverless** to start the instance creation process.

    5. On the **Instance Configuration** page, enter a name for your instance, and select a Cloud Provider. The instance name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

    6. Click **Cluster Location** to start the cluster creation process.

    7. On the **Cluster Location** page, enter a name for your cluster, select the cluster location, and then click **Finish**. The cluster name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

    The cluster page appears, displaying the cluster creation process. Depending on the chosen cloud provider and other settings, it may take a few seconds to several minutes to provision the cluster. Once the cluster has been provisioned, the **Cluster Dashboard** page will be displayed.

    Now you can get started configuring apps and data on your new cluster.
  </Tab>
</Tabs>

## Get the cluster service URLs

Next, you'll need to get your StreamNative Cloud cluster's Service URLs to configure your client applications. These URLs allow your applications to connect to and interact with your cluster.

Your cluster provides different Service URLs depending on which protocol you plan to use:

For Pulsar Protocol:

* **Broker Service URL**: Used to configure Pulsar clients to connect to the brokers for producing and consuming messages
* **HTTP Service URL**: Used for administrative operations via the Pulsar admin API

For Kafka Protocol:

* **Kafka Service URL**: Used as bootstrap servers in Kafka client configurations
* **Schema Registry URL**: Used to configure Kafka serializers and deserializers that work with schemas

You'll need these URLs in the upcoming steps when setting up your producer and consumer clients.

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

## Create a service account and API key

Next, to connect your client application to your cluster to produce and consume messages, you need to create a [Service Account](/cloud/security/authentication/service-accounts/manage-service-accounts) and choose an authentication mechanism: either [API Key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview) or [OAuth](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview).

API Key authentication is quicker to implement since you only need to create an API key in StreamNative Cloud. This guide uses API Key authentication.

<Note title="Note">
  Currently, you can't edit a service account. If you need a service account to have Super Admin access, make sure to enable it when creating the service account. By default, service accounts do not have Super Admin enabled.
</Note>

<Tabs>
  <Tab title="Cloud Console">
    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Accounts & Accesses**.

    2. On the left navigation pane, click **Service Accounts**.

    3. On the **Service Account** page, click **+ New Service Account**.

    4. On the **Create Service Account** dialog, enter a name for the service account, and then click **Confirm**.

    5. On the **Service Account** page, in the row of the service account you just created, click the **...** icon, and select **Create API Key** in the dropdown menu.

    6. On the **New API Key** dialog:

       * Enter a name for the API key
       * Set the expiration date
       * Select the instance you created in Step 2
       * Write a description for the API key
       * Click **Confirm**

    <Note title="Note">
      An API key and associated secret apply to the active StreamNative instance. If you add a new instance, you must create a new API key for producers and consumers on the new Pulsar instance. For more information, see [Use API Keys to Authenticate to StreamNative Cloud](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview).
    </Note>

    7. After the API key is created, you can see the API key shown in the **New API Key** dialog. Click the **Copy and close** button to copy the API key to your clipboard. **Please note that you cannot retrieve the API key later after closing the dialog.**
  </Tab>
</Tabs>

Note the API key as we will use them when configuring the producer and consumer clients in upcoming steps.


# Consume Messages
Source: https://docs.streamnative.io/clients/kafka-clients/go/tutorial/kafka-go-consume-messages



From another terminal, run the following command to run the consumer application which will read the events from the `purchases` topic and write the information to the terminal.

```bash theme={null}
./out/consumer
```

The consumer application will start and print any events it has not yet consumed and then wait for more events to arrive. On startup of the consumer, you should see output resembling this:

```bash theme={null}
Consumed event from topic purchases: key = awalther   value = batteries
Consumed event from topic purchases: key = htanaka    value = alarm clock
Consumed event from topic purchases: key = awalther   value = t-shirts
Consumed event from topic purchases: key = eabara     value = t-shirts
Consumed event from topic purchases: key = jbernard   value = t-shirts
Consumed event from topic purchases: key = eabara     value = alarm clock
Consumed event from topic purchases: key = jbernard   value = t-shirts
Consumed event from topic purchases: key = sgarcia    value = gift card
Consumed event from topic purchases: key = jsmith     value = t-shirts
Consumed event from topic purchases: key = jsmith     value = alarm clock
```

Rerun the producer to see more events, or feel free to modify the code as necessary to create more or different events.

Once you are done with the consumer, enter `Ctrl-C` to terminate the consumer application.


# Create Topic
Source: https://docs.streamnative.io/clients/kafka-clients/go/tutorial/kafka-go-create-topic



In a StreamNative Cluster, topics are grouped into [tenants](/cloud/manage-data-streams/tenant) and [namespaces](/cloud/manage-data-streams/namespace). In this guide, we will use the existing `public/default` namespace.

## Authorize the Service Account

<Note title="Note">
  If you create a super-user service account, you can skip this step.
</Note>

Before configuring the producer and consumer clients, you need to authorize the service account to grant the necessary permissions for the service account to interact with your StreamNative Cloud cluster.

To grant the service account permissions on the namespace level, follow these steps:

<Tabs>
  <Tab title="Cloud Console">
    1. Navigate to the **Namespace Dashboard** page by [switching to the namespace workspace](/cloud/get-started/cloud-console#switch-a-namespace).

    2. On the **Namespace Dashboard** page, click **Configuration** on the left navigation pane.

    3. On the **Namespace configuration** page, click **ADD ROLE**, and select the service account that you want to authorize.

    4. On the drop-down menu below the service account, select the proper permissions to assign to the service account. There are six permissions in total:

    * `consume`: allow the service account to consume messages.
    * `produce`: allow the service account to publish messages.
    * `functions`: allow the service account to submit and manage functions.
    * `sinks`: allow the service account to create and manage sink connectors.
    * `sources`: allow the service account to create and manage source connectors.
    * `packages`: allow the service account to upload and manage pulsar packages. If you want to submit a customized function/connector, then you will need to upload the function/connector’s JAR/NAR/Python file first, which requires the `packages` permission.

      <img alt="Authorize Service Account" />
  </Tab>
</Tabs>

In this guide, we will authorize the service account with the `produce` and `consume` permissions for the `public/default` namespace.

## Create a Topic

Navigate to the **Namespace Dashboard** page of the `public/default` namespace and follow the instructions below to create a topic, `purchases`, which you will use to produce and consume events.

<Tabs>
  <Tab title="StreamNative Console">
    To create a topic, follow these steps.

    1. Navigate to the desired **Namespace Dashboard** page by [switching to the namespace workspace](/cloud/get-started/cloud-console#switch-a-namespace).

    2. On the left navigation pane, under **Resources**, click **Topics**.

    3. Click **New Topic**.

       <img alt="screenshot of new topic dialog box" />

    4. Configure the topic, as outlined in the following table.

       | Item       | Description                                                                                                                                                                                                                                                                    |
       | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
       | Persistent | Configure the type of the topic. <br />- Persistent: messages in a persistent topic are durably persisted on the storage disk. <br />- Non-persistent: messages in a non-persistent topic are not persisted on the storage disk. <br /> By default, it is set to *Persistent*. |
       | Topic Name | Enter a name for the topic. It is a string of up to 40 characters, supporting lowercase letters (a-z), numeric characters (0-9), and the special character hyphen (-).                                                                                                         |
       | Partitions | (Optional) Configure the number of partitions for a partitioned topic. You can have from 1 up to 100 partitions per topic. It's recommended to have at least 1 partition.                                                                                                      |

    5. Click **Confirm**.
  </Tab>
</Tabs>


# Produce Messages
Source: https://docs.streamnative.io/clients/kafka-clients/go/tutorial/kafka-go-produce-messages



Execute the compiled producer binary in order to produce messages to the `purchases` topic.

```bash theme={null}
./out/producer
```

You should see output resembling this:

```bash theme={null}
Produced event to topic purchases: key = awalther   value = batteries
Produced event to topic purchases: key = htanaka    value = alarm clock
Produced event to topic purchases: key = awalther   value = t-shirts
Produced event to topic purchases: key = eabara     value = t-shirts
Produced event to topic purchases: key = jbernard   value = t-shirts
Produced event to topic purchases: key = eabara     value = alarm clock
Produced event to topic purchases: key = jbernard   value = t-shirts
Produced event to topic purchases: key = sgarcia    value = gift card
Produced event to topic purchases: key = jsmith     value = t-shirts
Produced event to topic purchases: key = jsmith     value = alarm clock
```


# What's Next
Source: https://docs.streamnative.io/clients/kafka-clients/go/tutorial/kafka-go-whats-next



* For the Go client API, checkout the [Go documentation](https://pkg.go.dev/github.com/confluentinc/confluent-kafka-go/kafka)


# Build Consumer
Source: https://docs.streamnative.io/clients/kafka-clients/node.js/tutorial/kafka-js-build-consumer



Next, create the JavaScript consumer application by pasting the following code into a file `consumer.js`.

```javascript theme={null}
const { Kafka } = require('kafkajs')

const kafka = new Kafka({
  // User-specific properties that you must set
  clientId: 'my-app',
  brokers: ['<BOOTSTRAP SERVERS>'],

  // Fixed properties
  ssl: true,
  sasl: {
    mechanism: 'plain',
    username: 'unused',
    password: 'token:<API KEY>',
  },
})

async function receive() {
  let topic = 'purchases'

  const consumer = kafka.consumer({
    groupId: 'kafka-nodejs-getting-started',
  })
  await consumer.connect()
  console.log('Connected to Kafka')

  await consumer.subscribe({ topic: topic, fromBeginning: true })

  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      let k = message.key.toString().padEnd(10, ' ')
      let value = message.value.toString()
      console.log(
        `Consumed event from topic ${topic}: key = ${k} value = ${value}`
      )
    },
  })
}

receive().catch((err) => {
  console.error(`Something went wrong:\n${err}`)
  process.exit(1)
})
```

Fill in the appropriate `<BOOTSTRAP SERVERS>` endpoint and `<API KEY>` in the `brokers` and `sasl.password` properties where the client configuration object is created.


# Build Producer
Source: https://docs.streamnative.io/clients/kafka-clients/node.js/tutorial/kafka-js-build-producer



Let's create the JavaScript producer application by pasting the following code into a file `producer.js`.

```javascript theme={null}
const { Kafka } = require('kafkajs')

const kafka = new Kafka({
  // User-specific properties that you must set
  clientId: 'my-app',
  brokers: ['<BOOTSTRAP SERVERS>'],

  // Fixed properties
  ssl: true,
  sasl: {
    mechanism: 'plain',
    username: 'unused',
    password: 'token:<API KEY>',
  },
})

async function send() {
  let topic = 'purchases'

  let users = ['eabara', 'jsmith', 'sgarcia', 'jbernard', 'htanaka', 'awalther']
  let items = ['book', 'alarm clock', 't-shirts', 'gift card', 'batteries']

  const producer = kafka.producer({})
  await producer.connect()

  let numEvents = 10
  for (let idx = 0; idx < numEvents; ++idx) {
    const key = users[Math.floor(Math.random() * users.length)]
    const value = Buffer.from(items[Math.floor(Math.random() * items.length)])

    let resp = await producer.send({
      topic: topic,
      messages: [{ key: key, value: value }],
    })
    let k = key.toString().padEnd(10, ' ')
    console.log(`Produced event to topic ${topic}: key = ${k} value = ${value}`)
  }

  await producer.disconnect()
}

send().catch((err) => {
  console.error(`Something went wrong:\n${err}`)
  process.exit(1)
})
```

Fill in the appropriate `<BOOTSTRAP SERVERS>` endpoint and `<API KEY>` in the `brokers` and `sasl.password` properties where the client configuration object is created.


# Cluster Setup
Source: https://docs.streamnative.io/clients/kafka-clients/node.js/tutorial/kafka-js-cluster-setup



We are going to need a StreamNative Cloud cluster for our client application to operate with. This guide can help you create your first StreamNative Cloud cluster.

## Create a StreamNative Cloud cluster

From within the StreamNative Cloud console, creating a new cluster is just a few clicks.

<Tabs>
  <Tab title="Cloud Console">
    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Organizations** to list your created organizations.

    2. Click the name of your first organization. In the figure below, the organization name is `Demos`.

       <img alt="screenshot of organization section" />

    3. On the **Select an Instance** card of the **Organization Dashboard** page, click the `+` button.

    4. On the **Choose the deployment type for your instance**, click **Deploy Serverless** to start the instance creation process.

    5. On the **Instance Configuration** page, enter a name for your instance, and select a Cloud Provider. The instance name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

    6. Click **Cluster Location** to start the cluster creation process.

    7. On the **Cluster Location** page, enter a name for your cluster, select the cluster location, and then click **Finish**. The cluster name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

    The cluster page appears, displaying the cluster creation process. Depending on the chosen cloud provider and other settings, it may take a few seconds to several minutes to provision the cluster. Once the cluster has been provisioned, the **Cluster Dashboard** page will be displayed.

    Now you can get started configuring apps and data on your new cluster.
  </Tab>
</Tabs>

## Get the cluster service URLs

Next, you'll need to get your StreamNative Cloud cluster's Service URLs to configure your client applications. These URLs allow your applications to connect to and interact with your cluster.

Your cluster provides different Service URLs depending on which protocol you plan to use:

For Pulsar Protocol:

* **Broker Service URL**: Used to configure Pulsar clients to connect to the brokers for producing and consuming messages
* **HTTP Service URL**: Used for administrative operations via the Pulsar admin API

For Kafka Protocol:

* **Kafka Service URL**: Used as bootstrap servers in Kafka client configurations
* **Schema Registry URL**: Used to configure Kafka serializers and deserializers that work with schemas

You'll need these URLs in the upcoming steps when setting up your producer and consumer clients.

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

## Create a service account and API key

Next, to connect your client application to your cluster to produce and consume messages, you need to create a [Service Account](/cloud/security/authentication/service-accounts/manage-service-accounts) and choose an authentication mechanism: either [API Key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview) or [OAuth](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview).

API Key authentication is quicker to implement since you only need to create an API key in StreamNative Cloud. This guide uses API Key authentication.

<Note title="Note">
  Currently, you can't edit a service account. If you need a service account to have Super Admin access, make sure to enable it when creating the service account. By default, service accounts do not have Super Admin enabled.
</Note>

<Tabs>
  <Tab title="Cloud Console">
    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Accounts & Accesses**.

    2. On the left navigation pane, click **Service Accounts**.

    3. On the **Service Account** page, click **+ New Service Account**.

    4. On the **Create Service Account** dialog, enter a name for the service account, and then click **Confirm**.

    5. On the **Service Account** page, in the row of the service account you just created, click the **...** icon, and select **Create API Key** in the dropdown menu.

    6. On the **New API Key** dialog:

       * Enter a name for the API key
       * Set the expiration date
       * Select the instance you created in Step 2
       * Write a description for the API key
       * Click **Confirm**

    <Note title="Note">
      An API key and associated secret apply to the active StreamNative instance. If you add a new instance, you must create a new API key for producers and consumers on the new Pulsar instance. For more information, see [Use API Keys to Authenticate to StreamNative Cloud](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview).
    </Note>

    7. After the API key is created, you can see the API key shown in the **New API Key** dialog. Click the **Copy and close** button to copy the API key to your clipboard. **Please note that you cannot retrieve the API key later after closing the dialog.**
  </Tab>
</Tabs>

Note the API key as we will use them when configuring the producer and consumer clients in upcoming steps.


# Consume Messages
Source: https://docs.streamnative.io/clients/kafka-clients/node.js/tutorial/kafka-js-consume-messages



From another terminal, run the following command to run the consumer application, which will read the events from the purchases topic and write the information to the terminal.

```bash theme={null}
node consumer.js
```

The consumer application will start and print any events it has not yet consumed and then wait for more events to arrive. On startup of the consumer, you should see output resembling this:

```bash theme={null}
Consumed event from topic purchases: key = jsmith     value = alarm clock
Consumed event from topic purchases: key = htanaka    value = alarm clock
Consumed event from topic purchases: key = sgarcia    value = book
Consumed event from topic purchases: key = jsmith     value = gift card
Consumed event from topic purchases: key = jsmith     value = gift card
Consumed event from topic purchases: key = htanaka    value = batteries
Consumed event from topic purchases: key = awalther   value = alarm clock
Consumed event from topic purchases: key = eabara     value = batteries
Consumed event from topic purchases: key = sgarcia    value = gift card
Consumed event from topic purchases: key = sgarcia    value = t-shirts
```

Rerun the producer to see more events, or feel free to modify the code as necessary to create more or different events.

Once you are done, enter `Ctrl-C` to terminate the consumer application.


# Create Project
Source: https://docs.streamnative.io/clients/kafka-clients/node.js/tutorial/kafka-js-create-project



Create a new directory anywhere you’d like for this project:

```bash theme={null}
mkdir kafka-nodejs-getting-started && cd kafka-nodejs-getting-started
```

Then install the Apache Kafka JS client library:

```bash theme={null}
npm install kafkajs
```


# Create Topic
Source: https://docs.streamnative.io/clients/kafka-clients/node.js/tutorial/kafka-js-create-topic



In a StreamNative Cluster, topics are grouped into [tenants](/cloud/manage-data-streams/tenant) and [namespaces](/cloud/manage-data-streams/namespace). In this guide, we will use the existing `public/default` namespace.

## Authorize the Service Account

<Note title="Note">
  If you create a super-user service account, you can skip this step.
</Note>

Before configuring the producer and consumer clients, you need to authorize the service account to grant the necessary permissions for the service account to interact with your StreamNative Cloud cluster.

To grant the service account permissions on the namespace level, follow these steps:

<Tabs>
  <Tab title="Cloud Console">
    1. Navigate to the **Namespace Dashboard** page by [switching to the namespace workspace](/cloud/get-started/cloud-console#switch-a-namespace).

    2. On the **Namespace Dashboard** page, click **Configuration** on the left navigation pane.

    3. On the **Namespace configuration** page, click **ADD ROLE**, and select the service account that you want to authorize.

    4. On the drop-down menu below the service account, select the proper permissions to assign to the service account. There are six permissions in total:

    * `consume`: allow the service account to consume messages.
    * `produce`: allow the service account to publish messages.
    * `functions`: allow the service account to submit and manage functions.
    * `sinks`: allow the service account to create and manage sink connectors.
    * `sources`: allow the service account to create and manage source connectors.
    * `packages`: allow the service account to upload and manage pulsar packages. If you want to submit a customized function/connector, then you will need to upload the function/connector’s JAR/NAR/Python file first, which requires the `packages` permission.

      <img alt="Authorize Service Account" />
  </Tab>
</Tabs>

In this guide, we will authorize the service account with the `produce` and `consume` permissions for the `public/default` namespace.

## Create a Topic

Navigate to the **Namespace Dashboard** page of the `public/default` namespace and follow the instructions below to create a topic, `purchases`, which you will use to produce and consume events.

<Tabs>
  <Tab title="StreamNative Console">
    To create a topic, follow these steps.

    1. Navigate to the desired **Namespace Dashboard** page by [switching to the namespace workspace](/cloud/get-started/cloud-console#switch-a-namespace).

    2. On the left navigation pane, under **Resources**, click **Topics**.

    3. Click **New Topic**.

       <img alt="screenshot of new topic dialog box" />

    4. Configure the topic, as outlined in the following table.

       | Item       | Description                                                                                                                                                                                                                                                                    |
       | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
       | Persistent | Configure the type of the topic. <br />- Persistent: messages in a persistent topic are durably persisted on the storage disk. <br />- Non-persistent: messages in a non-persistent topic are not persisted on the storage disk. <br /> By default, it is set to *Persistent*. |
       | Topic Name | Enter a name for the topic. It is a string of up to 40 characters, supporting lowercase letters (a-z), numeric characters (0-9), and the special character hyphen (-).                                                                                                         |
       | Partitions | (Optional) Configure the number of partitions for a partitioned topic. You can have from 1 up to 100 partitions per topic. It's recommended to have at least 1 partition.                                                                                                      |

    5. Click **Confirm**.
  </Tab>
</Tabs>


# Introduction
Source: https://docs.streamnative.io/clients/kafka-clients/node.js/tutorial/kafka-js-introduction



<Note title="Note">
  This tutorial uses the [KafkaJS](https://kafka.js.org/) library directly.
</Note>

In this tutorial, you will build Node.js client applications which produce and consume messages from an Apache Kafka® cluster.

As you learn how to run your first Kafka application, we recommend using [StreamNative Cloud](https://streamnative.io/deployment) so you don't have to run your own Kafka cluster and can focus on client development.

If you do not already have an account, you can follow the steps below to sign up for StreamNative Cloud. New signups receive \$200 to spend within StreamNative Cloud. No credit card is required for the first 30 days or until your credits run out.

## Sign up for StreamNative Cloud

<Note title="Note">
  If you have an email account configured for using Single Sign-On (SSO) with StreamNative Cloud, use that email address and password when signing up.
</Note>

To sign up, navigate to the [StreamNative Cloud Console login page](https://console.streamnative.cloud/?defaultMethod=signup). Follow the prompts to create an account.

After you click **Finish**, you might have to wait briefly for your first organization to be created.


# Prerequisites
Source: https://docs.streamnative.io/clients/kafka-clients/node.js/tutorial/kafka-js-prerequisites



Using Windows? You'll need to download [Windows Subsystem for Linux](https://learn.microsoft.com/en-us/windows/wsl/install).

This guide assumes that you have [Node.js](https://nodejs.org/en/download/) version 16 or later installed.


# Produce Messages
Source: https://docs.streamnative.io/clients/kafka-clients/node.js/tutorial/kafka-js-produce-messages



Run the producer by executing the following command:

```bash theme={null}
node producer.js
```

You should see output resembling this:

```bash theme={null}
Produced event to topic purchases: key = jsmith     value = alarm clock
Produced event to topic purchases: key = htanaka    value = alarm clock
Produced event to topic purchases: key = sgarcia    value = book
Produced event to topic purchases: key = htanaka    value = batteries
Produced event to topic purchases: key = sgarcia    value = gift card
Produced event to topic purchases: key = jsmith     value = gift card
Produced event to topic purchases: key = sgarcia    value = t-shirts
Produced event to topic purchases: key = awalther   value = alarm clock
Produced event to topic purchases: key = eabara     value = batteries
Produced event to topic purchases: key = jsmith     value = gift card
```


# What's Next
Source: https://docs.streamnative.io/clients/kafka-clients/node.js/tutorial/kafka-js-whats-next



* For the Kafka JS client API, checkout the [KafkaJS documentation](https://kafka.js.org/docs/introduction)


# Build Consumer
Source: https://docs.streamnative.io/clients/kafka-clients/python/tutorial/kafka-python-build-consumer



Next, create the Python consumer application by pasting the following code into a file `consumer.py`.

```python theme={null}
#!/usr/bin/env python

from confluent_kafka import Consumer

if __name__ == '__main__':

    config = {
        # User-specific properties that you must set
        'bootstrap.servers': '<BOOTSTRAP SERVERS>',
        'sasl.username':     'unused',
        'sasl.password':     'token:<API KEY>',

        # Fixed properties
        'security.protocol': 'SASL_SSL',
        'sasl.mechanisms':   'PLAIN',
        'group.id':          'kafka-python-getting-started',
        'auto.offset.reset': 'earliest'
    }

    # Create Consumer instance
    consumer = Consumer(config)

    # Subscribe to topic
    topic = "purchases"
    consumer.subscribe([topic])

    # Poll for new messages from Kafka and print them.
    try:
        while True:
            msg = consumer.poll(1.0)
            if msg is None:
                # Initial message consumption may take up to
                # `session.timeout.ms` for the consumer group to
                # rebalance and start consuming
                print("Waiting...")
            elif msg.error():
                print("ERROR: %s".format(msg.error()))
            else:
                # Extract the (optional) key and value, and print.
                print("Consumed event from topic {topic}: key = {key:12} value = {value:12}".format(
                    topic=msg.topic(), key=msg.key().decode('utf-8'), value=msg.value().decode('utf-8')))
    except KeyboardInterrupt:
        pass
    finally:
        # Leave group and commit final offsets
        consumer.close()
```

Fill in the appropriate `<BOOTSTRAP SERVERS>` endpoint and `<API KEY>` in the `bootstrap.servers` and `sasl.password` properties where the client configuration `config` object is created.


# Build Producer
Source: https://docs.streamnative.io/clients/kafka-clients/python/tutorial/kafka-python-build-producer



Let's create the Python producer application by pasting the following code into a file `producer.py`.

```python theme={null}
#!/usr/bin/env python

from random import choice
from confluent_kafka import Producer

if __name__ == '__main__':

    config = {
        # User-specific properties that you must set
        'bootstrap.servers': '<BOOTSTRAP SERVERS>',
        'sasl.username':     'unused',
        'sasl.password':     'token:<API KEY>',

        # Fixed properties
        'security.protocol': 'SASL_SSL',
        'sasl.mechanisms':   'PLAIN',
        'acks':              'all'
    }

    # Create Producer instance
    producer = Producer(config)

    # Optional per-message delivery callback (triggered by poll() or flush())
    # when a message has been successfully delivered or permanently
    # failed delivery (after retries).
    def delivery_callback(err, msg):
        if err:
            print('ERROR: Message failed delivery: {}'.format(err))
        else:
            print("Produced event to topic {topic}: key = {key:12} value = {value:12}".format(
                topic=msg.topic(), key=msg.key().decode('utf-8'), value=msg.value().decode('utf-8')))

    # Produce data by selecting random values from these lists.
    topic = "purchases"
    user_ids = ['eabara', 'jsmith', 'sgarcia', 'jbernard', 'htanaka', 'awalther']
    products = ['book', 'alarm clock', 't-shirts', 'gift card', 'batteries']

    count = 0
    for _ in range(10):
        user_id = choice(user_ids)
        product = choice(products)
        producer.produce(topic, product, user_id, callback=delivery_callback)
        count += 1

    # Block until the messages are sent.
    producer.poll(10000)
    producer.flush()
```

Fill in the appropriate `<BOOTSTRAP SERVERS>` endpoint and `<API KEY>` in the `bootstrap.servers` and `sasl.password` properties where the client configuration `config` object is created.


# Cluster Setup
Source: https://docs.streamnative.io/clients/kafka-clients/python/tutorial/kafka-python-cluster-setup



We are going to need a StreamNative Cloud cluster for our client application to operate with. This guide can help you create your first StreamNative Cloud cluster.

## Create a StreamNative Cloud cluster

From within the StreamNative Cloud console, creating a new cluster is just a few clicks.

<Tabs>
  <Tab title="Cloud Console">
    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Organizations** to list your created organizations.

    2. Click the name of your first organization. In the figure below, the organization name is `Demos`.

       <img alt="screenshot of organization section" />

    3. On the **Select an Instance** card of the **Organization Dashboard** page, click the `+` button.

    4. On the **Choose the deployment type for your instance**, click **Deploy Serverless** to start the instance creation process.

    5. On the **Instance Configuration** page, enter a name for your instance, and select a Cloud Provider. The instance name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

    6. Click **Cluster Location** to start the cluster creation process.

    7. On the **Cluster Location** page, enter a name for your cluster, select the cluster location, and then click **Finish**. The cluster name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

    The cluster page appears, displaying the cluster creation process. Depending on the chosen cloud provider and other settings, it may take a few seconds to several minutes to provision the cluster. Once the cluster has been provisioned, the **Cluster Dashboard** page will be displayed.

    Now you can get started configuring apps and data on your new cluster.
  </Tab>
</Tabs>

## Get the cluster service URLs

Next, you'll need to get your StreamNative Cloud cluster's Service URLs to configure your client applications. These URLs allow your applications to connect to and interact with your cluster.

Your cluster provides different Service URLs depending on which protocol you plan to use:

For Pulsar Protocol:

* **Broker Service URL**: Used to configure Pulsar clients to connect to the brokers for producing and consuming messages
* **HTTP Service URL**: Used for administrative operations via the Pulsar admin API

For Kafka Protocol:

* **Kafka Service URL**: Used as bootstrap servers in Kafka client configurations
* **Schema Registry URL**: Used to configure Kafka serializers and deserializers that work with schemas

You'll need these URLs in the upcoming steps when setting up your producer and consumer clients.

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

## Create a service account and API key

Next, to connect your client application to your cluster to produce and consume messages, you need to create a [Service Account](/cloud/security/authentication/service-accounts/manage-service-accounts) and choose an authentication mechanism: either [API Key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview) or [OAuth](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview).

API Key authentication is quicker to implement since you only need to create an API key in StreamNative Cloud. This guide uses API Key authentication.

<Note title="Note">
  Currently, you can't edit a service account. If you need a service account to have Super Admin access, make sure to enable it when creating the service account. By default, service accounts do not have Super Admin enabled.
</Note>

<Tabs>
  <Tab title="Cloud Console">
    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Accounts & Accesses**.

    2. On the left navigation pane, click **Service Accounts**.

    3. On the **Service Account** page, click **+ New Service Account**.

    4. On the **Create Service Account** dialog, enter a name for the service account, and then click **Confirm**.

    5. On the **Service Account** page, in the row of the service account you just created, click the **...** icon, and select **Create API Key** in the dropdown menu.

    6. On the **New API Key** dialog:

       * Enter a name for the API key
       * Set the expiration date
       * Select the instance you created in Step 2
       * Write a description for the API key
       * Click **Confirm**

    <Note title="Note">
      An API key and associated secret apply to the active StreamNative instance. If you add a new instance, you must create a new API key for producers and consumers on the new Pulsar instance. For more information, see [Use API Keys to Authenticate to StreamNative Cloud](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview).
    </Note>

    7. After the API key is created, you can see the API key shown in the **New API Key** dialog. Click the **Copy and close** button to copy the API key to your clipboard. **Please note that you cannot retrieve the API key later after closing the dialog.**
  </Tab>
</Tabs>

Note the API key as we will use them when configuring the producer and consumer clients in upcoming steps.


# Consume Messages
Source: https://docs.streamnative.io/clients/kafka-clients/python/tutorial/kafka-python-consume-messages



Make the `consumer.py` file executable by running the following command:

```bash theme={null}
chmod +x consumer.py
```

Run the consumer by executing the following command:

```bash theme={null}
./consumer.py
```

You should see output resembling this:

```bash theme={null}
Consumed event from topic purchases: key = awalther     value = batteries
Consumed event from topic purchases: key = awalther     value = gift card
Consumed event from topic purchases: key = awalther     value = book
Consumed event from topic purchases: key = htanaka      value = book
Consumed event from topic purchases: key = jbernard     value = alarm clock
Consumed event from topic purchases: key = eabara       value = gift card
Consumed event from topic purchases: key = jsmith       value = batteries
Consumed event from topic purchases: key = sgarcia      value = alarm clock
Consumed event from topic purchases: key = jsmith       value = book
Consumed event from topic purchases: key = sgarcia      value = alarm clock
```

Enter `Ctrl-C` to terminate the consumer application.


# Create Project
Source: https://docs.streamnative.io/clients/kafka-clients/python/tutorial/kafka-python-create-project



<Note title="Note">
  Please ensure you use version `2.6.0` or later of the `confluent-kafka` library.

  StreamNative Cloud does not support **librdkafka** versions `2.5.0` and `2.5.3`, or any client SDKs based on these versions (including confluent-kafka-python `2.5.0` and `2.5.3`). This is due to a backward compatibility regression in librdkafka that was fixed in version 2.6.0. For more information, see [librdkafka #4871](https://github.com/edenhill/librdkafka/issues/4871).

  For more details about version compatibility, see [StreamNative Cloud Kafka Compatibility](https://docs.streamnative.io/docs/kafka-compatibility).
</Note>

Create a new directory anywhere you’d like for this project:

```bash theme={null}
mkdir kafka-python-getting-started && cd kafka-python-getting-started
```

Create and activate a Python virtual environment to give yourself a clean, isolated environment for this project. You may use other virtual environment managers like `venv` if you prefer.

```bash theme={null}
virtualenv venv
source venv/bin/activate
```

Install the Apache Kafka Python client library:

```bash theme={null}
pip install confluent-kafka
```


# Create Topic
Source: https://docs.streamnative.io/clients/kafka-clients/python/tutorial/kafka-python-create-topic



In a StreamNative Cluster, topics are grouped into [tenants](/cloud/manage-data-streams/tenant) and [namespaces](/cloud/manage-data-streams/namespace). In this guide, we will use the existing `public/default` namespace.

## Authorize the Service Account

<Note title="Note">
  If you create a super-user service account, you can skip this step.
</Note>

Before configuring the producer and consumer clients, you need to authorize the service account to grant the necessary permissions for the service account to interact with your StreamNative Cloud cluster.

To grant the service account permissions on the namespace level, follow these steps:

<Tabs>
  <Tab title="Cloud Console">
    1. Navigate to the **Namespace Dashboard** page by [switching to the namespace workspace](/cloud/get-started/cloud-console#switch-a-namespace).

    2. On the **Namespace Dashboard** page, click **Configuration** on the left navigation pane.

    3. On the **Namespace configuration** page, click **ADD ROLE**, and select the service account that you want to authorize.

    4. On the drop-down menu below the service account, select the proper permissions to assign to the service account. There are six permissions in total:

    * `consume`: allow the service account to consume messages.
    * `produce`: allow the service account to publish messages.
    * `functions`: allow the service account to submit and manage functions.
    * `sinks`: allow the service account to create and manage sink connectors.
    * `sources`: allow the service account to create and manage source connectors.
    * `packages`: allow the service account to upload and manage pulsar packages. If you want to submit a customized function/connector, then you will need to upload the function/connector’s JAR/NAR/Python file first, which requires the `packages` permission.

      <img alt="Authorize Service Account" />
  </Tab>
</Tabs>

In this guide, we will authorize the service account with the `produce` and `consume` permissions for the `public/default` namespace.

## Create a Topic

Navigate to the **Namespace Dashboard** page of the `public/default` namespace and follow the instructions below to create a topic, `purchases`, which you will use to produce and consume events.

<Tabs>
  <Tab title="StreamNative Console">
    To create a topic, follow these steps.

    1. Navigate to the desired **Namespace Dashboard** page by [switching to the namespace workspace](/cloud/get-started/cloud-console#switch-a-namespace).

    2. On the left navigation pane, under **Resources**, click **Topics**.

    3. Click **New Topic**.

       <img alt="screenshot of new topic dialog box" />

    4. Configure the topic, as outlined in the following table.

       | Item       | Description                                                                                                                                                                                                                                                                    |
       | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
       | Persistent | Configure the type of the topic. <br />- Persistent: messages in a persistent topic are durably persisted on the storage disk. <br />- Non-persistent: messages in a non-persistent topic are not persisted on the storage disk. <br /> By default, it is set to *Persistent*. |
       | Topic Name | Enter a name for the topic. It is a string of up to 40 characters, supporting lowercase letters (a-z), numeric characters (0-9), and the special character hyphen (-).                                                                                                         |
       | Partitions | (Optional) Configure the number of partitions for a partitioned topic. You can have from 1 up to 100 partitions per topic. It's recommended to have at least 1 partition.                                                                                                      |

    5. Click **Confirm**.
  </Tab>
</Tabs>


# Introduction
Source: https://docs.streamnative.io/clients/kafka-clients/python/tutorial/kafka-python-introduction



In this tutorial, you will build Python client applications which produce and consume messages from an Apache Kafka® cluster.

As you learn how to run your first Kafka application, we recommend using [StreamNative Cloud](https://streamnative.io/deployment) so you don't have to run your own Kafka cluster and can focus on client development.

If you do not already have an account, you can follow the steps below to sign up for StreamNative Cloud. New signups receive \$200 to spend within StreamNative Cloud. No credit card is required for the first 30 days or until your credits run out.

## Sign up for StreamNative Cloud

<Note title="Note">
  If you have an email account configured for using Single Sign-On (SSO) with StreamNative Cloud, use that email address and password when signing up.
</Note>

To sign up, navigate to the [StreamNative Cloud Console login page](https://console.streamnative.cloud/?defaultMethod=signup). Follow the prompts to create an account.

After you click **Finish**, you might have to wait briefly for your first organization to be created.


# Prerequisites
Source: https://docs.streamnative.io/clients/kafka-clients/python/tutorial/kafka-python-prerequisites



Using Windows? You'll need to download [Windows Subsystem for Linux](https://learn.microsoft.com/en-us/windows/wsl/install).

This guide assumes that you already have [Python 3](https://www.python.org/downloads/) installed.

This instructions use `virtualenv` but you may use other virtual environment managers like `venv` if you prefer.

```bash theme={null}
pip3 install virtualenv
```


# Produce Messages
Source: https://docs.streamnative.io/clients/kafka-clients/python/tutorial/kafka-python-produce-messages



Make the `producer.py` file executable by running the following command:

```bash theme={null}
chmod +x producer.py
```

Run the producer by executing the following command:

```bash theme={null}
./producer.py
```

You should see output resembling this:

```bash theme={null}
Produced event to topic purchases: key = jbernard     value = alarm clock
Produced event to topic purchases: key = eabara       value = gift card
Produced event to topic purchases: key = jsmith       value = batteries
Produced event to topic purchases: key = sgarcia      value = alarm clock
Produced event to topic purchases: key = jsmith       value = book
Produced event to topic purchases: key = sgarcia      value = alarm clock
Produced event to topic purchases: key = awalther     value = batteries
Produced event to topic purchases: key = awalther     value = gift card
Produced event to topic purchases: key = awalther     value = book
Produced event to topic purchases: key = htanaka      value = book
```


# What's Next
Source: https://docs.streamnative.io/clients/kafka-clients/python/tutorial/kafka-python-whats-next



* For the Python client API, checkout the [Python documentation](https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html)


# Build Consumer
Source: https://docs.streamnative.io/clients/kafka-clients/spring-boot/tutorial/kafka-spring-boot-build-consumer



Next, create the Java consumer application by pasting the following code into a file `src/main/java/examples/Consumer.java`.

```java theme={null}
package examples;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Service;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.kafka.support.KafkaHeaders;
import org.springframework.messaging.handler.annotation.Header;

import java.io.IOException;

@Service
public class Consumer {

    private final Logger logger = LoggerFactory.getLogger(Consumer.class);

    @KafkaListener(id = "myConsumer", topics = "purchases", groupId = "spring-boot", autoStartup = "false")
    public void listen(String value,
        @Header(KafkaHeaders.RECEIVED_TOPIC) String topic,
        @Header(KafkaHeaders.RECEIVED_KEY) String key) {
        logger.info(String.format("Consumed event from topic %s: key = %-10s value = %s", topic, key, value));
    }
}
```

Once again, you can compile the code before proceeding by running the following command:

```bash theme={null}
gradle build
```

And you should see the following output:

```
BUILD SUCCESSFUL in 1s
```


# Build Producer
Source: https://docs.streamnative.io/clients/kafka-clients/spring-boot/tutorial/kafka-spring-boot-build-producer



Create a directory for the Spring Boot application resource file:

```bash theme={null}
mkdir -p src/main/resources
```

Paste the following configuration data into a file located at `src/main/resources/application.yaml`, substituting your cluster bootstrap servers endpoint and the API key and secret that you just created for the username and password fields, respectively, of the `spring.kafka.properties.sasl.jaas.config` value.

```yaml theme={null}
spring:
  kafka:
    bootstrap-servers: <BOOTSTRAP SERVERS>
    properties:
      security:
        protocol: SASL_SSL
      sasl:
        jaas:
          config: org.apache.kafka.common.security.plain.PlainLoginModule required username='unused' password='token:<API KEY>';
        mechanism: PLAIN
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer
    consumer:
      group-id: group_id
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
```

Create a directory for the Java files in this project:

```bash theme={null}
mkdir -p src/main/java/examples
```

We will use `SpringBootApplication` annotation for ease of use, auto-configuration and component scanning. Paste the following Java code into a file located at `src/main/java/examples/SpringBootWithKafkaApplication.java`.

```java theme={null}
package examples;

import examples.Producer;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.listener.MessageListenerContainer;
import org.springframework.kafka.config.KafkaListenerEndpointRegistry;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.WebApplicationType;
import org.springframework.context.annotation.Bean;


@SpringBootApplication
public class SpringBootWithKafkaApplication {

    private final Producer producer;

    public static void main(String[] args) {
        SpringApplication application = new SpringApplication(SpringBootWithKafkaApplication.class);
        application.setWebApplicationType(WebApplicationType.NONE);
        application.run(args);
    }

    @Bean
    public CommandLineRunner CommandLineRunnerBean() {
        return (args) -> {
            for (String arg : args) {
                switch (arg) {
                  case "--producer":
                        this.producer.sendMessage("awalther", "t-shirts");
                        this.producer.sendMessage("htanaka", "t-shirts");
                        this.producer.sendMessage("htanaka", "batteries");
                        this.producer.sendMessage("eabara", "t-shirts");
                        this.producer.sendMessage("htanaka", "t-shirts");
                        this.producer.sendMessage("jsmith", "book");
                        this.producer.sendMessage("awalther", "t-shirts");
                        this.producer.sendMessage("jsmith", "batteries");
                        this.producer.sendMessage("jsmith", "gift card");
                        this.producer.sendMessage("eabara", "t-shirts");
                        break;
                  case "--consumer":
                        MessageListenerContainer listenerContainer = kafkaListenerEndpointRegistry.getListenerContainer("myConsumer");
                        listenerContainer.start();
                        break;
                  default:
                        break;
                }
            }
        };
    }

    @Autowired
    SpringBootWithKafkaApplication(Producer producer) {
        this.producer = producer;
    }

    @Autowired
    private KafkaListenerEndpointRegistry kafkaListenerEndpointRegistry;

}
```

You can test the syntax before proceding by running the following command:

```bash theme={null}
gradle build
```

And you should see the following output:

```bash theme={null}
BUILD SUCCESSFUL in 1s
```


# Cluster Setup
Source: https://docs.streamnative.io/clients/kafka-clients/spring-boot/tutorial/kafka-spring-boot-cluster-setup



We are going to need a StreamNative Cloud cluster for our client application to operate with. This guide can help you create your first StreamNative Cloud cluster.

## Create a StreamNative Cloud cluster

From within the StreamNative Cloud console, creating a new cluster is just a few clicks.

<Tabs>
  <Tab title="Cloud Console">
    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Organizations** to list your created organizations.

    2. Click the name of your first organization. In the figure below, the organization name is `Demos`.

       <img alt="screenshot of organization section" />

    3. On the **Select an Instance** card of the **Organization Dashboard** page, click the `+` button.

    4. On the **Choose the deployment type for your instance**, click **Deploy Serverless** to start the instance creation process.

    5. On the **Instance Configuration** page, enter a name for your instance, and select a Cloud Provider. The instance name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

    6. Click **Cluster Location** to start the cluster creation process.

    7. On the **Cluster Location** page, enter a name for your cluster, select the cluster location, and then click **Finish**. The cluster name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

    The cluster page appears, displaying the cluster creation process. Depending on the chosen cloud provider and other settings, it may take a few seconds to several minutes to provision the cluster. Once the cluster has been provisioned, the **Cluster Dashboard** page will be displayed.

    Now you can get started configuring apps and data on your new cluster.
  </Tab>
</Tabs>

## Get the cluster service URLs

Next, you'll need to get your StreamNative Cloud cluster's Service URLs to configure your client applications. These URLs allow your applications to connect to and interact with your cluster.

Your cluster provides different Service URLs depending on which protocol you plan to use:

For Pulsar Protocol:

* **Broker Service URL**: Used to configure Pulsar clients to connect to the brokers for producing and consuming messages
* **HTTP Service URL**: Used for administrative operations via the Pulsar admin API

For Kafka Protocol:

* **Kafka Service URL**: Used as bootstrap servers in Kafka client configurations
* **Schema Registry URL**: Used to configure Kafka serializers and deserializers that work with schemas

You'll need these URLs in the upcoming steps when setting up your producer and consumer clients.

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

## Create a service account and API key

Next, to connect your client application to your cluster to produce and consume messages, you need to create a [Service Account](/cloud/security/authentication/service-accounts/manage-service-accounts) and choose an authentication mechanism: either [API Key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview) or [OAuth](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview).

API Key authentication is quicker to implement since you only need to create an API key in StreamNative Cloud. This guide uses API Key authentication.

<Note title="Note">
  Currently, you can't edit a service account. If you need a service account to have Super Admin access, make sure to enable it when creating the service account. By default, service accounts do not have Super Admin enabled.
</Note>

<Tabs>
  <Tab title="Cloud Console">
    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Accounts & Accesses**.

    2. On the left navigation pane, click **Service Accounts**.

    3. On the **Service Account** page, click **+ New Service Account**.

    4. On the **Create Service Account** dialog, enter a name for the service account, and then click **Confirm**.

    5. On the **Service Account** page, in the row of the service account you just created, click the **...** icon, and select **Create API Key** in the dropdown menu.

    6. On the **New API Key** dialog:

       * Enter a name for the API key
       * Set the expiration date
       * Select the instance you created in Step 2
       * Write a description for the API key
       * Click **Confirm**

    <Note title="Note">
      An API key and associated secret apply to the active StreamNative instance. If you add a new instance, you must create a new API key for producers and consumers on the new Pulsar instance. For more information, see [Use API Keys to Authenticate to StreamNative Cloud](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview).
    </Note>

    7. After the API key is created, you can see the API key shown in the **New API Key** dialog. Click the **Copy and close** button to copy the API key to your clipboard. **Please note that you cannot retrieve the API key later after closing the dialog.**
  </Tab>
</Tabs>

Note the API key as we will use them when configuring the producer and consumer clients in upcoming steps.


# Consume Messages
Source: https://docs.streamnative.io/clients/kafka-clients/spring-boot/tutorial/kafka-spring-boot-consume-messages



Run the following command to run the Spring Boot application for the Consumer.

```bash theme={null}
gradle bootRun --args='--consumer'
```

The consumer application will start and print any events it has not yet consumed and then wait for more events to arrive. On startup of the consumer, you should see output resembling this:

```bash theme={null}
2024-11-16T21:10:53.186-08:00  INFO 23554 --- [yConsumer-0-C-1] examples.Consumer                        : Consumed event from topic purchases: key = jsmith     value = gift card
2024-11-16T21:10:53.186-08:00  INFO 23554 --- [yConsumer-0-C-1] examples.Consumer                        : Consumed event from topic purchases: key = sgarcia    value = gift card
2024-11-16T21:10:53.186-08:00  INFO 23554 --- [yConsumer-0-C-1] examples.Consumer                        : Consumed event from topic purchases: key = sgarcia    value = t-shirts
2024-11-16T21:10:53.186-08:00  INFO 23554 --- [yConsumer-0-C-1] examples.Consumer                        : Consumed event from topic purchases: key = sgarcia    value = gift card
2024-11-16T21:10:53.186-08:00  INFO 23554 --- [yConsumer-0-C-1] examples.Consumer                        : Consumed event from topic purchases: key = sgarcia    value = book
2024-11-16T21:10:53.186-08:00  INFO 23554 --- [yConsumer-0-C-1] examples.Consumer                        : Consumed event from topic purchases: key = sgarcia    value = gift card
2024-11-16T21:10:53.186-08:00  INFO 23554 --- [yConsumer-0-C-1] examples.Consumer                        : Consumed event from topic purchases: key = sgarcia    value = t-shirts
2024-11-16T21:10:53.186-08:00  INFO 23554 --- [yConsumer-0-C-1] examples.Consumer                        : Consumed event from topic purchases: key = sgarcia    value = batteries
2024-11-16T21:10:53.186-08:00  INFO 23554 --- [yConsumer-0-C-1] examples.Consumer                        : Consumed event from topic purchases: key = htanaka    value = batteries
2024-11-16T21:10:53.186-08:00  INFO 23554 --- [yConsumer-0-C-1] examples.Consumer                        : Consumed event from topic purchases: key = htanaka    value = book
2024-11-16T21:10:53.186-08:00  INFO 23554 --- [yConsumer-0-C-1] examples.Consumer                        : Consumed event from topic purchases: key = awalther   value = book
2024-11-16T21:10:53.186-08:00  INFO 23554 --- [yConsumer-0-C-1] examples.Consumer                        : Consumed event from topic purchases: key = htanaka    value = t-shirts
2024-11-16T21:10:53.186-08:00  INFO 23554 --- [yConsumer-0-C-1] examples.Consumer                        : Consumed event from topic purchases: key = awalther   value = alarm clock
2024-11-16T21:10:53.186-08:00  INFO 23554 --- [yConsumer-0-C-1] examples.Consumer                        : Consumed event from topic purchases: key = htanaka    value = alarm clock
2024-11-16T21:10:53.186-08:00  INFO 23554 --- [yConsumer-0-C-1] examples.Consumer                        : Consumed event from topic purchases: key = sgarcia    value = gift card
2024-11-16T21:10:53.186-08:00  INFO 23554 --- [yConsumer-0-C-1] examples.Consumer                        : Consumed event from topic purchases: key = sgarcia    value = t-shirts
2024-11-16T21:10:53.186-08:00  INFO 23554 --- [yConsumer-0-C-1] examples.Consumer                        : Consumed event from topic purchases: key = sgarcia    value = book
```

Rerun the producer to see more events, or feel free to modify the code as necessary to create more or different events.

Once you are done, enter `Ctrl-C` to terminate the consumer application.


# Create Project
Source: https://docs.streamnative.io/clients/kafka-clients/spring-boot/tutorial/kafka-spring-boot-create-project



Create a new directory anywhere you’d like for this project:

```bash theme={null}
mkdir kafka-spring-boot-getting-started && cd kafka-spring-boot-getting-started
```

Create the following Gradle build file for the project, named `build.gradle`:

```gradle theme={null}
buildscript {
  repositories {
    jcenter()
  }
}

plugins {
  id 'org.springframework.boot' version "3.2.3"
  id 'io.spring.dependency-management' version '1.1.4'
  id 'java'
}

repositories {
  jcenter()
}

apply plugin: 'idea'

group = 'examples'
version = '0.0.1'
sourceCompatibility = 17

repositories {
  jcenter()
}

dependencies {
  implementation 'org.springframework.boot:spring-boot-starter-web:3.2.3'
  implementation 'org.apache.kafka:kafka-clients'
  implementation 'org.springframework.kafka:spring-kafka'
}

bootRun {
  systemProperties "java.security.manager": "allow"
}
```


# Create Topic
Source: https://docs.streamnative.io/clients/kafka-clients/spring-boot/tutorial/kafka-spring-boot-create-topic



In a StreamNative Cluster, topics are grouped into [tenants](/cloud/manage-data-streams/tenant) and [namespaces](/cloud/manage-data-streams/namespace). In this guide, we will use the existing `public/default` namespace.

## Authorize the Service Account

<Note title="Note">
  If you create a super-user service account, you can skip this step.
</Note>

Before configuring the producer and consumer clients, you need to authorize the service account to grant the necessary permissions for the service account to interact with your StreamNative Cloud cluster.

To grant the service account permissions on the namespace level, follow these steps:

<Tabs>
  <Tab title="Cloud Console">
    1. Navigate to the **Namespace Dashboard** page by [switching to the namespace workspace](/cloud/get-started/cloud-console#switch-a-namespace).

    2. On the **Namespace Dashboard** page, click **Configuration** on the left navigation pane.

    3. On the **Namespace configuration** page, click **ADD ROLE**, and select the service account that you want to authorize.

    4. On the drop-down menu below the service account, select the proper permissions to assign to the service account. There are six permissions in total:

    * `consume`: allow the service account to consume messages.
    * `produce`: allow the service account to publish messages.
    * `functions`: allow the service account to submit and manage functions.
    * `sinks`: allow the service account to create and manage sink connectors.
    * `sources`: allow the service account to create and manage source connectors.
    * `packages`: allow the service account to upload and manage pulsar packages. If you want to submit a customized function/connector, then you will need to upload the function/connector’s JAR/NAR/Python file first, which requires the `packages` permission.

      <img alt="Authorize Service Account" />
  </Tab>
</Tabs>

In this guide, we will authorize the service account with the `produce` and `consume` permissions for the `public/default` namespace.

## Create a Topic

Navigate to the **Namespace Dashboard** page of the `public/default` namespace and follow the instructions below to create a topic, `purchases`, which you will use to produce and consume events.

<Tabs>
  <Tab title="StreamNative Console">
    To create a topic, follow these steps.

    1. Navigate to the desired **Namespace Dashboard** page by [switching to the namespace workspace](/cloud/get-started/cloud-console#switch-a-namespace).

    2. On the left navigation pane, under **Resources**, click **Topics**.

    3. Click **New Topic**.

       <img alt="screenshot of new topic dialog box" />

    4. Configure the topic, as outlined in the following table.

       | Item       | Description                                                                                                                                                                                                                                                                    |
       | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
       | Persistent | Configure the type of the topic. <br />- Persistent: messages in a persistent topic are durably persisted on the storage disk. <br />- Non-persistent: messages in a non-persistent topic are not persisted on the storage disk. <br /> By default, it is set to *Persistent*. |
       | Topic Name | Enter a name for the topic. It is a string of up to 40 characters, supporting lowercase letters (a-z), numeric characters (0-9), and the special character hyphen (-).                                                                                                         |
       | Partitions | (Optional) Configure the number of partitions for a partitioned topic. You can have from 1 up to 100 partitions per topic. It's recommended to have at least 1 partition.                                                                                                      |

    5. Click **Confirm**.
  </Tab>
</Tabs>


# Introduction
Source: https://docs.streamnative.io/clients/kafka-clients/spring-boot/tutorial/kafka-spring-boot-introduction



In this tutorial, you will build a Spring Boot application which produces and consumes messages from an Apache Kafka® cluster.

As you learn how to run your first Kafka application, we recommend using [StreamNative Cloud](https://streamnative.io/deployment) so you don't have to run your own Kafka cluster and can focus on client development.

If you do not already have an account, you can follow the steps below to sign up for StreamNative Cloud. New signups receive \$200 to spend within StreamNative Cloud. No credit card is required for the first 30 days or until your credits run out.

## Sign up for StreamNative Cloud

<Note title="Note">
  If you have an email account configured for using Single Sign-On (SSO) with StreamNative Cloud, use that email address and password when signing up.
</Note>

To sign up, navigate to the [StreamNative Cloud Console login page](https://console.streamnative.cloud/?defaultMethod=signup). Follow the prompts to create an account.

After you click **Finish**, you might have to wait briefly for your first organization to be created.


# Prerequisites
Source: https://docs.streamnative.io/clients/kafka-clients/spring-boot/tutorial/kafka-spring-boot-prerequisites



Using Windows? You'll need to download [Windows Subsystem for Linux](https://learn.microsoft.com/en-us/windows/wsl/install).

This guide assumes that you already have:

* [Gradle](https://gradle.org/install/) installed.
* [Java 11](https://openjdk.java.net/install/) or later installed. Verify that `java -version` outputs a version number like `11.0.20` and ensure that the `JAVA_HOME` environment variable is set.


# Produce Messages
Source: https://docs.streamnative.io/clients/kafka-clients/spring-boot/tutorial/kafka-spring-boot-produce-messages



Run the following command to run the Spring Boot application for the Producer.

```bash theme={null}
gradle bootRun --args='--producer'
```

You should see output resembling this:

```bash theme={null}
2024-11-16T21:09:11.161-08:00  INFO 23421 --- [ad | producer-1] examples.Producer                        : Produced event to topic purchases: key = awalther   value = t-shirts
2024-11-16T21:09:11.162-08:00  INFO 23421 --- [ad | producer-1] examples.Producer                        : Produced event to topic purchases: key = htanaka    value = t-shirts
2024-11-16T21:09:11.162-08:00  INFO 23421 --- [ad | producer-1] examples.Producer                        : Produced event to topic purchases: key = htanaka    value = batteries
2024-11-16T21:09:11.162-08:00  INFO 23421 --- [ad | producer-1] examples.Producer                        : Produced event to topic purchases: key = eabara     value = t-shirts
2024-11-16T21:09:11.162-08:00  INFO 23421 --- [ad | producer-1] examples.Producer                        : Produced event to topic purchases: key = htanaka    value = t-shirts
2024-11-16T21:09:11.162-08:00  INFO 23421 --- [ad | producer-1] examples.Producer                        : Produced event to topic purchases: key = awalther   value = t-shirts
2024-11-16T21:09:11.162-08:00  INFO 23421 --- [ad | producer-1] examples.Producer                        : Produced event to topic purchases: key = eabara     value = t-shirts
2024-11-16T21:09:11.224-08:00  INFO 23421 --- [ad | producer-1] examples.Producer                        : Produced event to topic purchases: key = jsmith     value = book
2024-11-16T21:09:11.225-08:00  INFO 23421 --- [ad | producer-1] examples.Producer                        : Produced event to topic purchases: key = jsmith     value = batteries
2024-11-16T21:09:11.225-08:00  INFO 23421 --- [ad | producer-1] examples.Producer                        : Produced event to topic purchases: key = jsmith     value = gift card
```


# What's Next
Source: https://docs.streamnative.io/clients/kafka-clients/spring-boot/tutorial/kafka-spring-boot-whats-next



* For the Spring Boot API, checkout the [Spring for Apache Kafka](https://spring.io/projects/spring-kafka) documentation.


# StreamNative BYOC Overview
Source: https://docs.streamnative.io/cloud/clusters/byoc/byoc-overview



StreamNative delivers a fully-managed Pulsar solution operating within your cloud environment through the **Bring Your Own Cloud (BYOC)** deployment option. This approach is enabled by StreamNative Cloud's architecture, which separates the **control plane** from the **data plane**.

In a BYOC deployment:

* The infrastructure and network are under your ownership but managed by StreamNative.
* You utilize your own public cloud accounts (AWS, GCP, and Azure).
* The system requires minimal administration on your part.
* You control security protocols, maintain data visibility, and comply with data sovereignty requirements.

Below is a diagram illustrating the BYOC architecture within StreamNative Cloud.

<img alt="image of BYOC architecture" />

## Responsibility model

The responsibilities under the StreamNative BYOC model differ from those of the StreamNative Hosted model. For a comparison of responsibilities across different cloud deployment options, refer to the [Responsibility Model](https://streamnative.io/deployment/responsibility-model).

## How to provision a BYOC infrastructure pool

<span />

To deploy a StreamNative cluster within your cloud account, you must first set up a BYOC [infrastructure pool](/cloud/clusters/streamnative-cluster-overview#infrastructure-pools).

1. **Grant StreamNative Vendor Access**: Authorize the StreamNative Cloud control plane to access your cloud account by applying the [StreamNative Vendor Access](https://github.com/streamnative/terraform-managed-cloud) Terraform module. This module ensures only the minimum necessary access is used to establish your Pulsar clusters. For detailed information on our access approach, please see our [access model](/cloud/clusters/byoc/grant-vendor-access/byoc-on-aws/byoc-aws-access#access-model-in-aws).

2. **Create a Cloud Connection**: After granting access, establish a Cloud Connection to allow the StreamNative Cloud control plane to interact with your cloud account.

3. **Create Cloud Environments**: With a Cloud Connection in place, you can then create one or more Cloud Environments. Each Cloud Environment encompasses the essential infrastructure resources—compute, storage, and networking—needed for deploying Pulsar clusters.

After setting up one or more Cloud Environments, you can [create an Instance](/cloud/clusters/manage-instances/instance#create-an-instance) and [deploy a Pulsar Cluster](/cloud/clusters/manage-clusters/cluster#create-a-cluster) to your cloud account that the cloud connection uses.

You can watch the playlist of [Provisioning StreamNative BYOC Clusters](https://www.youtube.com/watch?v=7BOhr4-8yqo\&list=PL7-BmxsE3q4W5QnrusLyYt9_HbX4R7vEN).

<Note title="Note">
  When you [create an Instance](/tools/cli/snctl/snctl-tutorials#create-an-instance) with `snctl`, you need to specify a `poolRef`. This `poolRef` refers to your Cloud Connection.
</Note>

### BYOC Provisioning FAQs

* **Can I create multiple Cloud Environments in the same region under one Cloud Connection?** Yes - you can create multiple Cloud Environments in the same region under a single Cloud Connection.

* **Can I create multiple Cloud Environments in different regions under one Cloud Connection?** Yes - you can create multiple Cloud Environments in different regions under a single Cloud Connection.

* **Can I create multiple Cloud Connections pointing to the same AWS/GCP/Azure account?** Yes - within an organization, you can create multiple Cloud Connections that point to the same AWS, GCP, or Azure account. However, unless you have specific requirements, this is not recommended as it can cause confusion.

* **What are the requirements for geo replication?** For geo replication, all Pulsar clusters under the same Pulsar Instance must be deployed within the same Cloud Environments under the same Cloud Connection.

## Next steps

### Deploy BYOC Clusters on AWS

1. [Grant StreamNative access to your AWS account](/cloud/clusters/byoc/grant-vendor-access/byoc-on-aws/byoc-aws-access).
2. [Create a Cloud Connection](/cloud/clusters/byoc/create-cloud-connection) to your AWS account.
3. [Create Cloud Environments](/cloud/clusters/byoc/create-cloud-environment) in your AWS account.
4. [Create a BYOC Instance](/cloud/clusters/manage-instances/instance#create-streamnative-instance) using the established cloud connection.
5. [Create a BYOC cluster](/cloud/clusters/manage-instances/instance#create-streamnative-cluster).

If you need to configure a custom domain for your Pulsar cluster (BYOC Pro only), see [set up custom AWS DNS domain](/cloud/clusters/byoc/grant-vendor-access/byoc-on-aws/byoc-aws-dns-domain).

### Deploy BYOC Clusters on GCP

1. [Grant StreamNative access to your GCP project](/cloud/clusters/byoc/grant-vendor-access/byoc-on-gcp/byoc-gcp-access).
2. [Create a Cloud Connection](/cloud/clusters/byoc/create-cloud-connection) to your GCP project.
3. [Create Cloud Environments](/cloud/clusters/byoc/create-cloud-environment) in your GCP project.
4. [Create a BYOC Instance](/cloud/clusters/manage-instances/instance#create-streamnative-instance) using the established cloud connection.
5. [Create a BYOC cluster](/cloud/clusters/manage-instances/instance#create-streamnative-cluster).

### Deploy BYOC Clusters on Azure

1. [Grant StreamNative access to your Azure account](/cloud/clusters/byoc/grant-vendor-access/byoc-on-azure/byoc-azure-access).
2. [Create a Cloud Connection](/cloud/clusters/byoc/create-cloud-connection) to your Azure account.
3. [Create Cloud Environments](/cloud/clusters/byoc/create-cloud-environment) in your Azure account.
4. [Create a BYOC Instance](/cloud/clusters/manage-instances/instance#create-streamnative-instance) using the established cloud connection.
5. [Create a BYOC cluster](/cloud/clusters/manage-instances/instance#create-streamnative-cluster).


# Manage Cloud Connections on StreamNative Cloud
Source: https://docs.streamnative.io/cloud/clusters/byoc/create-cloud-connection



<Note title="Note">
  This document assumes that you have already run the [Vendor Access Module](https://github.com/streamnative/terraform-managed-cloud) for your respective cloud provider. StreamNative will not be able to connect to your cloud account until you have done so. If you have not yet done so, please run the Vendor Access Module for your cloud provider using the instructions provided ([AWS](/cloud/clusters/byoc/grant-vendor-access/byoc-on-aws/byoc-aws-access), [Azure](/cloud/clusters/byoc/grant-vendor-access/byoc-on-azure/byoc-azure-access), and [GCP](/cloud/clusters/byoc/grant-vendor-access/byoc-on-gcp/byoc-gcp-access)), and then return to this page to continue setting up your Cloud Connection.
</Note>

Cloud Connections allow StreamNative to connect to your AWS, GCP, or Azure account, and provision your Cloud Environment so that it can run Pulsar Clusters. You can create a Cloud Connection from [Cloud Console UI](/cloud/get-started/cloud-console), [`snctl`](/tools/cli/snctl/snctl-overview), or StreamNative's terraform provider.

## Create a Cloud Connection

A **Cloud Connection** represents a connection between StreamNative Cloud and your cloud account. It contains the necessary account information for StreamNative to access your cloud account but doesn't contain any credentials. StreamNative Cloud uses **assume role** to access your cloud account. To ensure StreamNative Cloud can access your cloud account, you need to grant StreamNative Cloud permission to assume the role in your cloud account. See [BYOC Overview](/cloud/clusters/byoc/byoc-overview) for more details. When you successfully create a Cloud Connection, it provisions a **Pool** that will be used for provisioning your Cloud Environment, which you'll later use for provisioning your [Instances](/cloud/clusters/manage-instances/instance) and [Clusters](/cloud/clusters/manage-clusters/cluster).

The information you will need to provide depends on the cloud provider you are using. Here is a summary of the information you need to provide for each cloud provider:

* **AWS**: The AWS **account ID**.
* **GCP**: The GCP **project ID**.
* **Azure**: The Azure **subscription ID**, **tenant ID**, **client ID**, and **support client ID**

Once you have the information you need, you can create a Cloud Connection using one of the following methods:

<Tabs>
  <Tab title="Console">
    **Tutorial**

    You can watch the video of creating a cloud connection using Cloud Console UI:

    [![Create Cloud Connection](https://img.youtube.com/vi/ubRcgeOhHKw/0.jpg)](https://www.youtube.com/watch?v=ubRcgeOhHKw\&list=PL7-BmxsE3q4W5QnrusLyYt9_HbX4R7vEN\&index=3)

    **Step-by-step guide**

    1. In the upper-right corner of Cloud Console, click your user profile, and in the dropdown menu, click **Cloud Environments**.

    2. On the **Cloud Environments** page, click **Cloud Connections** tab.

    3. On the **Cloud Connections** tab, click **+ New Cloud Connection** button to create a new Cloud Connection.

       <img alt="Create CloudConnection" />

    4. Enter the **name** of the Cloud Connection.

    5. Select the **connection provider** of the Cloud Connection and fill out the required fields:

       * **AWS**:
         * **AWS Account ID**: The AWS **Account ID**
       * **Google Cloud**:
         * **Google Cloud Project ID**: The Google Cloud **Project ID**
       * **Azure**:
         * **Subscription ID**: The Azure **Subscription ID**
         * **Tenant ID**: The Azure **Tenant ID**
         * **Client ID**: The Azure **Client ID**
         * **Support Client ID**: The Azure **Support Client ID**

    6. Select **Confirm if vendor access Terraform module is executed** checkbox after you have executed the [Vendor Access Module](https://github.com/streamnative/terraform-managed-cloud) for your cloud provider.

    7. Click **Submit** to create the Cloud Connection.
  </Tab>

  <Tab title="snctl">
    **Tutorial**

    You can watch the video of creating a cloud connection using `snctl`:

    [![Create Cloud Connection](https://img.youtube.com/vi/hBUeArMyV6c/0.jpg)](https://www.youtube.com/watch?v=ETr2EvKTht8\&list=PL7-BmxsE3q4W5QnrusLyYt9_HbX4R7vEN\&index=5)

    **Step-by-step guide**

    To create a Cloud Connection using snctl, use `snctl create cloudconnection`. Usage:

    ```bash theme={null}
    snctl create cloudconnection [NAME] [flags]
    ```

    | Flag                | Description                                                  |
    | ------------------- | ------------------------------------------------------------ |
    | --account-id        | The account ID of your AWS account if `type` is `aws`.       |
    | --client-id         | The client ID of your Azure account if `type` is `azure`.    |
    | -h, --help          | Displays Cloud Connection help message.                      |
    | --project-id        | The project ID of your GCP project if `type` is `gcp`..      |
    | --subscription-id   | The subscription ID of Azure account if `type` is `azure`.   |
    | --support-client-id | The support client ID of Azure account if `type` is `azure`. |
    | --tenant-id         | The tenant ID of Azure account if `type` is `azure`.         |
    | --type              | The type of cloud provider, one of: `aws`, `gcp` or `azure`. |

    **Examples**

    <Tabs>
      <Tab title="AWS">
        ```bash theme={null}
        snctl create cloudconnection shared-aws --type aws --account-id ACCOUNT_ID -O orgname
        ```

        * Replace `ACCOUNT_ID` with your AWS account ID.
      </Tab>

      <Tab title="GCP">
        ```bash theme={null}
        snctl create cloudconnection shared-gcp --type gcp --project-id GCP_PROJECT_ID -O orgname
        ```

        * Replace `GCP_PROJECT_ID` with your GCP project ID.
      </Tab>

      <Tab title="Azure">
        ```bash theme={null}
        snctl create cloudconnection shared-azure --type azure --subscription-id SUBSCRIPTION_ID --tenant-id TENANT_ID --client-id CLIENT_ID --support-client-id SUPPORT_CLIENT_ID -O orgname
        ```

        * Replace `SUBSCRIPTION_ID` with your Azure **Subscription ID**.
        * Replace `TENANT_ID` with your Azure **Tenant ID**.
        * Replace `CLIENT_ID` with your Azure **Client ID**.
        * Replace `SUPPORT_CLIENT_ID` with your Azure **Support Client ID**.
      </Tab>
    </Tabs>

    **Manifest file**

    Alternatively, you can prepare a manifest file `cloudconnection.yaml` to define a cloud connection, and then use `snctl` to create the cloud connection:

    <Tabs>
      <Tab title="AWS">
        ```yaml theme={null}
        apiVersion: cloud.streamnative.io/v1alpha1
        kind: CloudConnection
        metadata:
          name: CLOUD_CONNECTION_NAME
          namespace: YOUR_ORG_ID
        spec:
          aws:
            accountId: 'ACCOUNT_ID'
          type: aws
        ```

        * Replace `CLOUD_CONNECTION_NAME` with the name of the cloud connection.
        * Replace `ACCOUNT_ID` with your AWS account ID.
        * Replace `YOUR_ORG_ID` with your StreamNative Cloud organization ID.
      </Tab>

      <Tab title="GCP">
        ```yaml theme={null}
        apiVersion: cloud.streamnative.io/v1alpha1
        kind: CloudConnection
        metadata:
          name: CLOUD_CONNECTION_NAME
          namespace: YOUR_ORG_ID
        spec:
          gcp:
            projectId: 'GCP_PROJECT_ID'
          type: gcp
        ```

        * Replace `CLOUD_CONNECTION_NAME` with the name of the cloud connection.
        * Replace `GCP_PROJECT_ID` with your GCP project ID.
        * Replace `YOUR_ORG_ID` with your StreamNative Cloud organization ID.
      </Tab>

      <Tab title="Azure">
        ```yaml theme={null}
        apiVersion: cloud.streamnative.io/v1alpha1
        kind: CloudConnection
        metadata:
          name: CLOUD_CONNECTION_NAME
          namespace: YOUR_ORG_ID
        spec:
          gcp:
            subscriptionId: 'SUBSCRIPTION_ID'
            tenantId: 'TENANT_ID'
            clientId: 'CLIENT_ID'
            supportClientId: 'SUPPORT_CLIENT_ID'
          type: azure
        ```

        * Replace `CLOUD_CONNECTION_NAME` with the name of the cloud connection.
        * Replace `YOUR_ORG_ID` with your StreamNative Cloud organization ID.
        * Replace `SUBSCRIPTION_ID` with your Azure **Subscription ID**.
        * Replace `TENANT_ID` with your Azure **Tenant ID**.
        * Replace `CLIENT_ID` with your Azure **Client ID**.
        * Replace `SUPPORT_CLIENT_ID` with your Azure **Support Client ID**.
      </Tab>
    </Tabs>

    Then you can create the connection using the following command:

    ```
    snctl create -f cloudconnection.yaml
    ```

    After creating the cloud connection, you can view its details by running `snctl get cloudconnection <name>`. If StreamNative Cloud can successfully access your cloud account, the status `AllConditionStatusTrue` will show as `ready`.
  </Tab>

  <Tab title="Terraform">
    **Tutorial**

    You can watch the video of creating a cloud connection using Terraform:

    [![Create Cloud Connection using Terraform](https://img.youtube.com/vi/9h2_1AGy-I4/0.jpg)](https://www.youtube.com/watch?v=J7S7A_1Tshc\&list=PL7-BmxsE3q4W5QnrusLyYt9_HbX4R7vEN\&index=4)

    **Step-by-step guide**

    To create a Cloud Connection with terraform:

    1. Prepare `main.tf` to define the cloud connection.
    2. Run `terraform init` to initialize the terraform project.
    3. Run `terraform plan` to review the changes.
    4. Run `terraform apply` to create the cloud connection.

    For additional details, please refer to our [Terraform module documentation on Cloud Connections](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/resources/cloud_connection).

    **Cloud Connection Schema**

    | Field                  | Type           | Description                                                          |
    | ---------------------- | -------------- | -------------------------------------------------------------------- |
    | name, required         | String         | Name of the cloud connection                                         |
    | organization, required | String         | The organization name                                                |
    | aws, read-only         | List of Object | AWS configuration for the connection (see below for nested schema)   |
    | azure, read-only       | List of Object | Azure configuration for the connection (see below for nested schema) |
    | gcp, read-only         | List of Object | GCP configuration for the connection (see below for nested schema)   |
    | id, read-only          | String         | The ID of this resource.                                             |
    | type, read-only        | String         | Type of cloud connection, `aws`, `gcp`, or `azure`                   |
    | account\_id, read-only | (String)       | Nested Schema for aws                                                |

    **Examples**

    <Tabs>
      <Tab title="AWS">
        ```hcl theme={null}
        module "sn_managed_cloud" {
          source = "github.com/streamnative/terraform-managed-cloud//modules/aws/vendor-access?ref=v3.23.0"

          external_id = "YOUR_SNCLOUD_ORG_ID"
        }

        resource "streamnative_cloud_connection" "shared_aws" {
            depends_on = [ module.sn_managed_cloud ]
            organization = "YOUR_SNCLOUD_ORG_ID"
            name = "CLOUD_CONNECTION_NAME"

            type = "aws"
            aws {
                account_id = "ACCOUNT_ID"
            }
        }
        ```

        * Replace `YOUR_SNCLOUD_ORG_ID` with your StreamNative Cloud organization ID.
        * Replace `CLOUD_CONNECTION_NAME` with the name of the cloud connection.
        * Replace `ACCOUNT_ID` with your AWS account ID.
      </Tab>

      <Tab title="GCP">
        ```hcl theme={null}
        provider "google" {
          project = "YOUR_GCP_PROJECT_ID"
        }

        module "sn_managed_cloud" {
          source = "github.com/streamnative/terraform-managed-cloud//modules/gcp/vendor-access?ref=v3.23.0"
          project = "YOUR_GCP_PROJECT_ID"
          streamnative_org_id = "YOUR_SNCLOUD_ORG_ID"
        }

        resource "streamnative_cloud_connection" "shared_gcp" {
            depends_on = [ module.sn_managed_cloud ]
            organization = "orgname"
            name = "CLOUD_CONNECTION_NAME"

            type = "gcp"
            gcp {
                project_id = "GCP_PROJECT_ID"
            }
        }
        ```

        * Replace `YOUR_GCP_PROJECT_ID` with your GCP project ID.
        * Replace `GCP_PROJECT_ID` with your GCP project ID.
        * Replace `CLOUD_CONNECTION_NAME` with the name of the cloud connection.
      </Tab>

      <Tab title="Azure">
        ```hcl theme={null}
        resource "streamnative_cloud_connection" "shared_azure" {
            organization = "orgname"
            name = "CLOUD_CONNECTION_NAME"

            type = "azure"
            azure {
                client_id = "CLIENT_ID"
                subscription_id = "SUBSCRIPTION_ID"
                support_client_id = "SUPPORT_CLIENT_ID"
                tenant_id = "TENANT_ID"
            }
        }
        ```

        * Replace `CLOUD_CONNECTION_NAME` with the name of the cloud connection.
        * Replace `CLIENT_ID` with your Azure **Client ID**.
        * Replace `SUBSCRIPTION_ID` with your Azure **Subscription ID**.
        * Replace `SUPPORT_CLIENT_ID` with your Azure **Support Client ID**.
        * Replace `TENANT_ID` with your Azure **Tenant ID**.
      </Tab>
    </Tabs>
  </Tab>
</Tabs>

## Update a Cloud Connection

After a **Cloud Connection** is created, it cannot be updated. If you need to modify any information, you must delete the existing Cloud Connection and create a new one with the correct details.

## Delete a Cloud Connection

<Warning title="Warning">
  Before deleting a cloud connection, you must first delete all associated cloud environments.

  Please note that deleting a cloud connection is an irreversible action. Exercise caution when performing this operation.
</Warning>

<Tabs>
  <Tab title="Console">
    1. In the upper-right corner of Cloud Console, click your user profile, and in the dropdown menu, click **Cloud Environments**.

    2. On the **Cloud Environments** page, click **Cloud Connections** tab.

    3. On the **Cloud Connections** tab, find the cloud connection you want to delete, and click the ellipsis (**...**) on the right side of the row, and then click **Delete**.

    4. On the **Delete cloud connection** page, enter the name of the cloud connection, and click **Confirm**.
  </Tab>

  <Tab title="snctl">
    You can delete a cloud connection using `snctl`:

    ```bash theme={null}
    snctl delete cloudconnection CLOUD_CONNECTION_NAME
    ```

    Alternatively, if you have the manifest file of the cloud connection, you can delete the cloud connection by running:

    ```bash theme={null}
    snctl delete -f cloudconnection.yaml
    ```
  </Tab>

  <Tab title="Terraform">
    You can remove the cloud connection from your terraform code and run `terraform apply` to delete the cloud connection.
  </Tab>
</Tabs>

## Next steps

After establishing a Cloud Connection, you can create one or more [Cloud Environments](/cloud/clusters/byoc/create-cloud-environment) to deploy your Pulsar clusters.


# Manage Cloud Environments on StreamNative Cloud
Source: https://docs.streamnative.io/cloud/clusters/byoc/create-cloud-environment



<Note title="Note">
  This document assumes that you have already created a **Cloud Connection** allowing StreamNative to connect to your cloud account. If you have not yet done so, please [create a Cloud Connection](/cloud/clusters/byoc/create-cloud-connection) first, and then return to this page to continue setting up your Cloud Environment.
</Note>

A **Cloud Connection** provisions a **Pool** that enables you to deploy BYOC instances. Within that Pool, you can create **Cloud Environments**, which provision **Pool Members** in your designated cloud region. After creating a Cloud Environment, you can [create an Instance](/cloud/clusters/manage-instances/instance) and then [create a cluster](/cloud/clusters/manage-clusters/cluster) within that instance.

## Plan a Cloud Environment

Before you start provisioning a cloud environment, you need to plan for the required information. The information you need to provide depends on your cloud provider. Here is a summary of the required information:

### Region

Select the region where the **Cloud Environment** will be created.

<Note title="Note">
  For Azure, the region is the resource group name, please ensure that you have created it as the [doc](/cloud/clusters/byoc/grant-vendor-access/byoc-on-azure/byoc-azure-access#step-6-create-a-resource-group) describes.
</Note>

### VPC

Choose between using a StreamNative-managed VPC or your own VPC (aka `Bring Your Own Network`, or `BYON`).

#### StreamNative-managed VPC

If you decide to use a StreamNative-managed VPC, you need to provide the **CIDR** value for the VPC. Below are the recommended default CIDR values for each cloud provider:

* AWS
  * Default VPC CIDR: `10.60.0.0/16` (must between `/16` and `/28`), but we suggest using `/16` to `/18`, otherwise there will not be enough IP addresses.
  * The subnet CIDR will be calculated by VPC CIDR.
* GCP
  * Default VPC CIDR: `10.0.0.0/16`
  * Secondary ranges for pods and services:
    * Pods: `192.168.0.0/16`
    * Services: `192.168.64.0/18`
* Azure
  * Default VPC CIDR: `10.70.0.0/16`
  * Default Subnet CIDR: `10.70.0.0/24`

#### Bring You Own Network (BYON)

<span />

<Note title="BYOC Pro Feature">
  This feature is available in BYOC Pro. Please [contact us](https://streamnative.io/contact) if you are interested in BYOC Pro.
</Note>

If you decide to use your own VPC, please make sure that the VPC meets the following requirements:

<Tabs>
  <Tab title="AWS">
    * The VPC must be tagged with `Vendor=StreamNative` on both VPC and subnets.
    * For private subnets, tag them with `Type=private`.
    * For public subnets, tag them with `Type=public`.
  </Tab>

  <Tab title="GCP">
    No special tags are required.
  </Tab>

  <Tab title="Azure">
    No special tags are required.
  </Tab>
</Tabs>

If you don't tag the VPC and subnets with the required tags, the Cloud Environment provisioning will fail as we don't have the permission to access the VPC and subnets.

### Bring Your Own DNS

<span />

<Note title="BYOC Pro Feature">
  This feature is available in BYOC Pro. Please [contact us](https://streamnative.io/contact) if you are interested in BYOC Pro.
</Note>

By default, all the clusters will have a StreamNative-generated DNS record. If you want to use your own DNS, you need to provide the **DNS Zone ID** for a public hosted zone.

### Default Gateway

Select the endpoint type for the default gateway. A gateway exposes service endpoints externally and can be either **public** or **private**. If you select **private**, you must provide allowed IDs for creating privatelink services. The allowed IDs are the account IDs that you want to grant access to the private endpoints.

## Create a Cloud Environment

After you have planned all the required information, you can then provision a cloud environment through [Cloud Console](/cloud/get-started/cloud-console), [`snctl`](/tools/cli/snctl/snctl-overview), or StreamNative's Terraform provider.

<Tabs>
  <Tab title="Console">
    **Tutorial**

    You can watch the video of creating a cloud environment using Cloud Console UI:

    [![Create Cloud Environment](https://img.youtube.com/vi/ubRcgeOhHKw/0.jpg)](https://www.youtube.com/watch?v=ubRcgeOhHKw\&list=PL7-BmxsE3q4W5QnrusLyYt9_HbX4R7vEN\&index=3)

    **Step-by-step guide**

    1. In the upper-right corner of Cloud Console, click your user profile, and in the dropdown menu, click **Cloud Environments**.

    2. On the **Cloud Environments** page, click the **+ Create** button and select **Create environment**.

    3. On the **Cloud Connection** page, select the cloud connection you want to use, then click **Environment setup**.

    4. On the **Cloud Environment** page, fill out the required information based on what you have planned in the previous step.

       1. **Region**: Select the region where the Cloud Environment will be created.

       2. **Environment tag**: Select a tag for the Cloud Environment. This tag will be used for generating the name for the Cloud Environment.

       3. **VPC** \[BYOC Pro Feature]:

          * **StreamNative-managed VPC**: Provide the **Network CIDR** value for the VPC.

          * **Custom Network**: Provide the **Network Id** value of your VPC.

            * **AWS**: The **Network Id** is the VPC ID.

            * **GCP**: The **Network Id** is the Network Name.

            * **Azure**: The **Network Id** is the VNet Name.

       4. **Custom DNS** \[BYOC Pro Feature]:

          If you want to use your own DNS, check **Custom DNS** checkbox and provide the **DNS ID** and **DNS name** of your DNS zone.

    5. Select the CloudConnection, then click **Environment setup**.

       <img alt="Select CloudConnection" />

    6. Specify the region and other configurations, then click **Create**.

       <img alt="Create CloudEnvironment" />
  </Tab>

  <Tab title="snctl">
    **Tutorial**

    You can also watch the video of creating a cloud environment using `snctl`:

    [![Create Cloud Environment](https://img.youtube.com/vi/hBUeArMyV6c/0.jpg)](https://www.youtube.com/watch?v=ETr2EvKTht8\&list=PL7-BmxsE3q4W5QnrusLyYt9_HbX4R7vEN\&index=5)

    **Step-by-step guide**

    To create a Cloud Environment using snctl, use `snctl create cloudenvironment`. Usage:

    ```bash theme={null}
    snctl create cloudenvironment \
      --cloud-connection-name CLOUD_CONNECTION_NAME \
      --region REGION
    ```

    | Flag                          | Description                                                                                                                                                           |
    | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
    | --cloud-connection-name       | Required. The name of cloud connection name.                                                                                                                          |
    | --region                      | Required. The region of Cloud Environment. For Azure, it's the resource group name                                                                                    |
    | --zone                        | Optional. The zone of Cloud Environment. It will be zonal if this is configured.                                                                                      |
    | --network-cidr                | Optional. The network cidr of StreamNative-managed VPC. Cannot be specified if `network-id` is specified.                                                             |
    | --subnet-cidr                 | Optional. The subnet cidr of StreamNative-managed VPC. Only required for Azure environments when `cidr` is specified.                                                 |
    | --network-id                  | Optional. The network id of your existing VPC. Specify it when you want to use your own VPC. Cannot be specified if `cidr` is specified. **This is BYOC Pro feature** |
    | --dns-id                      | Optional. The dns id of your existing DNS zone. Specify it when you want to use your own DNS. **This is BYOC Pro feature**                                            |
    | --dns-name                    | Optional. The dns name of your existing DNS zone. Specify it when you want to use your own DNS. **This is BYOC Pro feature**                                          |
    | --default-gateway-access      | Optional. The access type of Pulsar endpoint. It can be `public` or `private`. Default to `public`.                                                                   |
    | --default-gateway-allowed-ids | Optional. The allowed ids of the default gateway, only can be set when `--default-gateway-access` is `private`.                                                       |
    | --environment-type            | Optional. The environment type, can be dev, test, staging, production, poc, qa, acc. Default to production.                                                           |

    **Basic Examples**

    <Tabs>
      <Tab title="AWS">
        ```bash theme={null}
        snctl create cloudenvionment --cloud-connection-name CLOUD_CONNECTION_NAME \
          --region REGION \
          -n ORG_NAME
        ```

        * `CLOUD_CONNECTION_NAME`: The name of the cloud connection.
        * `REGION`: The region where the Cloud Environment will be created.
        * `ORG_NAME`: The name of the organization.
      </Tab>

      <Tab title="GCP">
        ```bash theme={null}
        snctl create cloudenvironment --cloud-connection-name CLOUD_CONNECTION_NAME \
          --region REGION \
          -n ORG_NAME
        ```

        * `CLOUD_CONNECTION_NAME`: The name of the cloud connection.
        * `REGION`: The region where the Cloud Environment will be created.
        * `ORG_NAME`: The name of the organization.
      </Tab>

      <Tab title="Azure">
        ```bash theme={null}
        snctl create cloudenvionment --cloud-connection-name CLOUD_CONNECTION_NAME \
          --region REGION \
          -n ORG_NAME
        ```

        * `CLOUD_CONNECTION_NAME`: The name of the cloud connection.
        * `REGION`: The region where the Cloud Environment will be created.
        * `ORG_NAME`: The name of the organization.
      </Tab>
    </Tabs>

    **Cloud Environment with a Default Private Gateway Examples**

    Below are examples of creating a cloud environment with a default private gateway.

    <Tabs>
      <Tab title="AWS">
        ```bash theme={null}
        snctl create cloudenvionment \
          --cloud-connection-name CLOUD_CONNECTION_NAME \
          --region REGION \
          --default-gateway-access private \
          --default-gateway-allowed-ids AWS_ACCOUNT_ID_1,AWS_ACCOUNT_ID_2 \
          -n ORG_NAME
        ```

        * `CLOUD_CONNECTION_NAME`: The name of the cloud connection.
        * `REGION`: The region where the Cloud Environment will be created.
        * `ORG_NAME`: The name of the organization.
        * `AWS_ACCOUNT_ID_1,AWS_ACCOUNT_ID_2`: The AWS account IDs that are allowed to establish privatelink connections to the clusters.
      </Tab>

      <Tab title="GCP">
        ```bash theme={null}
        snctl create cloudenvionment \
          --cloud-connection-name CLOUD_CONNECTION_NAME \
          --region REGION \
          --default-gateway-access private \
          --default-gateway-allowed-ids GCP_PROJECT_ID_1,GCP_PROJECT_ID_2 \
          -n ORG_NAME
        ```

        * `CLOUD_CONNECTION_NAME`: The name of the cloud connection.
        * `REGION`: The region where the Cloud Environment will be created.
        * `ORG_NAME`: The name of the organization.
        * `GCP_PROJECT_ID_1,GCP_PROJECT_ID_2`: The GCP project IDs that are allowed to establish private service connect connections to the clusters.
      </Tab>

      <Tab title="Azure">
        ```bash theme={null}
        snctl create cloudenvionment \
          --cloud-connection-name CLOUD_CONNECTION_NAME \
          --region REGION \
          --default-gateway-access private \
          --default-gateway-allowed-ids AZURE_SUBSCRIPTION_ID_1,AZURE_SUBSCRIPTION_ID_2 \
          -n ORG_NAME
        ```

        * `CLOUD_CONNECTION_NAME`: The name of the cloud connection.
        * `REGION`: The region where the Cloud Environment will be created.
        * `ORG_NAME`: The name of the organization.
        * `AZURE_SUBSCRIPTION_ID_1,AZURE_SUBSCRIPTION_ID_2`: The Azure subscription IDs that are allowed to establish privatelink connections to the clusters.
      </Tab>
    </Tabs>

    **BYO-Network & BYO-DNS Examples**

    <Tabs>
      <Tab title="AWS">
        ```bash theme={null}
        snctl create cloudenvionment \
          --cloud-connection-name CLOUD_CONNECTION_NAME \
          --region REGION \
          --network-id NETWORK_ID \
          --dns-id DNS_ID \
          --dns-name DNS_NAME \
          -n ORG_NAME
        ```

        * `CLOUD_CONNECTION_NAME`: The name of the cloud connection.
        * `REGION`: The region where the Cloud Environment will be created.
        * `NETWORK_ID`: The ID of your existing VPC (i.e. `vpc-0123456789abcdef0`).
        * `DNS_ID`: The ID of your existing DNS zone (i.e. `Z08392741KXNBH5WMR9PQ`).
        * `DNS_NAME`: The name of your existing DNS zone (i.e. `byod.aws.example.com`).
        * `ORG_NAME`: The name of the organization.
      </Tab>

      <Tab title="GCP">
        ```bash theme={null}
        snctl create cloudenvionment \
          --cloud-connection-name CLOUD_CONNECTION_NAME \
          --region REGION \
          --network-id NETWORK_ID \
          --dns-id DNS_ID \
          --dns-name DNS_NAME \
          -n ORG_NAME
        ```

        * `CLOUD_CONNECTION_NAME`: The name of the cloud connection.
        * `REGION`: The region where the Cloud Environment will be created.
        * `NETWORK_ID`: The name of your existing VPC network (i.e. `my-vpc-network`).
        * `DNS_ID`: The name of your existing DNS zone (i.e. `my-dns-zone`).
        * `DNS_NAME`: The name of your existing DNS zone (i.e. `byod.gcp.example.com`).
        * `ORG_NAME`: The name of the organization.
      </Tab>

      <Tab title="Azure">
        ```bash theme={null}
        snctl create cloudenvionment \
          --cloud-connection-name CLOUD_CONNECTION_NAME \
          --region REGION \
          --network-id NETWORK_ID \
          --dns-id DNS_ID \
          --dns-name DNS_NAME \
          -n ORG_NAME
        ```

        * `CLOUD_CONNECTION_NAME`: The name of the cloud connection.
        * `REGION`: The region where the Cloud Environment will be created.
        * `NETWORK_ID`: The ID of your existing VPC (i.e. `rg-eastus-vnet`).
        * `DNS_ID`: The ID of your existing DNS zone (i.e. `/subscriptions/947a12b8-3649-8271-945c-8912f534a19d/resourceGroups/eastus/providers/Microsoft.Network/dnszones/byod.test.azure.example.com`).
        * `DNS_NAME`: The name of your existing DNS zone (i.e. `byod.test.azure.example.com`).
        * `ORG_NAME`: The name of the organization.
      </Tab>
    </Tabs>

    **Manifest file**

    Alternatively, you can prepare a manifest file `cloudenvironment.yaml` to define a cloud environment, and then use `snctl` to create the cloud environment:

    ```yaml theme={null}
    apiVersion: cloud.streamnative.io/v1alpha1
    kind: CloudEnvironment
    metadata:
      namespace: ORG_NAME
    spec:
      cloudConnectionName: CLOUD_CONNECTION_NAME
      defaultGateway:
        access: public
      dns:
        id: DNS_ID
        name: DNS_NAME
      network:
        id: NETWORK_ID
      region: REGION
    ```

    * Replace `CLOUD_CONNECTION_NAME` with the name of the cloud connection.
    * Replace `REGION` with the region where the Cloud Environment will be created.
    * Replace `ORG_NAME` with the name of the organization.
    * Replace `DNS_ID` with the ID of your existing DNS zone (i.e. `Z08392741KXNBH5WMR9PQ`).
    * Replace `DNS_NAME` with the name of your existing DNS zone (i.e. `byod.aws.example.com`).
    * Replace `NETWORK_ID` with the ID of your existing VPC (i.e. `vpc-0123456789abcdef0`).

    Then you can create the cloud environment by running `snctl create -f cloudenvironment.yaml`.

    The name of the cloud environment will be generated automatically based on the cloud connection name, region, and environment tag. You can use `snctl get cloudenvironment` to find the name of the cloud environment you just created.

    Then you can use `snctl get cloudenvironment <name>` to get the details of the cloud environment. If a Cloud Environment is successfully provisioned, all the conditions should be `True`.
  </Tab>

  <Tab title="Terraform">
    **Tutorial**

    You can also watch the video of creating a cloud environment using Terraform:

    [![Create Cloud Environment using Terraform](https://img.youtube.com/vi/9h2_1AGy-I4/0.jpg)](https://www.youtube.com/watch?v=J7S7A_1Tshc\&list=PL7-BmxsE3q4W5QnrusLyYt9_HbX4R7vEN\&index=4)

    **Step-by-step guide**

    To create a Cloud Environment with terraform:

    1. Prepare `main.tf` to define the cloud environment.
    2. Run `terraform init` to initialize the terraform project.
    3. Run `terraform plan` to review the changes.
    4. Run `terraform apply` to create the cloud environment.

    For additional details, please refer to our [Terraform module documentation on Cloud Environments](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/resources/cloud_environment).

    **Cloud Environment Schema**

    | Field                             | Type                                               | Description                                                            |
    | --------------------------------- | -------------------------------------------------- | ---------------------------------------------------------------------- |
    | organization, required            | String                                             | The organization name                                                  |
    | cloud\_connection\_name, required | String                                             | Name of the cloud connection                                           |
    | region, required                  | String                                             | The cloud region in which this environment will be created             |
    | zone, optional                    | String                                             | The zone of Cloud Environment. It will be zonal if this is configured. |
    | network                           | List of Object                                     | see 'network schema' below for nested schema                           |
    | network schema                    | cidr (String), id (String)                         | cidr and id cannot be specified together                               |
    | default\_gateway, optional        | List of Object                                     | see 'default\_gateway schema' below for nested schema                  |
    | default\_gateway schema           | access (String), private\_service (List of Object) | see 'private\_service schema' for nested schema                        |
    | private\_service schema           | allowed\_ids (List of String)                      |                                                                        |
    | dns, optional                     | List of Object                                     | see 'dns schema' below for nested schema                               |
    | dns schema                        | id (String), name (String)                         | The ID and name of an existing DNS zone                                |
    | environment\_tag, optional        | String                                             | Tag to identify the environment (e.g. "production", "staging")         |
    | timeouts                          | List of Object                                     | see 'timeouts schema' below for nested schema                          |
    | timeouts schema                   | create (String), delete (String), update (String)  | Timeouts for create/delete/update operations                           |

    **Basic Examples**

    <Tabs>
      <Tab title="AWS">
        ```json theme={null}
        module "sn_managed_cloud" {
          source = "github.com/streamnative/terraform-managed-cloud//modules/aws/vendor-access?ref=v3.23.0"

          external_id = "YOUR_ORG_ID"
        }

        resource "streamnative_cloud_connection" "shared_aws" {
            depends_on = [ module.sn_managed_cloud ]
            organization = "YOUR_ORG_ID"
            name = "YOUR_CLOUD_CONNECTION_NAME"

            type = "aws"
            aws {
                account_id = "YOUR_AWS_ACCOUNT_ID"
            }
        }

        resource "streamnative_cloud_environment" "aws_usw1_production" {
            depends_on = [ streamnative_cloud_connection.shared_aws ]
            organization = "YOUR_ORG_ID"
            region = "YOUR_REGION"
            cloud_connection_name = "YOUR_CLOUD_CONNECTION_NAME"

            network {
              cidr = "YOUR_NETWORK_CIDR"
            }
        }
        ```

        * Replace `YOUR_ORG_ID` with the name of the organization.
        * Replace `YOUR_REGION` with the region where the Cloud Environment will be created.
        * Replace `YOUR_CLOUD_CONNECTION_NAME` with the name of the cloud connection.
        * Replace `YOUR_NETWORK_CIDR` with the CIDR value for the VPC (i.e. `10.60.0.0/16`).
      </Tab>

      <Tab title="GCP">
        ```json theme={null}
        provider "google" {
          project = "YOUR_GCP_PROJECT_ID"
        }

        module "sn_managed_cloud" {
          source = "github.com/streamnative/terraform-managed-cloud//modules/gcp/vendor-access?ref=v3.23.0"
          project = "YOUR_GCP_PROJECT_ID"
          streamnative_org_id = "YOUR_ORG_ID"
        }

        resource "streamnative_cloud_connection" "shared_gcp" {
            depends_on = [ module.sn_managed_cloud ]
            organization = "YOUR_ORG_ID"
            name = "YOUR_CLOUD_CONNECTION_NAME"

            type = "gcp"
            gcp {
                project_id = "YOUR_GCP_PROJECT_ID"
            }
        }

        resource "streamnative_cloud_environment" "gcp_usw1_production" {
            depends_on = [ streamnative_cloud_connection.shared_gcp ]
            organization = "orgname"
            region = "YOUR_REGION"
            cloud_connection_name = "YOUR_CLOUD_CONNECTION_NAME"

            network {
              cidr = "YOUR_NETWORK_CIDR"
            }
        }
        ```

        * Replace `YOUR_ORG_ID` with the name of the organization.
        * Replace `YOUR_REGION` with the region where the Cloud Environment will be created (i.e. `us-west1`).
        * Replace `YOUR_CLOUD_CONNECTION_NAME` with the name of the cloud connection.
        * Replace `YOUR_NETWORK_CIDR` with the CIDR value for the VPC (i.e. `10.0.0.0/16`).
      </Tab>

      <Tab title="Azure">
        ```json theme={null}
        resource "streamnative_cloud_environment" "azure_eastus_production" {
            organization = "YOUR_ORG_ID"
            region = "YOUR_REGION"
            cloud_connection_name = "YOUR_CLOUD_CONNECTION_NAME"

            network {
              cidr = "YOUR_NETWORK_CIDR"
              subnet_cidr = "YOUR_SUBNET_CIDR"
            }
        }
        ```

        * Replace `YOUR_ORG_ID` with the name of the organization.
        * Replace `YOUR_REGION` with the region where the Cloud Environment will be created (i.e. `rg-eastus`).
        * Replace `YOUR_CLOUD_CONNECTION_NAME` with the name of the cloud connection.
        * Replace `YOUR_NETWORK_CIDR` with the CIDR value for the VPC (i.e. `10.70.0.0/16`).
        * Replace `YOUR_SUBNET_CIDR` with the CIDR value for the subnet (i.e. `10.70.0.0/24`).
      </Tab>
    </Tabs>

    **BYO-Network & BYO-DNS Examples**

    This example shows how to create a cloud environment using your own network and DNS.

    <Tabs>
      <Tab title="AWS">
        ```hcl theme={null}
        terraform {
          required_providers {
            streamnative = {
              source = "streamnative/streamnative"
              version = "v0.7.0"
            }
          }
        }

        resource "streamnative_cloud_environment" "example" {
          organization          = "YOUR_ORG_ID"
          region                = "YOUR_REGION"
          cloud_connection_name = "YOUR_CLOUD_CONNECTION_NAME"
          environment_type      = "YOUR_ENVIRONMENT_TYPE"

          network {
            id = "YOUR_NETWORK_ID"
          }

          dns {
            id   = "YOUR_DNS_ID"
            name = "YOUR_DNS_NAME"
          }
        }
        ```

        * Replace `YOUR_ORG_ID` with the name of the organization.
        * Replace `YOUR_REGION` with the region where the Cloud Environment will be created.
        * Replace `YOUR_CLOUD_CONNECTION_NAME` with the name of the cloud connection.
        * Replace `YOUR_ENVIRONMENT_TYPE` with the type of the environment.
        * Replace `YOUR_NETWORK_ID` with the ID of your existing VPC.
        * Replace `YOUR_DNS_ID` with the ID of your existing DNS zone.
        * Replace `YOUR_DNS_NAME` with the name of your existing DNS zone.
      </Tab>
    </Tabs>
  </Tab>
</Tabs>

## Monitoring the Provisioning Process

Creating a Cloud Environment through Terraform does not immediately create the Cloud Environment, but rather kicks off a process that creates it. Provisioning a Cloud Environment takes approximately 40 minutes. You will receive an email notification when the Cloud Environment is ready or if there are any issues.

At the same time, if you want to monitor the provisioning process, you can use `snctl`.

```bash theme={null}
snctl describe -O orgname cloudenvironment CLOUD_ENVIRONMENT_NAME
```

* Replace `CLOUD_ENVIRONMENT_NAME` with the name of the cloud environment.

Once `snctl describe` returns a `status` of `True` and a `type` of `Ready` your Cloud Environment has been provisioned, and you can create a Pulsar Cluster within it.

If this command returns an error state, you can try to gather more detailed information about the error with (the `-o yaml` flag outputs the full resource details in YAML format):

```bash theme={null}
snctl get -o yaml -O orgname cloudenvironment CLOUD_ENVIRONMENT_NAME
```

If the error persists please [reach out to the StreamNative support team](https://support.streamnative.io/hc/en-us/requests/new).

## Update a Cloud Environment

It is generally not recommended to update a Cloud Environment once it has been created. If you need to update a Cloud Environment, please reach out to [StreamNative support team](https://support.streamnative.io/hc/en-us/requests/new) to discuss your requirements.

## Delete a Cloud Environment

<Warning title="Warning">
  Deleting a Cloud Environment is an irreversible action. Please exercise caution when performing this operation.
</Warning>

<Tabs>
  <Tab title="Console">
    1. In the upper-right corner of Cloud Console, click your user profile, and in the dropdown menu, click **Cloud Environments**.

    2. On the **Cloud Environments** page, find the cloud environment you want to delete, and click the ellipsis (**...**) on the right side of the row, and then click **Delete**.

    3. On the **Delete cloud environment** page, enter the name of the cloud environment, and click **Confirm**.
  </Tab>

  <Tab title="snctl">
    You can delete a cloud environment using `snctl`:

    ```bash theme={null}
    snctl delete cloudenvironment CLOUD_ENVIRONMENT_NAME
    ```

    * Replace `CLOUD_ENVIRONMENT_NAME` with the name of the cloud environment.

    Alternatively, if you have the manifest file of the cloud environment, you can delete the cloud environment by running:

    ```bash theme={null}
    snctl delete -f cloudenvironment.yaml
    ```
  </Tab>

  <Tab title="Terraform">
    You can remove the cloud environment from your terraform code and run `terraform apply` to delete the cloud environment.
  </Tab>
</Tabs>

## Next steps

After setting up one or more Cloud Environments, you can proceed to create StreamNative [instances](/cloud/clusters/manage-instances/instance) and [clusters](/cloud/clusters/manage-clusters/cluster) within those environments.


# Set up Account Access for BYOC on Alibaba Cloud
Source: https://docs.streamnative.io/cloud/clusters/byoc/grant-vendor-access/byoc-on-alibaba/byoc-alibaba-cloud-access



Before you can provision a BYOC [infrastructure pool](/cloud/clusters/streamnative-cluster-overview#infrastructure-pools), you will need to authorize the StreamNative Cloud control plane to access your cloud accounts. This enables StreamNative to provision and manage clusters within your Alibaba Cloud account.

This document describes how to grant such access to StreamNative Cloud for an Alibaba Cloud account.

## Access model in Alibaba Cloud

StreamNative leverages advanced RAM features in Alibaba Cloud to ensure minimal and precise access, allowing for efficient management of only necessary resources:

* **Bootstrap/Provisioning Role**: This role handles the provisioning and maintenance of the underlying infrastructure like VPCs, ACK clusters (and associated node groups, and so on), RAM resources, and is also utilized for troubleshooting during incidents by StreamNative's SRE team. This role is also for automated management tasks and interacts with the ACK cluster to deploy and manage Pulsar clusters
* **Support Role:** This role is used by the StreamNative SRE and Support team for troubleshooting during incidents.

Both roles use a same permission policy that allows StreamNative’s cloud manager role to assume these roles within the customer’s account. These roles are safeguarded using several Alibaba Cloud security features:

* An external ID for role assumption, enhancing security when third parties access your Alibaba account (See [Use external IDs to prevent the confused deputy issue](https://www.alibabacloud.com/help/en/ram/use-cases/use-externalid-to-prevent-the-confused-deputy-problem).
* Tag-based access, through the `Vendor: StreamNative` tag, is used where applicable to enforce resources that are created with these tags and access is limited to only resources with the tag (See [Use tags to control access to resources](https://www.alibabacloud.com/help/en/resource-management/tag/user-guide/use-tags-to-control-access-to-resources-1)).
* All RAM policies are statically created by the customer (via [StreamNative Vendor Access](https://github.com/streamnative/terraform-managed-cloud) Terraform module) to limit access.

## Provision Alibaba Cloud Access

StreamNative facilitates the setup of necessary policies and roles through a <Tooltip>Terraform</Tooltip> module. This module can be provisioned in a standalone Terraform project (as documented here), but can also be integrated into existing Terraform projects.

For full documentation of inputs and outputs of the Terraform module, see the [module's README on GitHub](https://github.com/streamnative/terraform-managed-cloud/tree/main/modules/alicloud).

### Prerequisites

* New to Terraform? Learn the [Terraform Alibaba Cloud Getting Started Tutorial](https://registry.terraform.io/providers/aliyun/alicloud/latest/docs) to get a basic introduction.
* Install Terraform, version 1.3.0 or greater.
* Ensure you have created an organization through the StreamNative Cloud Console

If you run into issues, please contact [StreamNative Support team](https://support.streamnative.io/hc/en-us/requests/new).

### Step 1: Create a new project and instantiate the module

Terraform works by having Terraform codes (in the form of `*.tf` files) and state files that represent the current resources. If you are using Terraform locally, without a [remote state store](https://developer.hashicorp.com/terraform/language/state/remote), these files should be checked into source control for future updates. Create a new folder and add a file called `main.tf` with the following content, replacing the referenced variables.

```hcl theme={null}
provider "alicloud" {
  region = "ap-southeast-1"
}

module "vendor_access" {
  source          = "github.com/streamnative/terraform-managed-cloud//modules/alicloud/vendor-access?ref=v3.23.0"
  organization_ids = ["<YOUR_SNCLOUD_ORG_ID>"]
}
```

* `<YOUR_SNCLOUD_ORG_ID>`: your StreamNative Cloud organization ID. This is typically an ID like `o-xxxxx`. This can be found in your organization list or the top header of the application. If you have multiple organizations, you can put multiple organization id in this list

If you are using `git` as source control, you need to use the `git init` command to initialize this folder as a git project.

### Step 2: Initialize the Terraform

While the above Terraform code is all needed, the module needs to be downloaded to this Terraform project.

To do so, run `terraform init`.

This will download the module and required dependencies.

### Step 3: Create a shell with the correct Alibaba Cloud credentials

Terraform requires Alibaba Cloud credentials with the proper permissions in the target account to create the resources to grant access. The permissions required by the module are all Alibaba Cloud RAM permissions, specifically to managed roles, policies, and attachments. The Alibaba Cloud Managed Access policies of `AliyunRAMFullAccess` are sufficient to perform these operations.

All of the [Terraform Alibaba Cloud credentials mechanisms](https://registry.terraform.io/providers/aliyun/alicloud/latest/docs#authentication) are compatible with the Terraform module.

If you are new to Terraform and Alibaba Cloud, the following steps will provide credentials in your shell:

1. Follow the steps to create an [access key and secret](https://www.alibabacloud.com/help/en/ram/user-guide/create-an-accesskey-pair) for your user.
2. Set the `ALIBABA_CLOUD_ACCESS_KEY_ID`, `ALIBABA_CLOUD_ACCESS_KEY_SECRET`, and `ALIBABA_CLOUD_REGION` environment variables from the generated credentials.

### Step 4: Run the Terraform

After initialization, and with credentials in the shell, the next step is to run the Terraform with `terraform apply`.

This will create a Terraform plan which shows all the resources to be created.

### Step 5: Annotate the account ID to be used by StreamNative

Once completed, please note the account ID of the Alibaba Cloud account you have granted access to StreamNative Cloud. You will use this account ID to create a Cloud Connection.

## Next steps

After granting access, you can set up a Cloud Connection to allow the StreamNative Cloud control plane to interact with your Alibaba Cloud account.


# Set up Account Access for BYOC on AWS
Source: https://docs.streamnative.io/cloud/clusters/byoc/grant-vendor-access/byoc-on-aws/byoc-aws-access



Before you can provision a BYOC [infrastructure pool](/cloud/clusters/streamnative-cluster-overview#infrastructure-pools), you will need to authorize the StreamNative Cloud control plane to access your cloud accounts. This enables StreamNative to provision and manage clusters within your AWS account.

This document describes how to grant such access to StreamNative Cloud for an AWS account.

## Access model in AWS

StreamNative leverages advanced IAM features in AWS to ensure minimal and precise access, allowing for efficient management of only necessary resources. Best practices in AWS are adhered to, enhancing security. Access to customer accounts is segmented into two IAM roles:

* **Bootstrap/Provisioning Role**: This role handles the provisioning and maintenance of the underlying infrastructure like VPCs, EKS clusters (and associated node groups, and so on), IAM resources, and is also utilized for troubleshooting during incidents by StreamNative's SRE team.
* **Management Role**: This role is primarily for automated management tasks, having minimal permissions, mostly read-only, and also interacts with the EKS cluster to deploy and manage Pulsar clusters.

Both roles use a same permission policy that allows StreamNative’s cloud manager role to assume these roles within the customer’s account. These roles are safeguarded using several AWS security features:

* An external ID for role assumption, enhancing security when third parties access your AWS account (See [AWS's documentation on using third-party access](https://aws.amazon.com/blogs/apn/securely-using-external-id-for-accessing-aws-accounts-owned-by-others/).
* A permission boundary to restrict the permissions of dynamically created roles (refer to [AWS's documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_boundaries.html)).
* Policies are limited only to the required AWS services, actions, and resources within those services wherever possible.
* Tag-based access, through the `Vendor: StreamNative` tag, is used where applicable to enforce resources that are created with these tags and access is limited to only resources with the tag (See [AWS docs on tag-based access control](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_tags.html)).
* All IAM policies are statically created by the customer (via [StreamNative Vendor Access](https://github.com/streamnative/terraform-managed-cloud) Terraform module) to limit access.

These roles and policies can be implemented using the [StreamNative Vendor Access](https://github.com/streamnative/terraform-managed-cloud) Terraform module. The following diagram illustrates AWS access:

<img alt="image of BYOC AWS Access" />

## Provision AWS Access

StreamNative facilitates the setup of necessary policies and roles through a <Tooltip>Terraform</Tooltip> module. This module can be provisioned in a standalone Terraform project (as documented here), but can also be integrated into existing Terraform projects.

For full documentation of inputs and outputs of the Terraform module, see the [module's README on GitHub](https://github.com/streamnative/terraform-managed-cloud/tree/main/modules/aws).

You can also watch the video of provisioning AWS access:

[![Provide BYOC Permissions using Terraform](https://img.youtube.com/vi/HbRRzGP8zMg/0.jpg)](https://www.youtube.com/watch?v=HbRRzGP8zMg\&list=PL7-BmxsE3q4W5QnrusLyYt9_HbX4R7vEN\&index=2)

### Prerequisites

* New to Terraform? Learn the [Terraform AWS Getting Started Tutorial](https://developer.hashicorp.com/terraform/tutorials/aws-get-started) to get a basic introduction.
* Install Terraform, version 1.3.0 or greater.
* Ensure you have [created an organization](/cloud/security/access/resource-hierarchy/organizations#create-an-organization) through the StreamNative Cloud Console.
* If you want to use a [custom domain](/cloud/clusters/byoc/grant-vendor-access/byoc-on-aws/byoc-aws-dns-domain#custom-domains) (BYOC Pro only), you need your route53 zone ID.

If you run into issues, please contact [StreamNative Support team](https://support.streamnative.io/hc/en-us/requests/new).

### Step 1: Create a new project and instantiate the module

Terraform works by having Terraform codes (in the form of `*.tf` files) and state files that represent the current resources. If you are using Terraform locally, without a [remote state store](https://developer.hashicorp.com/terraform/language/state/remote), these files should be checked into source control for future updates.
Create a new folder and add a file called `main.tf` with the following content, replacing the referenced variables.

```hcl theme={null}
module "sn_managed_cloud" {
  source = "github.com/streamnative/terraform-managed-cloud//modules/aws/vendor-access?ref=v3.23.0"

  external_id = "<YOUR_SNCLOUD_ORG_ID>"
}
```

* `<YOUR_SNCLOUD_ORG_ID>`: your StreamNative Cloud organization ID. This is typically an ID like `o-xxxxx`. This can be found in your organization list or the top header of the application.

If you are using `git` as source control, you need to use the `git init` command to initialize this folder as a git project.

### Step 2: Initialize the Terraform

While the above Terraform code is all needed, the module needs to be downloaded to this Terraform project.

To do so, run `terraform init`.

This will download the module and required dependencies.

### Step 3: Create a shell with the correct AWS credentials

Terraform requires AWS credentials with the proper permissions in the target account to create the resources to grant access. The permissions required by the module are all AWS IAM permissions, specifically to managed roles, policies, and attachments. The AWS Managed Access policies of `IAMFullAccess` are sufficient to perform these operations.

All of the [Terraform AWS credentials mechanisms](https://registry.terraform.io/providers/hashicorp/aws/latest/docs#authentication-and-configuration) are compatible with the Terraform module.

If you are new to Terraform and AWS, the following steps will provide credentials in your shell:

1. Follow the steps to create an [access key and secret](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) for your user.
2. Set the `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_REGION` environment variables from the generated credentials.

### Step 4: Run the Terraform

After initialization, and with credentials in the shell, the next step is to run the Terraform with `terraform apply`.

This will create a Terraform plan which shows all the resources to be created.

### Step 5: Annotate the account ID to be used by StreamNative

Once completed, please note the account ID of the AWS account you have granted access to StreamNative Cloud. You will use this account ID to create a [Cloud Connection](/cloud/clusters/byoc/create-cloud-connection).

## Next steps

After granting access, you can [set up a Cloud Connection](/cloud/clusters/byoc/create-cloud-connection) to allow the StreamNative Cloud control plane to interact with your AWS account.


# Set up Custom Domain for BYOC on AWS
Source: https://docs.streamnative.io/cloud/clusters/byoc/grant-vendor-access/byoc-on-aws/byoc-aws-dns-domain



This document outlines the DNS configuration options available in StreamNative BYOC Pro, including how to set up a custom domain for use in a BYOC (Bring Your Own Cloud) deployment.

<Note title="BYOC Pro Feature">
  Custom domains are available in BYOC Pro. Please [contact us](https://streamnative.io/contact) if you are interested in BYOC Pro.
</Note>

## Domain options

StreamNative BYOC Pro offers two domain configurations: *Custom Domains* and *StreamNative Managed Domains*.

* **Custom Domains**: You delegate a DNS zone to the AWS account used for BYOC deployment. All service endpoints are created under this domain.
* **StreamNative Managed Domains**: StreamNative uses a domain within the `snio.cloud` domain, assigning a subzone specific to your cloud account.

For both types of domains, StreamNative utilizes the Kubernetes services like [external-dns](https://github.com/kubernetes-sigs/external-dns) and [cert-manager](https://cert-manager.io/) to managed DNS settings and TLS certificates.

Below is a diagram illustrating the domain options in BYOC:

<img alt="image of BYOC AWS DNS" />

<Tip title="Tip">
  * The diagram uses `sncloud.<custom_domain>` as an example of a custom domain.
  * You decide the domain name to delegate to StreamNative.
</Tip>

### Custom domains

With a custom domain, DNS names and TLS certificates for StreamNative clusters reside under a domain that you control. StreamNative requires a dedicated AWS Route53 zone for these domains to ensure that only necessary DNS records are modified. This allows AWS policies to be restricted specifically to this zone. This zone needs to be public to enable the generation of TLS certificates from Let's Encrypt.

### StreamNative managed domains

For managed domains, StreamNative sets up and manages a Route53 zone in your AWS account under the `snio.cloud` domain, based on your organization ID (e.g., `o-12345.snio.cloud`). StreamNative then delegates management of this subdomain.

<Note title="Note">
  Due to AWS IAM limitations, when using StreamNative-managed domains, the [AWS Access Module](/cloud/clusters/byoc/grant-vendor-access/byoc-on-aws/byoc-aws-access) must first be provisioned with the route53 zone argument set as a wildcard (`*`). Once the zone is created and its ID is known, it can be restricted to that specific zone ID.
</Note>

## Create a zone for a custom domain

StreamNative requires a dedicated Route53 zone. This section explains how to delegate a subzone from an existing domain, assuming you are using Route53 as your DNS provider.

For details about how to delegate a subzone, see your DNS provider docs.

### Create a subzone via delegation

The typical approach is to create a subdomain and delegate it from the parent domain.

* If your parent domain is on Route53, refer to the [Route53 documentation](https://aws.amazon.com/premiumsupport/knowledge-center/create-subdomain-route-53/).
* If your parent domain is **not** on Route53, follow the general instructions in this [AWS document](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/CreatingNewSubdomain.html).

### Register a new domain

If you prefer to register a new domain through Route53, consult the [AWS documentation](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/domain-register.html).

### Validate the zone

After creating and delegating the zone, it's crucial to validate it.

Command line tools, such as `dig`, allow for directly querying DNS records and ensuring that the `NS` records have propagated.

This example assumes that you have created a subzone `sncloud.myco.com`, and that had NS records from route53 of:

* ns-1654.awsdns-14.co.uk.
* ns-1513.awsdns-61.org.
* ns-449.awsdns-56.com.
* ns-755.awsdns-30.net.

After creating the NS record from the parent domain, the `dig NS sncloud.myco.com` should return a response like:

```
$ dig NS sncloud.myco.com

; <<>> DiG 9.18.1-1ubuntu1.3-Ubuntu <<>> NS sncloud.myco.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41380
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 9

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;sncloud.myco.com.		IN	NS

;; ANSWER SECTION:
sncloud.myco.com.	172800	IN	NS	ns-449.awsdns-56.com.
sncloud.myco.com.	172800	IN	NS	ns-755.awsdns-30.net.
sncloud.myco.com.	172800	IN	NS	ns-1513.awsdns-61.org.
sncloud.myco.com.	172800	IN	NS	ns-1654.awsdns-14.co.uk.

;; Query time: 251 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Fri Feb 10 15:47:36 MST 2023
;; MSG SIZE  rcvd: 367
```

You can see that the `ANSWER SECTION` has 4 records that match the 4 NS records above.

Another option is to use a tool like [DNS NS Lookup Checker](https://dnschecker.org/ns-lookup.php) to lookup records. However, this may take longer to validate.

### Provide the zone ID

After you complete the validation, you can use the zone ID to configure `hosted_zone_allowed_ids` in [StreamNative's Vendor Access Module](/cloud/clusters/byoc/grant-vendor-access/byoc-on-aws/byoc-aws-access). The format of the ID should be similar to `ZXXXXXXXXXXXXXXXXXXXXX` (the letter `Z` followed by 21 digits or uppercase letters).


# Set up Account Access for BYOC on Azure
Source: https://docs.streamnative.io/cloud/clusters/byoc/grant-vendor-access/byoc-on-azure/byoc-azure-access



<Note title="Note">
  This feature is currently in private preview. If you want to try it out or have any questions, [submit a ticket](https://support.streamnative.io/hc/en-us/requests/new) to the support team.
</Note>

Before you can provision a BYOC [infrastructure pool](/cloud/clusters/streamnative-cluster-overview#infrastructure-pools), you will need to authorize the StreamNative Cloud control plane to access your cloud accounts. This enables StreamNative to provision and manage clusters within your Azure account.

This document outlines the procedure for granting this access to StreamNative Cloud for Azure accounts and subscriptions.

## Access model in Azure

StreamNative Cloud leverages Azure's user-assigned managed identities to minimize access while managing only the required resources effectively. The service adheres to Azure best practices to ensure robust security. Access is divided into two specific user-assigned managed identities:

* **Bootstrap/Provisioning identity**: This identity handles provisioning and maintaining core infrastructure components like VPCs, AKS clusters (and associated node groups, and so on), IAM resources, and more. It is configured with a federated identity credential linked to StreamNative's cloud-manager GSA.
* **Supporting identity**: Used by the StreamNative SRE team for troubleshooting during incidents, this identity has minimal permissions limited to the Azure resource group (primarily read-only). It is also configured with a federated identity credential connected to StreamNative's cloud-support GSAs.

These identities utilize custom IAM policies that leverage several Azure features to enhance security and restrict access:

* A resource group is used to limit the scope of the managed identity to only the resources within the resource group.
* A resource group is used for the AKS and other resources for the BYOC cluster.
* An orgnazation specificed audience is used when exchange the GSA token to the customer identity. See Azure's documentation on [Overview of federated identity credentials in Microsoft Entra ID](https://learn.microsoft.com/en-us/graph/api/resources/federatedidentitycredentials-overview).
* Different permissions are used for different identities to ensure that the least privilege principle is followed. The following permissions are used for the identities:
  * Contributor role limited to scope of the AKS resource group for the Bootstrap/Provisioning identity.
  * Azure Kubernetes Service Cluster Admin Role limited to scope of the AKS resource group for the Bootstrap/Provisioning identity.
  * Role Based Access Control Administrator role limited to scope of the AKS resource group for the Bootstrap/Provisioning identity with conditional role assignment, which limited to the following Azure roles:
    * [Storage Account Backup Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-account-backup-contributor)
    * [Network Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#network-contributor)
    * [DNS Zone Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#dns-zone-contributor)
    * [Reader](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#reader)
    * [Storage Blob Data Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-contributor)
    * [Azure Kubernetes Service Cluster Admin Role](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#azure-kubernetes-service-cluster-admin-role)
    * [Azure Kubernetes Service Cluster User Role](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#azure-kubernetes-service-cluster-user-role)
    * [Custom Velero User Role](https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure?tab=readme-ov-file#specify-role)
  * Azure Kubernetes Service Cluster User Role limited to scope of the AKS resource group for the Supporting identity.
* All resource groups and IAM permissions are statically created by the customer (via automation) to limit access.

You can provision these resource groups and identities using StreamNative-provided automation. Currently, this automation is in the form of a Terraform module.

<img alt="Storage bucket in Google Cloud" />

## Provision Azure Access

StreamNative provides all the resource groups and identities via a <Tooltip>Terraform</Tooltip> module. This module can be provisioned in a standalone Terraform project (as documented here), but can also be integrated into existing Terraform projects.

For full documentation of inputs and outputs of the Terraform module, see the [module's README on GitHub](https://github.com/streamnative/terraform-managed-cloud/tree/main/modules/azure).

### Prerequisites

* New to Terraform? Learn the [Terraform Azure Getting Started Tutorial](https://developer.hashicorp.com/terraform/tutorials/azure-get-started) to get a basic introduction.
* Install Terraform, version 1.3.0 or greater.
* Ensure you have [created an organization](/cloud/security/access/resource-hierarchy/organizations#create-an-organization) through the StreamNative Cloud Console.

If you run into issues, please contact [StreamNative Support team](https://support.streamnative.io/hc/en-us/requests/new).

### Step 1: Create a new project and instantiate the module

Terraform works by having Terraform codes (in the form of `*.tf` files) and state files that represent the current resources. If you are using Terraform locally, without a [remote state store](https://developer.hashicorp.com/terraform/language/state/remote), these files should be checked into source control for future updates.
Create a new folder and add a file called `main.tf` with the following content, replacing the referenced variables.

```hcl theme={null}
provider "azurerm" {
  features {
  }
}

provider "azuread" {}

module "azure-sn-cloud-manager" {
  source = "github.com/streamnative/terraform-managed-cloud//modules/azure/sn-cloud-manager?ref=v3.23.0"

  streamnative_cloud_env = "production"
  resource_group_location = "<ANY_AZURE_REGION>"
  streamnative_org_id = "<YOUR_SNCLOUD_ORG_ID>"
}

output "subscription_id" {
  value       = module.azure-sn-cloud-manager.subscription_id
  description = "The subscription ID of the AKS cluster"
}

output "tenant_id" {
  value       = module.azure-sn-cloud-manager.tenant_id
  description = "The tenant ID of the AKS cluster"
}

output "client_id" {
  value       = module.azure-sn-cloud-manager.sn_automation_client_id
  description = "The client ID of the sn automation service principal for StreamNative Cloud automation"
}

output "support_client_id" {
  value       = module.azure-sn-cloud-manager.sn_support_client_id
  description = "The client ID of the sn support service principal for StreamNative Cloud support access"
}
```

* `<YOUR_SNCLOUD_ORG_ID>`: your StreamNative Cloud organization ID. This is typically an ID like `o-xxxxx`. This can be found in your organization list or the top header of the application.
* `<ANY_AZURE_REGION>`: any valid Azure region, this region doesn't hold any resources but just some managed identities, so it can be anywhere, such as `eastus` or `westus2`.

If you are using `git` as source control, you need to use the `git init` command to initialize this folder as a git project.

### Step 2: Initialize the Terraform

While the above Terraform code is all needed, the module needs to be downloaded to this Terraform project.

To do so, run `terraform init`.

This will download the module and required dependencies.

### Step 3: Create a shell with the correct Azure credentials

Terraform requires Azure credentials with the proper permissions in the target account to create the resources to grant access. The permissions required by the module are all Azure subscription permissions, specifically to manage the resource groups, manage the custom roles, and manage the user-assigned managed identities. The Azure built-in role of `Contributor` to the Azure subscription are sufficient to perform these operations.

All of the methods in [Authenticating to Azure](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs#authenticating-to-azure) are compatible with the Terraform module. The most common method is to use the `az` CLI to log in and set the credentials in the shell. You can check the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/) for detailed instructions.

### Step 4: Run the Terraform

After initialization, and with credentials in the shell, the next step is to run the Terraform with `terraform apply`.

This will create a Terraform plan which shows all the resources to be created. To see an example plan, see the [example plan](https://github.com/streamnative/terraform-managed-cloud/blob/main/modules/azure/README.md#using-sn-cloud-manager-and-vendor-access-modules-together) in the GitHub readme.

### Step 5: Provide the output to StreamNative

Once completed, provide the output of the `terraform apply` to your CSM or support representative.

### Step 6: Create a Resource Group

Unlike AWS and GCP, Azure doesn't allow to create resources in region directly. Instead, you need to create a **Resource Group** first, and then create resources in this **Resource Group**.
To create a **Resource Group**, you need to run the below terraform script:

```hcl theme={null}
provider "azurerm" {
  features {
  }
}

provider "azuread" {}
module "azure-managed-cloud" {
  source = "github.com/streamnative/terraform-managed-cloud//modules/azure/vendor-access?ref=v3.23.0"
  resource_group_name     = "<YOUR_AKS_CLUSTER_RESOURCE_GROUP_NAME>"
  resource_group_location = "<YOUR_DESIRED_AZURE_REGION>"

  streamnative_org_id = "<YOUR_SNCLOUD_ORG_ID>"
  sn_automation_client_id = "<CLIENT_ID>"
  sn_support_client_id = "<SUPPORT_CLIENT_ID>"
}
```

* `<YOUR_SNCLOUD_ORG_ID>`: your StreamNative Cloud organization ID. This is typically an ID like `o-xxxxx`. This can be found in your organization list or the top header of the application.
* `<YOUR_DESIRED_AZURE_REGION>`: the Azure region where you want to create the resources. This should be a valid Azure region, such as `eastus` or `westus2`, it can be different with the `<ANY_AZURE_REGION>`.
* `<YOUR_AKS_CLUSTER_RESOURCE_GROUP_NAME>`: the name of the resource group where the AKS cluster will be created. This should be a unique name within your Azure subscription.
* `<CLIENT_ID>`: the client ID of the sn automation service principal for StreamNative Cloud automation.
* `<SUPPORT_CLIENT_ID>`: the client ID of the sn support service principal for StreamNative Cloud support access.

You can get the `CLIENT_ID` and `SUPPORT_CLIENT_ID` from the output of the previous terraform apply or from the Azure Portal:
There will be a ResourceGroup called `sncloud-<ORG_ID>-manager-rg` in your Azure subscription, and it has two managed identities while one is `sncloud-<ORG_ID>-automation` and the other is `sncloud-<ORG_ID>-support`,
You can find the `Client ID` in the `sncloud-<ORG_ID>-automation` and `sncloud-<ORG_ID>-support` managed identities.

<Note title="Note">
  StreamNative requires deployment of an AKS cluster across all supported availability zones (1, 2, 3) in Azure.

  It's critical to note that not all regions have availability zone support, as documented in [Microsoft's Supported Azure regions list](https://learn.microsoft.com/en-us/azure/reliability/availability-zones-region-support), please use region which support the availability zone.

  Azure subscriptions may also impose **SKU**(Stock Keeping Unit) restrictions that could block BYOC environment provisioning, please execute the below command to verify that there is no such limit in the region:

  ```shell theme={null}
  az vm list-skus --location <REGION> --size Standard_D8s_V3 --all --output table
  ```

  It should print the below output:

  ```shell theme={null}
  ResourceType     Locations    Name             Zones    Restrictions
  ---------------  -----------  ---------------  -------  --------------
  virtualMachines  <REGION>     Standard_D8s_v3  1,2,3    None
  ```

  If the output looks like below:

  ```shell theme={null}
  ResourceType     Locations    Name             Zones    Restrictions
  sidebarTitle: Account Access for BYOC on Azure
  ---------------  -----------  ---------------  -------  -----------------------------------------------------------------------
  virtualMachines  <REGION>     Standard_D8s_v3  1,2,3    NotAvailableForSubscription, type: Zone, locations: westus2, zones: 3,1
  ```

  This means that there is a SKU limit in your subscription, please contact Azure support to remove the limit or change the region.
</Note>

## Next steps

After granting access, you can [set up a Cloud Connection](/cloud/clusters/byoc/create-cloud-connection) to allow the StreamNative Cloud control plane to interact with your Azure account.


# Set up Project Access for BYOC on Google Cloud
Source: https://docs.streamnative.io/cloud/clusters/byoc/grant-vendor-access/byoc-on-gcp/byoc-gcp-access



Before you can provision a BYOC [infrastructure pool](/cloud/clusters/streamnative-cluster-overview#infrastructure-pools), you will need to authorize the StreamNative Cloud control plane to access your cloud accounts. This enables StreamNative to provision and manage clusters within your GCP project.

This document outlines the procedure for granting this access to StreamNative Cloud for GCP projects.

## Access model in Google Cloud

StreamNative Cloud leverages GCP's IAM [role bindings](https://cloud.google.com/iam/docs/roles-overview) to manage access to customer's projects, allowing for efficient management of only necessary resources. Access to customer projects is segmented into three Google Service Accounts (abbr. `GSA`):

* **Provisioning GSA**: This service account handles the provisioning and maintenance of the underlying infrastructure like DNS Zone, VPCs, GKE clusters (and associated node groups, and so on). Default to `pool-automation@sncloud-production.iam.gserviceaccount.com`.

* **Management GSA**: This service account is primarily for automated management tasks, the core responsibility of this service account is interact with the GKE cluster to deploy and manage Pulsar clusters. Default to `cloud-manager@sncloud-production.iam.gserviceaccount.com`.

* **Support GSA**: This service account is used by the StreamNative SRE and Support team for troubleshooting during incidents. Default to `cloud-support-general@sncloud-production.iam.gserviceaccount.com`.

These GSA and role bindings can be implemented using the StreamNative [Vendor Access Terraform](https://github.com/streamnative/terraform-managed-cloud) module. The following diagram illustrates GCP access:

<img alt="BYOC Google Cloud Access" />

## Provision Google Cloud Access

StreamNative facilitates the setup of necessary service accounts and permissions via a <Tooltip>Terraform</Tooltip> module. This module can be provisioned in a standalone Terraform project (as documented here), but can also be integrated into existing Terraform projects.

For full documentation of inputs and outputs of the Terraform module, see the [module's README on GitHub](https://github.com/streamnative/terraform-managed-cloud/tree/main/modules/gcp/vendor-access).

### Prerequisites

* New to Terraform? Learn the [Terraform Google Cloud Getting Started Tutorial](https://developer.hashicorp.com/terraform/tutorials/gcp-get-started) to get a basic introduction.
* Install Terraform, version 1.3.0 or greater.
* Ensure you have [created an organization](/cloud/security/access/resource-hierarchy/organizations#create-an-organization) through the StreamNative Cloud Console.

If you run into issues, please contact [StreamNative Support team](https://support.streamnative.io/hc/en-us/requests/new).

### Step 1: Create a new project and instantiate the module

Terraform works by having Terraform codes (in the form of `*.tf` files) and state files that represent the current resources. If you are using Terraform locally, without a [remote state store](https://developer.hashicorp.com/terraform/language/state/remote), these files should be checked into source control for future updates.
Create a new folder and add a file called `main.tf` with the following content, replacing the referenced variables.

```hcl theme={null}
provider "google" {
  project = "<YOUR_GCP_PROJECT_ID>"
}

module "sn_managed_cloud" {
  source = "github.com/streamnative/terraform-managed-cloud//modules/gcp/vendor-access?ref=v3.23.0"
  project = "<YOUR_GCP_PROJECT_ID>"
  streamnative_org_id = "<YOUR_SNCLOUD_ORG_ID>"
}
```

* `<YOUR_GCP_PROJECT_ID>`: your Google Project ID.
* `<YOUR_SNCLOUD_ORG_ID>`: your StreamNative Cloud organization ID. This is typically an ID like `o-xxxxx`. This can be found in your organization list or the top header of the application.

If you are using `git` as source control, you need to use the `git init` command to initialize this folder as a git project.

### Step 2: Initialize the Terraform

While the above Terraform code is all needed, the module needs to be downloaded to this Terraform project.

To do so, run `terraform init`.

This will download the module and required dependencies.

### Step 3: Create a shell with the correct GCP credentials

Terraform requires GCP credentials with the proper permissions in the target project to create the resources to grant access. The permissions required by the module are all GCP project permissions, specifically to manage the GCP services, roles, and service accounts. The GCP built-in role of `Editor` to the GCP project is sufficient to perform these operations.

All of the methods in [Authenticating to GCP](https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/provider_reference#authentication) are compatible with the Terraform module.

### Step 4: Run the Terraform

After initialization, and with credentials in the shell, the next step is to run the Terraform with `terraform apply`.

This will create a Terraform plan which shows all the resources to be created. To see an example plan, see the [example plan](https://github.com/streamnative/terraform-managed-cloud/tree/main/modules/gcp/vendor-access#get-started) in the GitHub readme.

### Step 5: Provide the output to StreamNative

Once completed, provide the output of the `terraform apply` to your CSM or support representative.

## Next steps

After granting access, you can [set up a Cloud Connection](/cloud/clusters/byoc/create-cloud-connection) to allow the StreamNative Cloud control plane to interact with your GCP project.


# Manage StreamNative Instances
Source: https://docs.streamnative.io/cloud/clusters/manage-instances/instance



A **StreamNative Instance** represents a cohesive group of clusters functioning together as a unified entity.

<Note>
  The instance management content in this section applies to both Kafka and Pulsar clusters. However, the cluster management content within these pages is primarily focused on Pulsar Clusters. For Kafka Cluster management, see the [Kafka Cluster Guide](/kafka/kafka-cluster-guide).
</Note>

Each instance is uniquely identified by a **Uniform Resource Name (URN)** that follows the structure `urn:sn:pulsar:organization-id:pulsar-instance-name`. You can retrieve the organization name and instance name using the `snctl get organizations <organization-id>` and `snctl get pulsarinstance <instance-name>` commands, respectively. When a Pulsar client connects to a Pulsar cluster using [OAuth2 authentication](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview), this URN is required as **audience** to use for the authentication process.

After [creating an organization](/cloud/security/access/resource-hierarchy/organizations#create-an-organization), you can create one or more Serverless, Dedicated, or BYOC instances within that organization. However, before creating BYOC instances, you must first [provision one or more BYOC infrastructure pools](/cloud/clusters/byoc/byoc-overview#provision-byoc-infa-pools) in your cloud accounts.

## Instance Overview

An Instance is a logical entity that groups one or more clusters. Clusters can be distributed across multiple geographic regions and replicated between them using geo-replication.

### Instance Types

StreamNative supports three types of instances:

* **Serverless**: A fully managed instance on StreamNative's cloud infrastructure that automatically scales based on the workload.
* **Dedicated**: A dedicated instance on StreamNative's cloud infrastructure that you can fully control the underlying resources allocated to the instance.
* **BYOC / BYOC Pro**: A Bring Your Own Cloud (BYOC) instance that you can manage and control in your cloud accounts. A **BYOC Pro** instance is an advanced version of **BYOC** that supports advanced private networking and security features.

### Availability Modes

The availability mode of an instance determines how clusters are distributed across availability zones within a geographic region. StreamNative supports two availability modes:

* **Regional**: Clusters are distributed across multiple availability zones within a geographic region, providing higher availability and fault tolerance.
* **Zonal**: Clusters are deployed in a single availability zone within a geographic region.

Currently, StreamNative Cloud only supports the **Regional** availability mode. Support for **Zonal** availability mode will be available in a future release.

All the clusters within an instance must use the same availability mode.

### Pool Reference

An instance is deployed to one infrastructure pool, which can be either **Fully Hosted** (for **Serverless** and **Dedicated** instances) or **BYOC** (for **BYOC** instances). A pool reference identifies which pool will be used for deploying instances.

* For **Serverless** and **Dedicated** instances, the pool reference points to StreamNative's fully hosted pools. The `pool_namespace` is always set to `streamnative`.
* For **BYOC** or **BYOC Pro** instances, the pool reference points to pools that are provisioned through cloud environments in your cloud accounts. The `pool_namespace` is set to the organization ID where the cloud environment was created.

## Next steps

* [Manage Serverless Instances](/cloud/clusters/manage-instances/manage-serverless-instances)
* [Manage Dedicated Instances](/cloud/clusters/manage-instances/manage-dedicated-instances)
* [Manage BYOC Instances](/cloud/clusters/manage-instances/manage-byoc-instances)

After creating an instance, you can create a cluster within the instance.

* [Work with clusters](/cloud/clusters/manage-clusters/cluster)


# Manage BYOC Instances on StreamNative Cloud
Source: https://docs.streamnative.io/cloud/clusters/manage-instances/manage-byoc-instances



## Prerequisites

Before creating a BYOC instance and clusters, you need to prepare for the BYOC infrastructure. See [BYOC Overview](/cloud/clusters/byoc/byoc-overview) for more information.

After you have prepared the BYOC infrastructure, you can create a BYOC instance and clusters.

# Create an instance

When creating an instance, you need to make a choice of cloud provider.

<Tabs>
  <Tab title="Console">
    <Note title="Note">
      Please make sure you have created a cloud environment before creating a BYOC instance.
    </Note>

    In the UI, you will create a **Dedicated** instance and a cluster within the instance.

    1. Navigate to the [**Organization Dashboard**](/cloud/get-started/cloud-console#organization-dashboard).

    2. Click **Instances** on the left navigation pane to go to the **Instances** page.

    3. Click **+ New Instance** button to start the instance creation process.

    4. On the **Choose the deployment type for your instance** page, click **Deploy BYOC** to start the instance creation process. If you see the dialog "Cloud Environment required", you need to create a cloud environment first. After creating the cloud environment, return to step 1 to create the instance.

    5. On the **Instance Configuration** page, enter the **Instance Name** and select the **Cloud Provider**.

       | Item             | Description                                                                                                                                                                                       |
       | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
       | Instance Name    | Enter a name for the instance. An instance name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-40 characters. |
       | Cloud Connection | Select the cloud connection                                                                                                                                                                       |

    6. Click **Cluster Location** to enter the cluster details.

    ## **Cluster Configuration**

    The configuration steps may vary based on the cluster profile selection. Please follow the instructions specific to the selected cluster profile outlined below.

    ### **Create Latency Optimized Cluster**

    Follow the details listed below to create a cluster based on **Latency Optimized Profile**.

    #### **Cluster Details**

    Enter the **Cluster Name**, select the **Cloud Environment** from the dropdown list, select **Latency Optimized Cost Profile**, select **Availability Zone**, then click **Lakehouse Table**.

    | Item              | Description                                                                                                                                                                                    |
    | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
    | Cluster Name      | Enter a name for the cluster. A cluster name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters. |
    | Cloud Environment | Select the cloud environment from the dropdown list.                                                                                                                                           |
    | Cluster Profile   | Select Latency Optimized Cluster profile                                                                                                                                                       |
    | Availability Zone | Select **Single AZ** to deploy the instance across single availability zone or **Multiple AZ** for Multiple Availability Zone                                                                  |

    #### **Lakehouse Tables**

    You can optionally enable Lakehouse table by selecting a **Catalog Provider**, selecting a **registered catalog from dropdown**, and optionally applying the setting to all topics.

    #### **Cluster Operations**

    On the **Cluster Operation** page:

    1. Select the **Release Channel**: **LTS** or **Rapid**. Please note that **Rapid** channel is required for running **Cost Optimized Cluster**.

    2. (Optional) Enable or disable the **Features** you want on your cluster. You can enable or disable these features at any time by editing your cluster after it has been created.

       <img alt="screenshot of the cluster features section" />

    3. (Optional) If you have a Enterprise or Production support plan, you can customize the maintenance window for the cluster. Otherwise, you can skip this step.

    4. (Optional) Expand the **Add optional custom configurations** section to specify optional custom configuration parameters.

    5. Click **Cluster Size**.

    ### **Create Cost Optimized Cluster**

    Follow the details listed below to create a cluster based on **Cost Optimized Profile**.

    #### **Cluster Details**

    Enter the **Cluster Name**, select the **Cloud Environment** from the dropdown list, select **Availability Zone**, then click **Lakehouse Storage Configuration**.

    | Item              | Description                                                                                                                                                                                    |
    | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
    | Cluster Name      | Enter a name for the cluster. A cluster name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters. |
    | Cloud Environment | Select the cloud environment from the dropdown list.                                                                                                                                           |
    | Cluster Profile   | Select Cost Optimized Profile.                                                                                                                                                                 |
    | Availability Zone | Select **Single AZ** to deploy the instance across single availability zone or **Multiple AZ** for Multiple Availability Zone                                                                  |

    #### **Lakehouse Tables**

    1. Select the **Storage Location**: **Use Your Own Bucket** or **Use Existing BYOC Bucket**.

       * For the **Use Your Own Bucket** option, enter following details

         * **AWS role ARN**
         * **Region**
         * **Bucket name**
         * **Bucket path**
         * **Confirm that StreamNative has been granted the necessary permissions to access your bucket.**

       * For the **Use Existing BYOC Bucket**, the bucket is created by BYOC Cloud environment.

    2. Select **Catalog integration** : Enable or Disable

       * **Enable Catalog integration**
         * **Lakehouse tables** :
           * **Managed Table**
             * **Select catalog provider**
               * **Databricks Unity Catalog** - Select a registered catalog from the dropdown. To register a catalog first, see [Register a catalog](/cloud/lakehouse/catalogs/register-catalog).
               * **Snowflake Open Catalog** - Select a registered catalog from the dropdown. To register a catalog first, see [Register a catalog](/cloud/lakehouse/catalogs/register-catalog).
           * **External Table**
             * **Select catalog provider**
               * **Snowflake Open Catalog** - Select a registered catalog from the dropdown. To register a catalog first, see [Register a catalog](/cloud/lakehouse/catalogs/register-catalog).
               * **Amazon S3 Tables** - Select a registered catalog from the dropdown. To register a catalog first, see [Register a catalog](/cloud/lakehouse/catalogs/register-catalog).
       * **Disable Catalog integration**
         * **Storage table format**
           * **Delta Lake** : This option allows you to write topics data as Delta Tables
           * **Apache Iceberg** : This option allows you write topics data as Iceberg tables.

    3. Click **Cluster Size**.

    4. On the **Cluster Size** page:

       * Use the slider to adjust the throughput according to your needs.
       * In the **Advanced** section, you can manually configure the number of brokers, bookies, and their corresponding resources.
       * Note that bookies configuration is only required for **Latency Optimized Clusters**. For **Cost Optimized Clusters**, bookie configuration is not needed.

       The estimated **Monthly base cost** for your cluster configuration is displayed in the right navigation pane.

    5. Click **Finish** to start the cluster creation process.

    The cluster page appears, showing the cluster creation process. Depending on the chosen cloud provider and other settings, it might take 10-15 minutes to provision a dedicated cluster. Once the cluster is ready, the page will show **Cluster Provisioned successfully** and you can click **Go To The Dashboard** to access the **Cluster Dashboard** page.
  </Tab>

  <Tab title="Terraform">
    You can define the **BYOC** or **BYOC Pro** Instance in a Terraform configuration file as below:

    ```hcl theme={null}
    resource "streamnative_pulsar_instance" "test-instance" {
      organization = "<your-organization>"
      name = "<your-instance-name>"
      availability_mode = "regional"
      pool_name = "<your-byoc-pool-name>"
      pool_namespace = "<your-byoc-pool-organization>"
      type = "byoc" # or "byoc-pro"
      engine = "classic" # or "ursa"
    }
    ```

    * `organization`: The organization ID. Replace `<your-organization>` with the actual organization ID.
    * `name`: The name of the BYOC instance. Replace `<your-instance-name>` with the actual name.
    * `availability_mode`: The availability mode of the instance. Currently, only `regional` is supported.
    * `pool_name`: The name of the cloud connection that runs the **BYOC** or **BYOC Pro** instance.
    * `pool_namespace`: The organization ID where the cloud connection is created.
    * `type`: The type of the instance. It can be either `byoc` or `byoc-pro`. Note that a **BYOC** instance can only be created in a **BYOC** pool (cloud environment), and a **BYOC Pro** instance can only be created in a **BYOC Pro** pool (cloud environment).
    * `engine`: The data streaming engine for the instance. It can be either `classic` or `ursa`.

    See [PulsarInstance](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/resources/pulsar_instance) for more information.

    After you defined the instance, you can continue to define the **Cluster** resource. See [work with clusters](/cloud/clusters/manage-clusters/cluster) for more information.
  </Tab>
</Tabs>

## Manage instances

<Tabs>
  <Tab title="Console">
    To view instances created for an organization, follow these steps.

    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Organizations**.

    2. Click the name of the organization you want to check.

    3. Select **Instances** from the left navigation pane.

    4. On the **Instances** page, you should able to see the list of instances available for the organization. In each **Instance Card**, you are able to see **Instance Name**, **Status**, **Cloud Provider**, **Number of Clusters**, and etc. You can also click the right arrow icon to go the **Instance Dashboard** page.

       <img alt="Instances Dashboard" />

    5. On the **Instance Dashboard** page, you are able to see the list of clusters available for the instance. In each **Cluster Card**, you are able to see **Cluster Name**, **Status**, **Number of Topics**, **Number of Subscriptions**, **Number of Producers**, **Number of Consumers**, and etc. You can also click the right arrow icon to go the **Cluster Dashboard** page.

       <img alt="Clusters Dashboard" />
  </Tab>

  <Tab title="snctl">
    To list of the all instances available for an organization, run the following command:

    ```bash theme={null}
    snctl get pulsarinstances -O <your-org-id>
    ```

    If you want to get more details about an instance, you can run the following command:

    ```bash theme={null}
    snctl get pulsarinstance <your-instance-name> -O <your-org-id>
    ```

    You should be able to get the details of the instance.

    ```yaml theme={null}
    spec:
      auth:
        apikey: {}
      availabilityMode: regional
      poolRef:
        name: shared-gcp
        namespace: streamnative
      type: serverless
    status:
      auth:
        oauth2:
          audience: urn:sn:pulsar:<your-org-id>:<your-instance-name>
          issuerURL: https://auth.streamnative.cloud/
        type: oauth2
      conditions:
        - lastTransitionTime: '2024-11-29T21:34:08Z'
          message: a payment method is not required because discount is active
          reason: HasActiveDiscount
          status: 'True'
          type: SubscriptionReady
        - lastTransitionTime: '2024-11-29T21:34:08Z'
          reason: Created
          status: 'True'
          type: ResourceServerReady
        - lastTransitionTime: '2024-11-29T21:34:09Z'
          reason: Created
          status: 'True'
          type: ServiceAccountReady
        - lastTransitionTime: '2024-11-29T21:34:10Z'
          reason: AllConditionStatusTrue
          status: 'True'
          type: Ready
    ```

    * `status.auth.oauth2.audience`: The audience of the Pulsar instance. It will be used for the [Oauth2 authentication](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview).
    * `status.auth.oauth2.issuerURL`: The oauth2 issuer URL of the Pulsar instance. It will be used for the [Oauth2 authentication](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview).
    * In the `conditions` section, you can see the status of the instance. If all conditions are `True`, the instance is ready.
  </Tab>

  <Tab title="Terraform">
    If you want to get more details about an instance, you can define a data source in the Terraform configuration file.

    ```hcl theme={null}
    data "streamnative_pulsar_instance" "test-instance" {
      organization = "<your-organization>"
      name = "<your-instance-name>"
    }
    ```

    You should be able to get the details of the instance.

    * `id`: The ID of the instance.
    * `availability_mode`: The availability mode of the instance. Currently, only `regional` is supported.
    * `oauth2_audience`: The audience of the Pulsar instance. It will be used for the [Oauth2 authentication](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview).
    * `oauth2_issuer_url`: The oauth2 issuer URL of the Pulsar instance. It will be used for the [Oauth2 authentication](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview).
    * `pool_name`: The name of the infrastructure pool that runs the **Serverless** instance. Currently it only supports Google Cloud (`shared-gcp`).
    * `pool_namespace`: The namespace of the infrastructure pool (`streamnative`).
    * `ready`: The status of the instance. If the Pulsar instance is ready, it is `True`.

    You can checkout [PulsarInstance](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/data-sources/pulsar_instance) for more information.
  </Tab>
</Tabs>

## Delete an instance

<Note title="Note">
  You cannot delete an instance if there are resources associated with the instance.
</Note>

<Tabs>
  <Tab title="Console">
    1. Navigate to the **Instances** page.

    2. Click the ellipsis at the top right corner of the instance card that you want to delete, and then click **Delete**.

    3. In the **Delete instance** dialog, enter the instance name and then click **Confirm**.
  </Tab>

  <Tab title="snctl">
    There are two ways to delete an instance.

    * Delete the instance by the instance name.

      ```bash theme={null}
      snctl delete pulsarinstance <your-instance-name>
      ```

    * Delete the instance by the instance manifest file `instance.yaml`.

      ```bash theme={null}
      snctl delete -f instance.yaml
      ```
  </Tab>

  <Tab title="Terraform">
    Remove the instance resource from the Terraform configuration file and run `terraform apply` to delete the instance.
  </Tab>
</Tabs>

## Next steps

* [Work with clusters](/cloud/clusters/manage-clusters/cluster)

## Related topics

* Check other types of instances:
  * [Serverless Instances](/cloud/clusters/manage-instances/manage-serverless-instances)
  * [Dedicated Instances](/cloud/clusters/manage-instances/manage-dedicated-instances)


# Manage Dedicated Instances on StreamNative Cloud
Source: https://docs.streamnative.io/cloud/clusters/manage-instances/manage-dedicated-instances



<Note title="Note">
  **Cost Optimized Cluster Profile** is currently not available for Dedicated instances.
</Note>

## Create an instance

<Tabs>
  <Tab title="Console">
    In the UI, you will create a **Dedicated** instance and a cluster within the instance.

    1. Navigate to the [**Organization Dashboard**](/cloud/get-started/cloud-console#organization-dashboard).

    2. Click **Instances** on the left navigation pane to go to the **Instances** page.

    3. Click **+ New Instance** button to start the instance creation process.

    4. On the **Choose the deployment type for your instance** page, click **Deploy Dedicated** to start the instance creation process.

    5. On the **Instance Configuration** page, enter the **Instance Name** and select the **Cloud Provider**.

       | Item           | Description                                                                                                                                                                                       |
       | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
       | Instance Name  | Enter a name for the instance. An instance name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-40 characters. |
       | Cloud Provider | Select the cloud provider. Currently, Google Cloud is available.                                                                                                                                  |

    6. Click **Cluster Location** to start the cluster creation process.

    7. On the **Cluster Location** page, enter the **Cluster Name**, select the **Location** from the dropdown list, and select the **Availability Zone (AZ)**, then click **Cluster Operation**.

       | Item         | Description                                                                                                                                                                                    |
       | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
       | Cluster Name | Enter a name for the cluster. A cluster name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters. |
       | Location     | Select the location from the dropdown list. The location represents the available regions in the designated cloud provider.                                                                    |
       | AZ           | Select the availability zone from the dropdown list: Single AZ or Multi AZ.                                                                                                                    |

       <Note title="Need a Different Region?">
         If the region you need is not available in the dropdown list, [contact StreamNative support](https://support.streamnative.io/hc/en-us/requests/new) to discuss your region requirements. We may be able to accommodate specific region requests for Dedicated clusters.
       </Note>

    8. On the **Cluster Operation** page:

       1. Select the **Release Channel**: **LTS** or **Rapid**.

       2. (Optional) Enable or disable the **Features** you want on your cluster. You can enable or disable these features at any time by editing your cluster after it has been created.

          <img alt="screenshot of the cluster features section" />

       3. (Optional) Expand the **Add optional custom configurations** section to specify optional custom configuration parameters.

       4. Click **Cluster Size**.

    9. On the **Cluster Size** page:

       * Use the slider to adjust the throughput according to your needs.
       * In the **Advanced** section, you can manually configure the number of brokers, bookies, and their corresponding resources.

       The estimated **Monthly base cost** for your cluster configuration is displayed in the right navigation pane.

    10. Click **Finish** to start the cluster creation process.

    The cluster page appears, showing the cluster creation process. Depending on the chosen cloud provider and other settings, it might take 10-15 minutes to provision a dedicated cluster. Once the cluster is ready, the page will show **Cluster Provisioned successfully** and you can click **Go To The Dashboard** to access the **Cluster Dashboard** page.
  </Tab>

  <Tab title="Terraform">
    You can define the Dedicated Instance in a Terraform configuration file as below:

    ```hcl theme={null}
    resource "streamnative_pulsar_instance" "test-instance" {
      organization = "<your-organization>"
      name = "<your-instance-name>"
      availability_mode = "regional"
      pool_name = "<streamnative-dedicated-pool-name>"
      pool_namespace = "streamnative"
      type = "dedicated"
    }
    ```

    * `organization`: The organization ID. Replace `<your-organization>` with the actual organization ID.
    * `name`: The name of the Dedicated Instance. Replace `<your-instance-name>` with the actual name.
    * `availability_mode`: The availability mode of the instance. Currently, only `regional` is supported.
    * `pool_name`: The name of the infrastructure pool that runs the **Dedicated** instance. Currently it only supports Google Cloud (`shared-gcp`).
    * `pool_namespace`: The namespace of the infrastructure pool (`streamnative`).
    * `type`: The type of the instance. For **Dedicated** instances, set it to `dedicated`.

    See [PulsarInstance](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/resources/pulsar_instance) for more information.

    After you defined the instance, you can continue to define the **Cluster** resource. See [work with clusters](/cloud/clusters/manage-clusters/cluster) for more information.
  </Tab>
</Tabs>

## Manage instances

<Tabs>
  <Tab title="Console">
    To view instances created for an organization, follow these steps.

    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Organizations**.

    2. Click the name of the organization you want to check.

    3. Select **Instances** from the left navigation pane.

    4. On the **Instances** page, you should able to see the list of instances available for the organization. In each **Instance Card**, you are able to see **Instance Name**, **Status**, **Cloud Provider**, **Number of Clusters**, and etc. You can also click the right arrow icon to go the **Instance Dashboard** page.

       <img alt="Instances Dashboard" />

    5. On the **Instance Dashboard** page, you are able to see the list of clusters available for the instance. In each **Cluster Card**, you are able to see **Cluster Name**, **Status**, **Number of Topics**, **Number of Subscriptions**, **Number of Producers**, **Number of Consumers**, and etc. You can also click the right arrow icon to go the **Cluster Dashboard** page.

       <img alt="Clusters Dashboard" />
  </Tab>

  <Tab title="snctl">
    To list of the all instances available for an organization, run the following command:

    ```bash theme={null}
    snctl get pulsarinstances -O <your-org-id>
    ```

    If you want to get more details about an instance, you can run the following command:

    ```bash theme={null}
    snctl get pulsarinstance <your-instance-name> -O <your-org-id>
    ```

    You should be able to get the details of the instance.

    ```yaml theme={null}
    spec:
      auth:
        apikey: {}
      availabilityMode: regional
      poolRef:
        name: shared-gcp
        namespace: streamnative
      type: serverless
    status:
      auth:
        oauth2:
          audience: urn:sn:pulsar:<your-org-id>:<your-instance-name>
          issuerURL: https://auth.streamnative.cloud/
        type: oauth2
      conditions:
        - lastTransitionTime: '2024-11-29T21:34:08Z'
          message: a payment method is not required because discount is active
          reason: HasActiveDiscount
          status: 'True'
          type: SubscriptionReady
        - lastTransitionTime: '2024-11-29T21:34:08Z'
          reason: Created
          status: 'True'
          type: ResourceServerReady
        - lastTransitionTime: '2024-11-29T21:34:09Z'
          reason: Created
          status: 'True'
          type: ServiceAccountReady
        - lastTransitionTime: '2024-11-29T21:34:10Z'
          reason: AllConditionStatusTrue
          status: 'True'
          type: Ready
    ```

    * `status.auth.oauth2.audience`: The audience of the Pulsar instance. It will be used for the [Oauth2 authentication](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview).
    * `status.auth.oauth2.issuerURL`: The oauth2 issuer URL of the Pulsar instance. It will be used for the [Oauth2 authentication](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview).
    * In the `conditions` section, you can see the status of the instance. If all conditions are `True`, the instance is ready.
  </Tab>

  <Tab title="Terraform">
    If you want to get more details about an instance, you can define a data source in the Terraform configuration file.

    ```hcl theme={null}
    data "streamnative_pulsar_instance" "test-instance" {
      organization = "<your-organization>"
      name = "<your-instance-name>"
    }
    ```

    You should be able to get the details of the instance.

    * `id`: The ID of the instance.
    * `availability_mode`: The availability mode of the instance. Currently, only `regional` is supported.
    * `oauth2_audience`: The audience of the Pulsar instance. It will be used for the [Oauth2 authentication](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview).
    * `oauth2_issuer_url`: The oauth2 issuer URL of the Pulsar instance. It will be used for the [Oauth2 authentication](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview).
    * `pool_name`: The name of the infrastructure pool that runs the **Serverless** instance. Currently it only supports Google Cloud (`shared-gcp`).
    * `pool_namespace`: The namespace of the infrastructure pool (`streamnative`).
    * `ready`: The status of the instance. If the Pulsar instance is ready, it is `True`.

    You can checkout [PulsarInstance](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/data-sources/pulsar_instance) for more information.
  </Tab>
</Tabs>

## Delete an instance

<Note title="Note">
  You cannot delete an instance if there are resources associated with the instance.
</Note>

<Tabs>
  <Tab title="Console">
    1. Navigate to the **Instances** page.

    2. Click the ellipsis at the top right corner of the instance card that you want to delete, and then click **Delete**.

    3. In the **Delete instance** dialog, enter the instance name and then click **Confirm**.
  </Tab>

  <Tab title="snctl">
    There are two ways to delete an instance.

    * Delete the instance by the instance name.

      ```bash theme={null}
      snctl delete pulsarinstance <your-instance-name>
      ```

    * Delete the instance by the instance manifest file `instance.yaml`.

      ```bash theme={null}
      snctl delete -f instance.yaml
      ```
  </Tab>

  <Tab title="Terraform">
    Remove the instance resource from the Terraform configuration file and run `terraform apply` to delete the instance.
  </Tab>
</Tabs>

## Next steps

* [Work with clusters](/cloud/clusters/manage-clusters/cluster)

## Related topics

* Check other types of instances:
  * [Dedicated Instances](/cloud/clusters/manage-instances/manage-dedicated-instances)
  * [BYOC Instances](/cloud/clusters/manage-instances/manage-byoc-instances)


# Manage Serverless Instances on StreamNative Cloud
Source: https://docs.streamnative.io/cloud/clusters/manage-instances/manage-serverless-instances



## Create an instance

<Tabs>
  <Tab title="Console">
    In the UI, you will create a **Serverless** instance and a cluster within the instance.

    1. Navigate to the [**Organization Dashboard**](/cloud/get-started/cloud-console#organization-dashboard).

    2. Click **Instances** on the left navigation pane to go to the **Instances** page.

    3. Click **+ New Instance** button to start the instance creation process.

    4. On the **Choose the deployment type for your instance** page, click **Deploy Serverless** to start the instance creation process.

    5. On the **Instance Configuration** page, enter the **Instance Name** and select the **Cloud Provider**.

       | Item           | Description                                                                                                                                                                                       |
       | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
       | Instance Name  | Enter a name for the instance. An instance name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-40 characters. |
       | Cloud Provider | Select the cloud provider. Currently, Google Cloud is available.                                                                                                                                  |

    6. Click **Cluster Location** to start the cluster creation process.

    7. On the **Cluster Name** page, enter the **Cluster Name** and select the **Location** from the dropdown list.

       | Item         | Description                                                                                                                                                                                    |
       | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
       | Cluster Name | Enter a name for the cluster. A cluster name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters. |
       | Location     | Select the location from the dropdown list. The location represents the available regions in the designated cloud provider.                                                                    |

    8. Click **Finish**.

    The cluster page appears, showing the cluster creation process. Depending on the chosen cloud provider and other settings, it might take a few seconds to several minutes to provision the cluster. Once the cluster is ready, the **Cluster Dashboard** page appears.
  </Tab>

  <Tab title="snctl">
    Edit a file named `instance.yaml` with the following content:

    ```yaml theme={null}
    apiVersion: cloud.streamnative.io/v1alpha1
    kind: PulsarInstance
    metadata:
      name: <your-instance-name>
      namespace: <your-org-id>
    spec:
      availabilityMode: regional
      poolRef:
        name: shared-gcp
        namespace: streamnative
      type: serverless
    ```

    * `metadata.name`: The name of the Serverless Instance. Replace `<your-instance-name>` with the actual name.
    * `metadata.namespace`: The organization ID. Replace `<your-org-id>` with the actual organization ID.
    * `spec.availabilityMode`: The availability mode of the instance. Currently, only `regional` is supported.
    * `spec.poolRef`: Refer to the infrastructure pool that runs the **Serverless** instance. Currently it only supports Google Cloud.
    * `spec.type`: The type of the instance. For **Serverless** instances, set it to `serverless`.

    Run the following command to provision the Serverless Instance:

    ```bash theme={null}
    snctl create -f instance.yaml
    ```

    Once the command is completed, you should be able to see the following message:

    ```bash theme={null}
    pulsarinstance.cloud.streamnative.io/<your-instance-name> created
    ```

    Once you have created the instance, you can continue to create a cluster. See [work with clusters](/cloud/clusters/manage-clusters/cluster) for more information.
  </Tab>

  <Tab title="Terraform">
    You can define the Serverless Instance in a Terraform configuration file as below:

    ```hcl theme={null}
    resource "streamnative_pulsar_instance" "test-instance" {
      organization = "<your-organization>"
      name = "<your-instance-name>"
      availability_mode = "regional"
      pool_name = "shared-gcp"
      pool_namespace = "streamnative"
      type = "serverless"
    }
    ```

    * `organization`: The organization ID. Replace `<your-organization>` with the actual organization ID.
    * `name`: The name of the Serverless Instance. Replace `<your-instance-name>` with the actual name.
    * `availability_mode`: The availability mode of the instance. Currently, only `regional` is supported.
    * `pool_name`: The name of the infrastructure pool that runs the **Serverless** instance. Currently it only supports Google Cloud (`shared-gcp`).
    * `pool_namespace`: The namespace of the infrastructure pool (`streamnative`).
    * `type`: The type of the instance. For **Serverless** instances, set it to `serverless`.

    See [PulsarInstance](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/resources/pulsar_instance) for more information.

    After you defined the instance, you can continue to define the **Cluster** resource. See [work with clusters](/cloud/clusters/manage-clusters/cluster) for more information.
  </Tab>
</Tabs>

## Manage instances

<Tabs>
  <Tab title="Console">
    To view instances created for an organization, follow these steps.

    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Organizations**.

    2. Click the name of the organization you want to check.

    3. Select **Instances** from the left navigation pane.

    4. On the **Instances** page, you should able to see the list of instances available for the organization. In each **Instance Card**, you are able to see **Instance Name**, **Status**, **Cloud Provider**, **Number of Clusters**, and etc. You can also click the right arrow icon to go the **Instance Dashboard** page.

       <img alt="Instances Dashboard" />

    5. On the **Instance Dashboard** page, you are able to see the list of clusters available for the instance. In each **Cluster Card**, you are able to see **Cluster Name**, **Status**, **Number of Topics**, **Number of Subscriptions**, **Number of Producers**, **Number of Consumers**, and etc. You can also click the right arrow icon to go the **Cluster Dashboard** page.

       <img alt="Clusters Dashboard" />
  </Tab>

  <Tab title="snctl">
    To list of the all instances available for an organization, run the following command:

    ```bash theme={null}
    snctl get pulsarinstances -O <your-org-id>
    ```

    If you want to get more details about an instance, you can run the following command:

    ```bash theme={null}
    snctl get pulsarinstance <your-instance-name> -O <your-org-id>
    ```

    You should be able to get the details of the instance.

    ```yaml theme={null}
    spec:
      auth:
        apikey: {}
      availabilityMode: regional
      poolRef:
        name: shared-gcp
        namespace: streamnative
      type: serverless
    status:
      auth:
        oauth2:
          audience: urn:sn:pulsar:<your-org-id>:<your-instance-name>
          issuerURL: https://auth.streamnative.cloud/
        type: oauth2
      conditions:
        - lastTransitionTime: '2024-11-29T21:34:08Z'
          message: a payment method is not required because discount is active
          reason: HasActiveDiscount
          status: 'True'
          type: SubscriptionReady
        - lastTransitionTime: '2024-11-29T21:34:08Z'
          reason: Created
          status: 'True'
          type: ResourceServerReady
        - lastTransitionTime: '2024-11-29T21:34:09Z'
          reason: Created
          status: 'True'
          type: ServiceAccountReady
        - lastTransitionTime: '2024-11-29T21:34:10Z'
          reason: AllConditionStatusTrue
          status: 'True'
          type: Ready
    ```

    * `status.auth.oauth2.audience`: The audience of the Pulsar instance. It will be used for the [Oauth2 authentication](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview).
    * `status.auth.oauth2.issuerURL`: The oauth2 issuer URL of the Pulsar instance. It will be used for the [Oauth2 authentication](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview).
    * In the `conditions` section, you can see the status of the instance. If all conditions are `True`, the instance is ready.
  </Tab>

  <Tab title="Terraform">
    If you want to get more details about an instance, you can define a data source in the Terraform configuration file.

    ```hcl theme={null}
    data "streamnative_pulsar_instance" "test-instance" {
      organization = "<your-organization>"
      name = "<your-instance-name>"
    }
    ```

    You should be able to get the details of the instance.

    * `id`: The ID of the instance.
    * `availability_mode`: The availability mode of the instance. Currently, only `regional` is supported.
    * `oauth2_audience`: The audience of the Pulsar instance. It will be used for the [Oauth2 authentication](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview).
    * `oauth2_issuer_url`: The oauth2 issuer URL of the Pulsar instance. It will be used for the [Oauth2 authentication](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview).
    * `pool_name`: The name of the infrastructure pool that runs the **Serverless** instance. Currently it only supports Google Cloud (`shared-gcp`).
    * `pool_namespace`: The namespace of the infrastructure pool (`streamnative`).
    * `ready`: The status of the instance. If the Pulsar instance is ready, it is `True`.

    You can checkout [PulsarInstance](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/data-sources/pulsar_instance) for more information.
  </Tab>
</Tabs>

## Delete an instance

<Note title="Note">
  You cannot delete an instance if there are resources associated with the instance.
</Note>

<Tabs>
  <Tab title="Console">
    1. Navigate to the **Instances** page.

    2. Click the ellipsis at the top right corner of the instance card that you want to delete, and then click **Delete**.

    3. In the **Delete instance** dialog, enter the instance name and then click **Confirm**.
  </Tab>

  <Tab title="snctl">
    There are two ways to delete an instance.

    * Delete the instance by the instance name.

      ```bash theme={null}
      snctl delete pulsarinstance <your-instance-name>
      ```

    * Delete the instance by the instance manifest file `instance.yaml`.

      ```bash theme={null}
      snctl delete -f instance.yaml
      ```
  </Tab>

  <Tab title="Terraform">
    Remove the instance resource from the Terraform configuration file and run `terraform apply` to delete the instance.
  </Tab>
</Tabs>

## Next steps

* [Work with clusters](/cloud/clusters/manage-clusters/cluster)

## Related topics

* Check other types of instances:
  * [Dedicated Instances](/cloud/clusters/manage-instances/manage-dedicated-instances)
  * [BYOC Instances](/cloud/clusters/manage-instances/manage-byoc-instances)


# Networking in StreamNative Cloud
Source: https://docs.streamnative.io/cloud/networking/networking



StreamNative Cloud supports the following networking solutions:

* BYOC Clusters are accessible through secure public endpoints or Private Link connections.

  BYOC clusters support only one endpoint. If you use Private Link, your cluster will not have public endpoints, and you can access your cluster only from Private Endpoints in accounts you have registered with StreamNative Cloud.

* BYOC Pro clusters are accessible through secure public endpoints, Private Link connections, VPC/VNet peering, or AWS Transit Gateway.

  BYOC Pro clusters can have multiple endpoints, including both public and private endpoints.

* Serverless and Dedicated clusters are accessible through secure public endpoints.

All connections to StreamNative Cloud are encrypted with TLS 1.2 and require authentication using [OAuth2](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview) or [API keys](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview), regardless of network configuration.

After a cluster has been provisioned, you cannot change its networking solution type between public and private.

StreamNative Cloud uses the following ports and protocols for StreamNative Cloud services:

**Control Plane Services**

| Service                    | Port | Protocol |
| -------------------------- | ---- | -------- |
| StreamNative Cloud Console | 443  | HTTPS    |
| StreamNative Cloud API     | 443  | HTTPS    |
| StreamNative Metrics API   | 443  | HTTPS    |

**Data Plane Services**

| Service                     | Port | Protocol |
| --------------------------- | ---- | -------- |
| Pulsar Broker Service       | 6651 | TLS      |
| Pulsar HTTPS Service        | 443  | HTTPS    |
| Pulsar Websocket Service    | 443  | TLS      |
| Kafka Broker Service        | 9093 | TLS      |
| Kafka Schema Registry       | 443  | HTTPS    |
| Kafka Connect Admin Service | 443  | HTTPS    |
| MQTT Service                | 8883 | TLS      |

## Considerations for public vs. private networking type

Using a private or public connectivity with StreamNative Cloud is a trade-off:

* With private networking, your cluster cannot be accessed from the public endpoints, eliminating potential security threats.

* Private networking requires you to manage the peered or linked networks to ensure all your client applications and developers have the needed access to StreamNative Cloud.

* If you use private networking (VPC peering, VNet peering, or Private Links), you cannot directly connect from your local laptop or an on-premises data center to StreamNative Cloud.

  To do this, you must first route to a shared services VPC or VNet that you own and connect that to StreamNative Cloud using VPC/VNet peering (along with a proxy) or Private Link.

  If you are interested in this configuration for StreamNative Cloud, contact your StreamNative sales representative.

* IP addresses for ingress public endpoints are not static. These ingress public endpoints include the endpoints of each StreamNative cluster, such as the Pulsar broker service, Pulsar admin service, Pulsar websocket service, Kafka broker service, Kafka schema registry, Kafka Connect admin service, and MQTT service. They also include the endpoints for StreamNative's control plane services.

  The endpoints can assume any public IP, without a specific range.

* For outbound traffic from StreamNative Cloud, [Static Egress IP](/cloud/networking/static-egress-ip) provides fixed IP addresses that you can use to configure firewall rules and network allowlists in your own environments.

* Native Pulsar or Kafka clients are not designed to work seamlessly in forward proxy environments. If you are producing HTTPs records, consider using the [REST API](/cloud/build/pulsar-clients/connect-restapi) instead.

## Public networking solutions

StreamNative Cloud provides secure and scalable data streaming services accessible via public endpoints. This public connectivity is available for all cluster types, offering flexibility and ease of access. Key features include:

1. **Cross-organization sharing**: Services can be securely shared across different organizations.

2. **Availability**: Public connectivity is available for all cluster types.

3. **BYOC flexibility**: For BYOC clusters, public endpoints can be disabled if private networking is preferred.

4. **Enhanced security**: All public endpoints are protected by a robust proxy layer, which provides defense against various network-level threats, including:
   * Denial of Service (DoS) attacks
   * Distributed Denial of Service (DDoS) attacks
   * SYN flooding
   * Other common network-level attack vectors

This combination of accessibility and security measures ensures that StreamNative Cloud can meet diverse organizational needs while maintaining a strong security posture.

## Private networking solutions

StreamNative Cloud includes support for data streaming services that are shared privately with organizations on private networks and offer additional customization and controls for security and privacy. Private networking are currently supported in StreamNative BYOC (& BYOC Pro) clusters only.

StreamNative Cloud clusters using private networking solutions are not accessible from the public endpoints.

The following table summarizes the private networking solutions supported by StreamNative Cloud. For details on each solution, see the specific documentation for each networking type.

| Cloud service provider | Supported networing solution                                                                    | Supported cluster type |
| ---------------------- | ----------------------------------------------------------------------------------------------- | ---------------------- |
| AWS                    | [AWS PrivateLink](/cloud/networking/networking-on-aws/aws-privatelink/aws-privatelink-overview) | BYOC, BYOC Pro         |
|                        | AWS VPC peering                                                                                 | BYOC Pro               |
|                        | AWS Transit Gateway                                                                             | BYOC Pro               |
| Azure                  | Azure Private Link                                                                              | BYOC, BYOC Pro         |
|                        | Azure VNet peering                                                                              | BYOC Pro               |
| Google Cloud           | Google Cloud Private Service Connect                                                            | BYOC, BYOC Pro         |
|                        | Google Cloud VPC peering                                                                        | BYOC Pro               |


# Static Egress IP
Source: https://docs.streamnative.io/cloud/networking/static-egress-ip

Configure static egress IP addresses for outbound traffic from StreamNative Cloud

## What is Static Egress IP?

Static Egress IP provides a fixed, predictable IP address for outbound traffic originating from StreamNative Cloud. Instead of dynamic IP addresses that change over time, your environment uses the same IP address for all outbound connections, making it easier to:

* Configure firewall rules on your side
* Set up network allowlists for external services
* Maintain consistent security policies
* Simplify network monitoring and logging

## Supported Traffic Types

Static Egress IP applies to outbound data flows including:

* **Connectors**: Both source and sink connectors that connect to external systems
* **Pulsar Functions**: Functions that make outbound network calls to external APIs or services
* **Other outbound traffic**: Any network traffic initiated from your StreamNative Cloud environment

<Note>
  Static Egress IP only affects outbound (egress) traffic from StreamNative Cloud to external systems. It does not affect inbound (ingress) traffic to your environment.
</Note>

## Availability

### Cluster Types

Static Egress IP is **enabled by default** for:

* **All new Dedicated clusters**
* **BYOC clusters**
* **Serverless clusters**

<Warning>
  Some older legacy clusters may not have Static Egress IP enabled. If you have a legacy cluster that needs this feature, contact StreamNative support to enable it.
</Warning>

### Cloud Provider Support

| Cloud Provider | Status                                                                      |
| -------------- | --------------------------------------------------------------------------- |
| AWS            | ✅ Enabled by default                                                        |
| Azure          | ✅ Enabled by default                                                        |
| Alibaba Cloud  | ✅ Enabled by default                                                        |
| GCP            | ✅ Enabled by default in new environments. Contact support for more details. |

## IP Address Scoping

Static egress IP addresses are assigned at the environment level. All clusters within the same environment share the same static egress IP address:

* **BYOC**: All clusters in the same BYOC environment share one static egress IP
* **Dedicated**: All clusters in the same pool and region (e.g., `aws/us-east-1`) share one static egress IP
* **Serverless**: All clusters in the same pool and region (e.g., `gcp/us-west-1`) share one static egress IP

<Note>
  The static egress IP belongs to the environment (the underlying infrastructure including Kubernetes, networking, DNS, and applications). Clusters run within these environments and inherit the environment's static egress IP for their outbound traffic.
</Note>

## Accessing Your Static Egress IP

<Warning>
  Static egress IP addresses are currently not exposed in the StreamNative Cloud Console UI, CLI, or API. This functionality will be added in future releases.
</Warning>

To retrieve your static egress IP address:

**Open a support ticket** through the StreamNative support portal.

When contacting support, specify:

* Your environment ID or cluster name
* The purpose for needing the static egress IP (firewall configuration, allowlist setup, etc.)

The StreamNative team will provide the IP address so you can configure your firewall or network policies accordingly.

## Use Cases

### Firewall Configuration

Configure your firewall to allow traffic from StreamNative Cloud:

```bash theme={null}
# Example firewall rule (adjust for your firewall system)
# Allow traffic from StreamNative static egress IP
iptables -A INPUT -s <STATIC_EGRESS_IP> -j ACCEPT
```

### Database Allowlists

Add the static egress IP to your database allowlist:

```sql theme={null}
-- Example for PostgreSQL (adjust for your database)
-- Add StreamNative egress IP to allowed connections
-- Contact support to get your actual static egress IP
```

### API Gateway Configuration

Configure your API gateway to accept requests from StreamNative:

```yaml theme={null}
# Example API gateway configuration
allowedIPs:
  - "<STATIC_EGRESS_IP>"  # Replace with your actual IP from support
```

## Future Enhancements

StreamNative is working to make static egress IP addresses accessible through:

* **StreamNative Cloud Console UI**: View and manage egress IPs in the web interface
* **CLI (snctl)**: Retrieve egress IP information via command line
* **API**: Programmatic access to egress IP configuration

These features are planned for future releases. Check the [release notes](/release-notes) for updates.

## Related Topics

* [Networking Overview](/cloud/networking/networking): Learn about StreamNative Cloud networking architecture
* [AWS Networking](/cloud/networking/networking-on-aws/aws-networking-overview): AWS-specific networking configurations
* [Security Overview](/cloud/security/security-overview): Understand StreamNative Cloud security features


# Manage Pulsar ACLs
Source: https://docs.streamnative.io/cloud/security/access/access-control-lists/authorization-and-acls



In Pulsar, the authentication provider is responsible for properly identifying clients and associating the clients with role tokens. If you only enable authentication, an authenticated role token can be used to access all resources in the cluster. Authorization is the process that determines the operations performed by Pulsar clients.

Superusers have the role tokens with the most privileges. The superusers can create and destroy tenants, and have full access to all tenant resources. When a superuser creates a tenant, the tenant is assigned with an the administrator role. A client with the administrator role can create, modify and destroy namespaces, and grant and revoke permissions to other roles on these namespaces.

This topic describes how to authorize Pulsar components through the StreamNative Cloud Console. In addition, you can authorize Pulsar components through the `pulsar-admin` or `pulsar-perf CLI` tool. For details, see [pulsar-admin](https://pulsar.apache.org/reference/#/next/pulsar-admin/) and [pulsar-perf](https://pulsar.apache.org/reference/#/next/pulsar-perf/pulsar-perf).

## Authorize tenants

When you create a tenant, you can specify an administrator for the tenant.

1. On the left navigation pane, in the **Admin** section, click **Tenants**.

2. Click **New Tenant** and a dialog box displays.

   <img alt="screenshot of creating a tenant" />

3. In the **Add Roles** field, select a user or one or more service accounts as the administrator of the tenant.

4. Click **Confirm**.

In addition, you can add or remove an administrator for an existing tenant.

1. On the left navigation pane, in the **Admin** section, click **Tenants**.

2. Click **Edit** in the **Action** column.

3. In the **Add Role** field, select one or more administrators for the tenant.

## Authorize namespaces

To authorize a namespace through the StreamNative Cloud Console, follow these steps.

1. On the left navigation pane, in the **Admin** section, click **Namespaces**.

2. Select the **Policies** tab.

3. In the **Authorization** area, select a role for the namespace and then grant or revoke permissions to the role in this namespace.

   * consume: grant/revoke the consuming action.
   * produce: grant/revoke the producing action.
   * functions: grant/revoke the Pulsar functions action.

## Authorize topics

To authorize a topic through the StreamNative Cloud Console, follow these steps.

1. On the left navigation pane, in the **Resources** section, click **Topics**.

2. Click the topic name link.

3. If the topic is partitioned, in the **Partitions** area, click the partitioned topic link.

4. Select the **Policies** tab and configure the authorization policies for the topic.

   <img alt="screenshot of topic policies" />

5. In the **Authorization** area, select a role for the topic and then grant or revoke permissions to the role in this topic by adding or deleting the following:

   * consume: grant/revoke the consuming action.
   * produce: grant/revoke the producing action.
   * functions: grant/revoke the Pulsar functions action.


# Managed Kafka ACLs
Source: https://docs.streamnative.io/cloud/security/access/access-control-lists/kafka-acls



Kafka ACLs are the primary means of controlling access in a Kafka cluster. They enable Kafka administrators to define who can read from or write to a Kafka topic, who can create topics, and who can manage the cluster, among other actions. Each ACL contains a principal, a permission type, an operation, a resource type (e.g., cluster, topic, or group), and name.

Although StreamNative Cloud provides a fully compatible Kafka service at the protocol layer, it doesn't support Kafka ACLs directly. Instead, it uses Pulsar ACLs to control access to Kafka topics. This document describes how to map Kafka ACLs to [Pulsar ACLs](/cloud/security/access/access-control-lists/authorization-and-acls) and manage them on StreamNative Cloud.

## Understand Pulsar & Kafka ACLs

Both Kafka and Pulsar have access control lists (ACLs) to control access to resources. You grant permissions to principals (users or service accounts) to perform actions on resources.

Pulsar allows you to grant permissions to users or service accounts at the namespace level or topic level.

* If you grant the permissions at the namespace level, then the permissions apply to all the topics under the namespace.
* If you grant the permissions at the topic level, then the permissions apply to the specific topic.

### Understand Pulsar & Kafka actions

Pulsar supports the following authorization actions:

* produce
* consume
* functions
* sources
* sinks
* packages

Kafka has ACL operations similar to Pulsar authorization actions:

* READ
* IDEMPOTENT\_WRITE
* WRITE
* DESCRIBE
* CREATE
* DELETE
* ALTER
* DESCRIBE\_CONFIGS
* ANY
* ALTER\_CONFIGS
* CLUSTER\_ACTION
* UNKNOWN
* ALL

In Pulsar, you can use `pulsar-admin topics grant-permission` command to grant permissions to a topic. Here is an example on how to grant `produce` action to a topic:

```bash theme={null}
pulsar-admin topics grant-permission \
    --actions produce \
    --role alice \
    test-topic
```

Similarly, in Kafka, you can use `kafka-acls.sh` command to grant permissions to a topic. Here is an example on how to grant `WRITE` and `CREATE` operations to a topic:

```bash theme={null}
bin/kafka-acls.sh --bootstrap-server localhost:9092 \
  --add --allow-principal User:alice \
  --operation Write --operation Create --topic test-topic
```

## Mapping between Kafka and Pulsar ACLs

Because StreamNative Cloud doesn't support Kafka ACLs directly, you need to map Kafka ACLs to Pulsar authorization actions. Below table shows the mapping between Kafka ACL operations and Pulsar authorization actions.

<Note title="Note">
  StreamNative Cloud only supports the 'produce' and 'consume' actions for topics. The principal with **Super Admin** (also known as **Super User**) permission can perform all operations.
</Note>

| Kafka ACL Operation | Pulsar Authorization Action |
| ------------------- | --------------------------- |
| READ                | `consume`                   |
| WRITE               | `produce`                   |
| IDEMPOTENT\_WRITE   | `produce`                   |
| CREATE              | Super User                  |
| DELETE              | Super User                  |
| ALTER               | Super User                  |
| DESCRIBE            | `produce` or `consume`      |
| DESCRIBE\_CONFIGS   | `produce` or `consume`      |
| CLUSTER\_ACTION     | N/A                         |
| ALTER\_CONFIGS      | N/A                         |
| UNKNOWN             | N/A                         |
| ALL                 | Super User                  |
| ANY                 | Super User                  |

## Managed Pulsar ACLs using `pulsar-admin`

You can use `pulsar-admin` CLI to manage the ACLs, for example, grant `produce` and `consume` actions to role (aka principal) `test-role` in `test-tenant/namespace1` namespace.

```Bash theme={null}
pulsar-admin namespaces grant-permission test-tenant/namespace1 \
    --actions produce,consume \
    --role test-role
```

Here is another example that you can grant permissions on a client role to perform specific actions on a given topic.

```Bash theme={null}
pulsar-admin topics grant-permission \
    --actions produce,consume \
    --role test-role \
    persistent://test-tenant/namespace1/tp1
```

To learn more about how to manage Pulsar ACLs, see [Manage Pulsar ACLs](/cloud/security/access/access-control-lists/authorization-and-acls).


# Control Access to StreamNative Cloud
Source: https://docs.streamnative.io/cloud/security/access/access-control-overview



This section provides information on the key mechanisms for controlling access to StreamNative Cloud resources, including the StreamNative Cloud resource hierarchy, role-based access control (RBAC), and access control lists (ACLs).

## Resource hierarchy

StreamNative Cloud organizes resources in a hierarchical structure to manage access and permissions effectively. The top-level resource is the [organization](/cloud/security/access/resource-hierarchy/organizations), which can contain multiple instances. Each instance can include various resources, such as Pulsar/Kafka Clusters, Workspaces, and more. This hierarchy allows you to manage access and permissions at different levels, ensuring that you can control access to resources based on your organizational structure and security policies.

See [Resources on StreamNative Cloud](/cloud/security/access/resource-hierarchy/resources-overview) for more details.

## Role-based access control (RBAC)

Role-based access control (RBAC) allows you to manage access to StreamNative Cloud resources by assigning predefined roles to users, service accounts, and other entities. Each role is associated with a set of permissions that determine the actions that can be performed on specific resources. By using RBAC, you can control access to resources based on roles and responsibilities, ensuring that only authorized users can perform certain actions.

See [Role-Based Access Control](/cloud/security/access/rbac/rbac-overview) for more details.

## Access control lists (ACLs)

Access control lists (ACLs) provide fine-grained access control to Pulsar resources in StreamNative Cloud. ACLs allow you to control which users or service accounts can perform specific actions on Pulsar resources, such as tenants, namespaces, topics, and more. By using ACLs, you can ensure that only authorized users can access and manage Pulsar resources, enhancing security and preventing unauthorized access. ACLs can be used in conjunction with RBAC to provide a comprehensive access control mechanism.

See [Authorization and ACLs](/cloud/security/access/access-control-lists/authorization-and-acls) for more details.


# Role-Based Access Control (V1) (Deprecated)
Source: https://docs.streamnative.io/cloud/security/access/rbac/cloud-rbac



<Note title="Note">
  This documentation covers RBAC Version 1. For information about RBAC Version 2, see [RBAC](/cloud/security/access/rbac/rbac-overview).

  This feature is deprecated, please use [RBAC V2](/cloud/security/access/rbac/rbac-overview).
</Note>

# Overview

Role-based access control (RBAC) allows you to control what level of access users have to your organization's resources, such as Pulsar clusters, tenants, and service accounts.

RBAC provides standardized ways to control resources in the StreamNative console as well as in Pulsar Clusters provided by StreamNative. A role can be thought of as a group of permissions. A user can be granted a role. These users that have a particular role can do certain actions and are not allowed to do others. They experience different UI experiences based on their permissions. They might not have direct access to Pulsar or are unable to perform certain operations on Pulsar. All roles are granted on an individual instance basis. The user who creates the organization is **always** an organization admin and is the billing account associated with that organization. This cannot be changed.

# Roles

The current roles that StreamNative provides are the following:

* admin
* read-only
* tenant-admin
* basic

## Organization Admin

Organization Admins have full control over everything in their organization. Users granted the Organization Admin role have the following capabilities:

* Viewing and managing [billing details](/cloud/billing/billing-overview)
* Viewing and managing [secrets](/cloud/security/secret)
* Inviting users to your organization, deleting users, and modifying their roles.
* Managing Pulsar Instances and Pulsar Clusters, **including deleting Pulsar clusters**.

## Read Only

The read only role allows users to read information from the StreamNative Console, but disallows the modification or deletion of any resources. Users with the Read Only role have the following capabilities:

* Peek at topics.
* View clusters, tenants, and namespaces.
* View users.
* View functions and connectors.

## Tenant Admin

The Tenant Admin role grants full administrative capabilities over a specific tenant. Tenant Admins have the following capabilities:

* Create, modify, and delete namespaces within your tenant.

## Basic Role

The Basic Role is the default role assigned to all users when they are invited to an organization. It cannot be removed from any user - if you need to reduce permissions further beyond this, you must delete a user from your organization. To access additional details in the console, you must assign users additional roles.
The basic role grants access to the following resources in the organization:

* Pulsar Instances (list and read access)
* Pulsar Clusters (list and read access)

No other permissions are granted to this role.

# StreamNative Cloud Resources

The resources that StreamNative provides as part of its cloud services are the following:

* Organization
* Organization Usage
* Pulsar Instance
* Pulsar Cluster
* Secrets
* Service Accounts
* Generated Billing Links
* Metrics
* Users
* Rolebinding

A table indicating the level of access each role has to the resources is below.

| Role Name    | Read                                 | List                                 | Create                                    | Delete                                    | Update                                    |
| ------------ | ------------------------------------ | ------------------------------------ | ----------------------------------------- | ----------------------------------------- | ----------------------------------------- |
| admin        | all                                  | all                                  | all                                       | all except Organization                   | all                                       |
| read-only    | all except Secrets, Service Accounts | all except Secrets, Service Accounts | none                                      | none                                      | none                                      |
| tenant-admin | Pulsar Instance, Pulsar Cluster      | Pulsar Instance, Pulsar Cluster      | no cloud resources, only pulsar resources | no cloud resources, only pulsar resources | no cloud resources, only pulsar resources |
| basic        | Pulsar Instance, Pulsar Cluster      | Pulsar Instance, Pulsar Cluster      | none                                      | none                                      | none                                      |

# Pulsar Resources

Pulsar provides some resources listed below. Tenants are the basis of the current authorization scheme for Pulsar. Pulsar Super Users are granted full access to the cluster including the ability to list tenants and manipulate them all. Individual principals (users) can be granted access to an individual tenant and then are able to manipulate the hierarchy under that tenant as shown below. They are not able to manipulate the cluster itself.

* Super Users (Cluster Administrators)
* Tenant
  * Can be replicated
  * Lowest current measure of Pulsar control
* Namespace
  * policies
* Topic
  * policies
  * Namespace Bundle
* Message
* Principal (access account)
* Function
* Sink
* Source
* pfSQL

A table indicating the level of access each role has to the resources is below.

| Role Name    | Read                            | List                            | Create                                                   | Delete                                                   | Update                                                   |
| ------------ | ------------------------------- | ------------------------------- | -------------------------------------------------------- | -------------------------------------------------------- | -------------------------------------------------------- |
| admin        | all                             | all                             | all                                                      | all                                                      | all                                                      |
| read-only    | all                             | all                             | none                                                     | none                                                     | none                                                     |
| tenant-admin | only the tenant they administer | only the tenant they administer | only namespaces and topics in the tenant they administer | only namespaces and topics in the tenant they administer | only namespaces and topics in the tenant they administer |
| basic        | none                            | none                            | none                                                     | none                                                     | none                                                     |

# Roles and Role Stacking

As mentioned before, roles can be stacked. As such, when you have a role stack, you have multiple roles assigned to a user. All users have at least the basic role, so all abilities stack from there. The abilities granted may be revoked at any time in the console by removing the role associated with that user from the user console. Roles grant abilities on Pulsar as well as on our Cloud objects. An example, but not comprehensive, stack of role abilities is shown below.

### Cloud Abilities

| Role Stack                                 | Read                                | List                                | Create | Delete | Update |
| ------------------------------------------ | ----------------------------------- | ----------------------------------- | ------ | ------ | ------ |
| admin + any role                           | all                                 | all                                 | all    | all    | all    |
| basic+read-only+tenant-admin to one tenant | all except Secrets, ServiceAccounts | all except Secrets, ServiceAccounts | none   | none   | none   |

### Pulsar Abilities

| Role Stack                                 | Read | List | Create                                                   | Delete                                                   | Update                                                   |
| ------------------------------------------ | ---- | ---- | -------------------------------------------------------- | -------------------------------------------------------- | -------------------------------------------------------- |
| admin + any role                           | all  | all  | all                                                      | all                                                      | all                                                      |
| basic+read-only+tenant-admin to one tenant | all  | all  | only namespaces and topics in the tenant they administer | only namespaces and topics in the tenant they administer | only namespaces and topics in the tenant they administer |

As you can see from the tables, the role with the highest abilities is what is granted when stacking roles.

# The Console

The console experience is different for each role. The UI items listed are reflected to the abilities that the role is granted. Some items are hidden and will never display, some things will only display if a user is granted a particular role. Some users, for example, that are tenant admins will only see certain tenants in a cluster. This is because they only have access to those tenants and no others.

# `snctl pulsar admin`, PulsarCtl and Pulsar-Admin

These CLI tools use direct Pulsar permissions. These permissions will need to be granted on top of the RBAC roles granted in the console. An example of granting access to a tenant on the pulsar cluster level is shown below:

```
shCopy code
$ pulsarctl tenants create my-tenant --admin-roles my-tenant-admin

# The principal of my-tenant-admin needs to match the principal used in pulsarctl for this to work

```

UI operates on instances which include all clusters under that instance. manipulating individual clusters is not a path forward we support in the UI and unexpected behavior will occur

# Inviting Users

To invite a user, go to the users page, and click on the "Invite User" button. This will open up a modal where you can enter in a user's email.

<img alt="tenant admin" />

You must first enter a valid email in order to assign them a role. If the user is already in your organization, you cannot invite them again; you must instead [modify their role](/cloud/security/access/rbac/cloud-rbac#modifying-roles).

To assign a user Tenant Admin capabilities, you must select the tenant(s) you want to give the user by clicking the `+` button, and selecting the tenants you want the user to administer.

<img alt="tenant admin" />

# Modifying Roles

You must be an Organization Admin to modify roles.

To modify a user's roles, go to the users page, and click on the ellipses menu (`...`), and click Edit Roles. This opens up the Edit Role menu, where roles can be added by clicking the checkbox next to a role.

# Best Practices

* We recommend limiting the number of Organization Admins in your Organization as much as possible.
* As a default, we recommend giving most users the read only role.

# Limitations

Currently, StreamNative RBAC does not support role assignment via `snctl`.

**Can roles be combined?**
Yes - roles can be combined by assigning a user multiple roles within the invite / edit user modal.

# Application Resources

<Note title="Prerequisite">
  Application Resources require Pulsar 3.2.2.7 or later.
</Note>

## Getting Started

1. Download [snctl](https://docs.streamnative.io/docs/snctl-overview) and login to StreamNative Cloud with your organization.
2. Create a [service account](https://docs.streamnative.io/docs/service-accounts)
3. Create a [role](#role-management) and wait for status to be ready.
4. Create a [role binding](#role-binding-management) and wait for status to
   be ready.
5. Configure your CLI, you may choose to use `snctl` or `pulsarctl`.
   a. Configure your [snctl](https://docs.streamnative.io/streamnative-cli/snctl-overview) with service context and service account identity.
   b. Configure your [pulsarctl](https://docs.streamnative.io/docs/pulsarctl-overview#configure-pulsarctl) with new service account identity.
6. Enjoy the feature! :)

<Note title="How to Ensure the Resource Is Ready">
  "wait for status to be ready" means you should `Get` this resource to check the status of it.

  E.g:

  ```yaml theme={null}
  apiVersion: cloud.streamnative.io/v1alpha1
  kind: Role
  generation: 6
  name: application-describer
  namespace: <namespace>
  spec:
    permissions:
      - pulsar.namespaces.describe
      - pulsar.topics.describe
      - pulsar.subscriptions.describe
      - pulsar.policies.describe
  status:
    conditions:
      - lastTransitionTime: '2024-05-11T08:04:12Z'
        message: Ready
        observedGeneration: 6
        status: 'True'
        type: Ready
  ```

  * The `generation` means the resource current version.
  * The condition `observedGeneration` means applied resource version
  * The condition type `Ready`'s `status` means the applied status. applied or not applied.
</Note>

## Role Management

<Note title="Note">
  The role modifier permission is not exposed to end users. [Contact StreamNative](https://support.streamnative.io/hc/en-us) to manage roles that require this permission.
</Note>

### Snctl

* **List**

```shell theme={null}
snctl get role
```

* **Get**

```shell theme={null}
snctl get role <ROLE_NAME> -o yaml
```

### Permission Matrix

| Permissions                   | Operation-Allowed (\* means wildcard)                                                                                                                                                                                                                                                                                                                                                   |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| pulsar.namespaces.describe    | pulsar.tenant\_operation.list\_namespace,pulsar.tenant\_operation.get\_bundle                                                                                                                                                                                                                                                                                                           |
| pulsar.namespaces.create      | pulsar.tenant\_operation.create\_namespace                                                                                                                                                                                                                                                                                                                                              |
| pulsar.namespaces.delete      | pulsar.tenant\_operation.delete\_namespace                                                                                                                                                                                                                                                                                                                                              |
| pulsar.namespaces.alter       | pulsar.namespace\_operation.add\_bundle,pulsar.namespace\_operation.delete\_bundle,pulsar.namespace\_operation.clear\_backlog                                                                                                                                                                                                                                                           |
| pulsar.topics.create          | pulsar.namespace\_operation.create\_topic                                                                                                                                                                                                                                                                                                                                               |
| pulsar.topics.describe        | pulsar.topic\_operation.lookup,pulsar.topic\_operation.get\_topic,pulsar.topic\_operation.get\_topics,pulsar.topic\_operation.get\_bundle\_range,pulsar.topic\_operation.get\_metadata,pulsar.topic\_operation.get\_backlog\_size,pulsar.topic\_operation.get\_stats                                                                                                                    |
| pulsar.topics.delete          | pulsar.namespace\_operation.delete\_topic                                                                                                                                                                                                                                                                                                                                               |
| pulsar.topics.alter           | pulsar.topic\_operation.compact,pulsar.topic\_operation.offload,pulsar.topic\_operation.unload,pulsar.topic\_operation.add\_bundle\_range,pulsar.topic\_operation.terminate,pulsar.topic\_operation.delete\_bundle\_range,pulsar.topic\_operation.delete\_metadata,pulsar.topic\_operation.update\_metadata,pulsar.namespace\_operation.trim\_topic,pulsar.topic\_operation.trim\_topic |
| pulsar.messages.produce       | pulsar.topic\_operation.lookup,pulsar.topic\_operation.produce                                                                                                                                                                                                                                                                                                                          |
| pulsar.messages.consume       | pulsar.topic\_operation.lookup,pulsar.topic\_operation.consume,pulsar.topic\_operation.subscribe,pulsar.namespace\_operation.unsubscribe,pulsar.topic\_operation.unsubscribe,pulsar.topic\_operation.consume,pulsar.topic\_operation.peek\_messages                                                                                                                                     |
| pulsar.subscriptions.create   | pulsar.topic\_operation.subscribe                                                                                                                                                                                                                                                                                                                                                       |
| pulsar.subscriptions.delete   | pulsar.topic\_operation.unsubscribe,pulsar.namespace\_operation.unsubscribe                                                                                                                                                                                                                                                                                                             |
| pulsar.subscriptions.alter    | pulsar.topic\_operation.expired\_messages,pulsar.topic\_operation.reset\_cursor,pulsar.topic\_operation.skip,pulsar.topic\_operation.set\_replicated\_subscription\_status                                                                                                                                                                                                              |
| pulsar.subscriptions.describe | pulsar.topic\_operation.get\_subscriptions,pulsar.topic\_operation.get\_replicated\_subscription\_status,pulsar.topic\_operation.lookup                                                                                                                                                                                                                                                 |
| pulsar.policies.describe      | pulsar.policy*operation*\*.read                                                                                                                                                                                                                                                                                                                                                         |
| pulsar.policies.alter         | pulsar.policy*operation*\*.write                                                                                                                                                                                                                                                                                                                                                        |

## Role Binding Management

You can bind a service account or user to specific role with permissions.

### Template Example

```yaml theme={null}
apiVersion: cloud.streamnative.io/v1alpha1
kind: RoleBinding
metadata:
  name: client-1
  namespace: <namespace>
spec:
  roleRef:
    apiGroup: cloud.streamnative.io
    kind: Role
    name: application-describer
  subjects:
    - apiGroup: cloud.streamnative.io
      kind: ServiceAccount
      name: client-1
```

### Snctl

* **Create**

```shell theme={null}
snctl create rolebinding <ROLE_BINDING_NAME> --role ROLE_NAME --serviceaccount <SERVICE_ACCOUNT_NAME>
```

* **List**

```shell theme={null}
snctl get rolebinding
```

* **Get**

```shell theme={null}
snctl get rolebinding <ROLE_BINDING_NAME> -o yaml
```

* **Delete**

```shell theme={null}
snctl delete rolebinding <ROLE_BINDING_NAME>
```

* **Apply**

```shell theme={null}
snctl apply -f role_binding_name.yaml
```

## Role Binding With Condition Management

With RBAC Conditions, you can choose to grant access to principals only if specified conditions are met. For example, you could grant access only to specific tenants or namespaces.

### Template Example

```yaml theme={null}
apiVersion: cloud.streamnative.io/v1alpha1
kind: RoleBinding
metadata:
  name: client-1
  namespace: <namespace>
spec:
  roleRef:
    apiGroup: cloud.streamnative.io
    kind: Role
    name: application-describer
  subjects:
    - apiGroup: cloud.streamnative.io
      kind: ServiceAccount
      name: client-1
  conditionGroup:
    relation: 1 # and
    conditions:
      - type: 0 # srn
        operator: 0 # key-match
        srn:
          tenant: public
```

### Snctl

* **Create**

```shell theme={null}
snctl create rolebinding <ROLE_BINDING_NAME> --role ROLE_NAME --serviceaccount <SERVICE_ACCOUNT_NAME>
```

based on the command we have several optional options for binding condition:

```shell theme={null}
--srn-instance string           condition srn instance
--srn-cluster string            condition srn cluster
--srn-tenant string             condition srn tenant
--srn-namespace string          condition srn namespace
--srn-topic string              condition srn topic name
--srn-topic-domain string       condition srn topic domain
--srn-subscription string       condition srn subscription name
```

* **Get**

```shell theme={null}
snctl get rolebinding <ROLE_BINDING_NAME> -o yaml
```

* **Delete**

```shell theme={null}
snctl delete rolebinding <ROLE_BINDING_NAME>
```

* **Apply**

```shell theme={null}
snctl apply -f role_binding_name.yaml
```

### Condition Concept

Conditions are specified in the role bindings of a resource's allow policy. When a condition exists, the access request is granted only if the condition expression is evaluated as true. Each condition expression is a set of logic statements that specify one or more attributes to check.

### Condition Format

```json theme={null}
{
  relation: <relation_code>
  conditions: [
    {
      type: <condition_type_code>,
      operator: <condition_type_suported_operator>,
      <condition_entity_key>: <condition_entity_body>
    }
  ]
  relationGroups: [
    {
      relation: <relation_code>
      conditions: [
        {
          type: <condition_type_code>,
          operator: <condition_type_suported_operator>,
          <condition_entity_key>: <condition_entity_body>
        }
      ]
      relationGroups: [
        {
          ...
        }
        // here's recursive, the structure will be a tree
      ]
    }
  ]
}
```

The condition format includes several parts:

* Relation: the relation defines the logical relationship between conditions.
  * The AND(1) relation needs all the conditions to be TRUE.
  * The OR(0) relation only needs one of the conditions to be TURE.
* Conditions: the set of conditions. You can check here to get supported conditions and format.
* RelationGroups(not supported yet): This field allowed us to create a logical tree. For example, some users might need
  a condition expression like this - (conditionA && conditionB) || (conditionC && condition D)

### Conditions

#### StreamNative Resource Name Condition

**Type: SRN(0)**

The StreamNative Resource Name condition’s type is SRN, whose type code is 0.

**Operator: key\_match(0) / regex\_match(1)**

The condition supports two operators:

1. KeyMatch: the key match mode supports using the wildcard(\*) in any level of SRN to indicate **ANY** meaning. The operator code is 0.
2. RegexMatch: the regex match mode supports regex expression to match in any level of SRN. The operator code is 1. (not supported yet)

The SRN is a unified resource name to indicate a specific resource. The format looks like this:

```javascript theme={null}
// json format
{
 "schema":"srn",
 "version": "v1",
 "organization": "<organization>",
 "instance": "<instance>",
 "cluster" : "<cluster>",
 "tenant"  : "<tenant>",
 "namespace": "<namespace>",
 "topicDomain" : "<topic_domain>(persistent/non-persistent)",
 "topicName" : "<topic_name>"
}

// string format

srn://v1/<organization>/<instance>/<cluster>/<tenant>/<namespace>/<topic_domain>/<topic_name>
```


# Manage Role Bindings on StreamNative Cloud
Source: https://docs.streamnative.io/cloud/security/access/rbac/manage-rbac-role-bindings



This guide covers both basic and advanced techniques for managing role bindings. Recommend to reviewing the [Predefined Role](/cloud/security/access/rbac/manage-rbac-roles) for a complete list of available roles and for examples of how to binding them.

You can manage role bindings by using [snctl](/tools/cli/snctl/snctl-overview) or [StreamNative Terraform Provider](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs). Support for the Cloud Console will be available soon.

## Role Bindings

Role bindings are used to bind roles to principals. They are defined as `RoleBinding` resources in the Cloud API. The schema is as follows:

```yaml theme={null}
apiVersion: cloud.streamnative.io/v1alpha1
kind: RoleBinding
metadata:
  name: <name>
  namespace: <namespace>
spec:
  resourceNameRestriction:
    common:
      instance: <instanceName>
      cluster: <clusterID>
      tenant: <tenantName>
      namespace: <namespaceName>
  roleRef:
    apiGroup: cloud.streamnative.io
    kind: ClusterRole
    name: <roleName>
  subjects:
    - apiGroup: cloud.streamnative.io
      kind: <subject type> # User, ServiceAccount, Identity Pool
      name: <subject name>
```

* `roleRef`: Reference to the **[Predefined Role](/cloud/security/access/rbac/manage-rbac-roles)**.
* `subjects`: List of subjects (also known as principals) to be bound to the role. It can be a \[**User Account**]]\(/cloud/security/authentication/user-accounts), a [**ServiceAccount**](/cloud/security/authentication/service-accounts/service-accounts), or an [**IdentityPool**](/cloud/security/authentication/oidc-identity-providers/manage-oidc-identity-pools).
* `resourceNameRestriction`: Optional field to restrict the role binding to specific resources. It can be used to limit the scope of the role binding to a specific resource. Refer to the [Conditional Role Bindings](#conditional-role-bindings) section for more details.

## Create Role Bindings

You can create a role binding by using the following methods:

<Note title="Note">
  Each organization can have a maximum of 1,500 role bindings. If you have reached or exceeded the limit, you must delete unused role bindings before creating new ones.
</Note>

<Tabs>
  <Tab title="snctl">
    You can create a role binding by running the following command to bind a predefined role `<predefined-role-name>` to a service account `<service-account-name>`.

    ```bash theme={null}
    snctl create rolebinding <role-binding-name> \
      --clusterrole <predefined-role-name> \
      --serviceaccount <service-account-name>
    ```

    Alternatively, you can prepare the manifest file `rolebinding.yaml` to bind a predefined role to a service account.

    ```yaml theme={null}
    apiVersion: cloud.streamnative.io/v1alpha1
    kind: RoleBinding
    metadata:
      name: <name>
      namespace: <namespace>
    spec:
      roleRef:
        apiGroup: cloud.streamnative.io
        kind: ClusterRole
        name: <predefined-role-name>
      subjects:
        - apiGroup: cloud.streamnative.io
          kind: ServiceAccount
          name: <service-account-name>
    ```

    Then apply it using `snctl apply`.

    ```bash theme={null}
    snctl apply -f rolebinding.yaml
    ```

    After creating the role binding, you can verify it by running the following command:

    ```bash theme={null}
    snctl get rolebinding <name>
    ```

    You should be able to see the role binding is in the `Ready` state.
  </Tab>

  <Tab title="Terraform">
    You can create a role binding by preparing the role binding definition in the Terraform configuration and applying the changes. Below is an example of binding a predefined role to a service account.

    ```hcl theme={null}
    terraform {
      required_providers {
        streamnative = {
          source = "streamnative/streamnative"
        }
      }
    }


    provider "streamnative" {
        # Please replace path use your own key file path
        key_file_path = "/path/to/your/service/account/key.json"
    }


    resource "streamnative_service_account" "metrics-account" {
      organization = "<organization-id>"
      name = "<service-account-name>"
      admin = false
    }


    resource "streamnative_rolebinding" "metrics-viewer" {
      organization          = "<organization-id>"
      name                  = "metrics-viewer"
      cluster_role_name     = "<predefined-role-name>"
      service_account_names = ["<service-account-name>"]
    }
    ```

    Please replace the placeholders with your actual values.

    * `<organization-id>`: The ID of your organization.
    * `<predefined-role-name>`: The name of the predefined role.
    * `<service-account-name>`: The name of the service account.

    After preparing the Terraform configuration, you can apply the changes using the following command:

    ```bash theme={null}
    terraform init
    terraform validate
    terraform plan
    terraform apply
    ```
  </Tab>
</Tabs>

## Update Role Bindings

You can update a role binding by using the following methods:

<Tabs>
  <Tab title="snctl">
    You can use `snctl edit` to update a role binding directly.

    ```bash theme={null}
    snctl edit rolebinding <name>
    ```

    Alternatively, you can update the file `rolebinding.yaml` and apply it using `snctl apply`.

    ```bash theme={null}
    snctl apply -f rolebinding.yaml
    ```
  </Tab>

  <Tab title="Terraform">
    You can update the role binding definition in the Terraform configuration and apply the changes.
  </Tab>
</Tabs>

## Delete Role Bindings

You can delete a role binding by using the following methods:

<Tabs>
  <Tab title="snctl">
    Delete a role binding:

    ```bash theme={null}
    snctl delete rolebinding <name>
    ```
  </Tab>

  <Tab title="Terraform">
    You can simply remove the role binding definition from the Terraform configuration and apply the changes.
  </Tab>
</Tabs>

## Query Role Bindings

You can efficiently query for RoleBinding resources by using a label selector with the -l flag.

### Search Role Bindings By Role Name

To find all Role Bindings associated with a specific role, use the `rolebinding.role` label.

-`${role_name}` refers to a [predefined role](/cloud/security/access/rbac/manage-rbac-roles#quick-reference).

```bash theme={null}
snctl get rolebinding -l rolebinding.role=tenant-admin
```

### Search Role Bindings By Account Name

To find all Role Bindings assigned to a specific user or service account, use the `rolebinding.subject` label.

The value for the `rolebinding.subject` label must be a sanitized version of the account name. You must replace the **@** symbol with an underscore **\_**.

```bash theme={null}
snctl get rolebinding -l rolebinding.subject=user_example.com
```

## Conditional Role Bindings

While basic role bindings associate a role with an account across the entire organization, conditional role bindings provide more granular control by scoping permissions to specific resources.

For example, you may want to restrict a `topic-producer` role to only work within a specific namespace, or on topics with names that start with a certain prefix. StreamNative Cloud allows you to express these conditions by specifying resource attributes.

The following example shows how to bind the `topic-producer` role to a service account named `service-account-1` with conditions that limit its access to:

* Instance: `ins-a`
* Cluster: `cluster-a`
* Tenant: `tenant-a`
* Namespace: `ns-a`

<Tabs>
  <Tab title="snctl">
    ```bash theme={null}
    snctl create rolebinding <role-binding-name> \
       --clusterrole topic-producer \
       --serviceaccount service-account-1 \
       --resource-common-instance ins-a \
       --resource-common-cluster cluster-a \
       --resource-common-tenant tenant-a \
       --resource-common-namespace ns-a
    ```
  </Tab>

  <Tab title="Terraform">
    ```hcl theme={null}
    resource "streamnative_rolebinding" "user-a-topic-xxx-producer" {
      organization          = "<organization-id>"
      name                  = "<role-binding-name>"
      cluster_role_name     = "topic-producer"
      service_account_names = ["service-account-1"]

      resource_name_restriction {
        common_instance  = "ins-a"
        common_cluster   = "cluster-a"
        common_tenant    = "tenant-a"
        common_namespace = "ns-a"
      }
    }
    ```
  </Tab>
</Tabs>

With these conditions, `service-account-1` can only produce messages to topics within the specified namespace (`tenant-a/ns-a`) on that particular instance and cluster.

### Available Resource Attributes

You can set conditions on the following resource attributes when creating a role binding.

| snctl CLI Flag                         | Terraform Attribute         | Description                                         |
| -------------------------------------- | --------------------------- | --------------------------------------------------- |
| `--resource-common-organization`       | `common_organization`       | The organization name                               |
| `--resource-common-instance`           | `common_instance`           | The StreamNative Cloud instance                     |
| `--resource-common-cluster`            | `common_cluster`            | The Pulsar cluster                                  |
| `--resource-common-tenant`             | `common_tenant`             | The tenant                                          |
| `--resource-common-namespace`          | `common_namespace`          | The namespace                                       |
| `--resource-common-topic`              | `common_topic`              | The topic name                                      |
| `--resource-pulsar-topic-domain`       | `pulsar_topic_domain`       | The topic domain (`persistent` or `non-persistent`) |
| `--resource-pulsar-subscription-name`  | `pulsar_subscription_name`  | The subscription name                               |
| `--resource-kafka-consumerGroup-name`  | `kafka_consumergroup_name`  | The Kafka consumer group name                       |
| `--resource-kafka-transaction-id`      | `kafka_transaction_id`      | The Kafka transaction ID                            |
| `--resource-schema-subject`            | `schema_subject`            | The schema subject                                  |
| `--resource-cloud-serviceAccount-name` | `cloud_serviceaccount_name` | The service account name                            |
| `--resource-cloud-apikey-name`         | `cloud_apikey_name`         | The API key name                                    |
| `--resource-cloud-secret-name`         | `cloud_secret_name`         | The secret name                                     |
| `--resource-cloud-catalog-name`        | `cloud_catalog_name`        | The lakehouse catalog name                          |
| `--resource-cloud-connection-name`     | `cloud_connection_name`     | The connection name                                 |
| `--resource-cloud-environment-name`    | `cloud_environment_name`    | The environment name                                |

### Advanced Conditions with Functions

For more sophisticated access control, you can use functions within the resource condition values to match patterns. This is useful for granting permissions to a group of resources, such as all topics with a specific prefix.

The following functions are supported:

* `startsWith('prefix')`: Matches resources that start with the given prefix.
* `endsWith('suffix')`: Matches resources that end with the given suffix.
* `matches('regex')`: Matches resources using a regular expression. The syntax follows Google's [RE2 syntax](https://github.com/google/re2/wiki/Syntax).


# Predefined Roles on StreamNative Cloud
Source: https://docs.streamnative.io/cloud/security/access/rbac/manage-rbac-roles



Role-Based Access Control (RBAC) is the core mechanism for managing permissions in StreamNative Cloud. It allows administrators to grant specific permissions to [principals](/cloud/security/access/rbac/rbac-overview#principal) to perform actions on specific resources.
This feature is enabled by default for all cluster types (Serverless, Dedicated, and BYOC) and all organizations, offering a standardized way to secure your resources. Note the following:

* **Multiple Role Assignments**: A [principal](/cloud/security/access/rbac/rbac-overview#principal) can be assigned multiple roles. When a [principal](/cloud/security/access/rbac/rbac-overview#principal) attempts to access a resource, the request is allowed if any of their assigned roles grant the necessary permission.
* **Resource Visibility**: If a [principal](/cloud/security/access/rbac/rbac-overview#principal) lacks read permission for a resource, that resource will not be visible to them in list commands or in the Cloud Console.
* **Console Access**: To use the StreamNative Cloud Console, a user must be assigned at least one read-only role, such as [org-readonly](#org-readonly), [cluster-readonly](#cluster-readonly), or [tenant-readonly](#tenant-readonly).
* **Permissions are inherited down the resource hierarchy**: `Organization → Instance → Cluster → Tenant → Namespace → Topic`. For example, a **[principal](/cloud/security/access/rbac/rbac-overview#principal)** with the **org-admin** role can manage all resources within that organization, while a **tenant-owner** can only manage the specific tenant they are bound to and its sub-resources (like namespaces and topics).

<Note title="Limitations">
  - A maximum of 10,000 role bindings are allowed per organization.
  - Currently, only [principal](/cloud/security/access/rbac/rbac-overview#principal) with the [org-admin](#org-admin) or [account-admin](#account-admin) role can create and manage service accounts and binding a role to a principal.
</Note>

## Quick Reference

The following table summarizes all available predefined roles to help you quickly find the right one for your needs.

| Role Name                                         | Scope                                   | Summary of Responsibilities                                                                                   |
| :------------------------------------------------ | :-------------------------------------- | :------------------------------------------------------------------------------------------------------------ |
| [org-admin](#org-admin)                           | Organization                            | Full administrative control over all resources in the organization.                                           |
| [org-readonly](#org-readonly)                     | Organization                            | View all resources and settings in the organization without modification rights.                              |
| [org-operator](#org-operator)                     | Organization                            | Perform organization-wide operational tasks like monitoring and basic troubleshooting.                        |
| [metrics-viewer](#metrics-viewer)                 | Organization                            | Access metrics endpoints for monitoring and observability.                                                    |
| [account-admin](#account-admin)                   | Organization                            | Manage user and service accounts, including invitations, deletions, and role assignments.                     |
| [billing-admin](#billing-admin)                   | Organization                            | View and manage billing and subscription information.                                                         |
| [catalog-owner](#catalog-owner)                   | Organization/Catalog                    | Full administrative control over specified catalogs.                                                          |
| [catalog-operator](#catalog-operator)             | Organization/Catalog                    | View and update specified catalogs without deletion rights.                                                   |
| [instance-owner](#instance-owner)                 | Instance                                | Full administrative control over one or more specified instances and all resources within them.               |
| [instance-readonly](#instance-readonly)           | Instance                                | View a specific instance and all its resources without modification rights.                                   |
| [instance-operator](#instance-operator)           | Instance                                | Perform operational tasks within a specific instance.                                                         |
| [cluster-owner](#cluster-owner)                   | Cluster                                 | Full administrative control over one or more specified clusters and all resources within them.                |
| [cluster-readonly](#cluster-readonly)             | Cluster                                 | View specified clusters and all their resources without modification rights.                                  |
| [cluster-operator](#cluster-operator)             | Cluster                                 | Perform operational tasks within specified clusters.                                                          |
| [schema-owner](#schema-owner)                     | Cluster/Schema Registry                 | Full administrative control over the Schema Registry for one or more specified subjects.                      |
| [schema-manager](#schema-manager)                 | Cluster/Schema Registry                 | Manage schema evolution and compatibility policies for the Schema Registry in one or more specified subjects. |
| [schema-reader](#schema-reader)                   | Cluster/Schema Registry                 | Read schema definitions from the Schema Registry in one or more specified subjects.                           |
| [schema-writer](#schema-writer)                   | Cluster/Schema Registry                 | Create and update schemas in the Schema Registry for one or more specified subjects.                          |
| [consumer-group-owner](#consumer-group-owner)     | Cluster/Kafka Consumer Group            | Full administrative control over one or more specified Kafka consumer groups.                                 |
| [consumer-group-reader](#consumer-group-reader)   | Cluster/Kafka Consumer Group            | Perform operational tasks like resetting offsets on one or more specified Kafka consumer groups.              |
| [transactional-id-owner](#transactional-id-owner) | Cluster/Kafka TransactionId             | Full administrative control over one or more specified Kafka transaction IDs.                                 |
| [tenant-owner](#tenant-owner)                     | Tenant                                  | Full administrative control over one or more specified tenants and their namespaces/topics.                   |
| [tenant-readonly](#tenant-readonly)               | Tenant                                  | View one or more specified tenants and all their resources without modification rights.                       |
| [tenant-operator](#tenant-operator)               | Tenant                                  | Perform operational tasks within one or more specified tenants.                                               |
| [namespace-owner](#namespace-owner)               | Namespace                               | Full administrative control over one or more specified namespaces and their topics.                           |
| [namespace-readonly](#namespace-readonly)         | Namespace                               | View one or more specified namespaces and all their resources without modification rights.                    |
| [namespace-operator](#namespace-operator)         | Namespace                               | Perform operational tasks within one or more specified namespaces.                                            |
| [topic-owner](#topic-owner)                       | Topic                                   | Full administrative control over one or more specified topics.                                                |
| [topic-readonly](#topic-readonly)                 | Topic                                   | View the configuration and stats for one or more specified topics, without data access.                       |
| [topic-producer](#topic-producer)                 | Instance/Cluster/Tenant/Namespace/Topic | Produce messages to topics within the bound resource scope.                                                   |
| [topic-consumer](#topic-consumer)                 | Instance/Cluster/Tenant/Namespace/Topic | Consume messages from topics within the bound resource scope.                                                 |

You can use [snctl](/tools/cli/snctl/snctl-overview) to list and get predefined roles.

* **Get list of predefined roles**: `snctl get clusterroles`
* **Get details of a predefined role**: `snctl get clusterrole <name> -o yaml`

<Note title="Note">
  - For human users: We recommend assigning roles with broader scopes (like [org-operator](#org-operator) or [cluster-readonly](#cluster-readonly)) to \[User Accounts]]\(/cloud/security/authentication/user-accounts). This is useful when inviting people to manage or view high-level resources.
  - For applications: It is best practice to use [Service Accounts](/cloud/security/authentication/service-accounts/service-accounts) with more granular, task-specific roles (like [namespace-topic-write](#namespace-topic-write) or [topic-read](#topic-read)) to follow the principle of least privilege.
  - For fine-grained control: Combine roles to grant broad read-only access while restricting administrative rights to specific resources. For example, assign a user both [cluster-readonly](#cluster-readonly) and a more specific [tenant-owner](#tenant-owner) role.
</Note>

## Organization

These are the highest-level roles, granting broad permissions across your entire StreamNative Cloud organization. The roles available at this scope are:

* [org-admin](#org-admin)
* [org-readonly](#org-readonly)
* [org-operator](#org-operator)
* [metrics-viewer](#metrics-viewer)
* [billing-admin](#billing-admin)
* [account-admin](#account-admin)

Follow the instructions below to create a role binding for an organization role.

<Tabs>
  <Tab title="snctl">
    * Ensure you have downloaded [snctl](/tools/cli/snctl/snctl-overview) and [logged in to your StreamNative Cloud organization](/tools/cli/snctl/snctl-overview#sign-in-to-an-organization).
    * To bind a role to a different principal type
      * For [Service Accounts](/cloud/security/authentication/service-accounts/service-accounts), replace `--user` with `--serviceaccount`.
      * For [IdentityPool](/cloud/security/authentication/oidc-identity-providers/manage-oidc-identity-pools), replace `--user` with `--identitypool`.
    * The example below binds the `org-admin` role. To bind a different organization role, replace `org-admin` in the `--clusterrole` argument and in the role binding's name.

    ```shell theme={null}
    snctl create rolebinding org-admin-user1 \
      --clusterrole org-admin \
      --user user1@example.com
    ```
  </Tab>

  <Tab title="Terraform">
    * You can refer [StreamNative Terraform](https://github.com/streamnative/terraform-provider-streamnative/blob/main/examples/rolebinding/main.tf) get more details.
    * To bind a role to [Service Accounts](/cloud/security/authentication/service-accounts/service-accounts), use the `service_account_names` argument instead of `user_names`.
    * The example below binds the `org-admin` role. To bind a different organization role, replace `org-admin` in the `cluster_role_name` argument and in the role binding's name.

    ```hcl theme={null}
    terraform {
      required_providers {
        streamnative = {
          version = "0.10.0"
          source  = "streamnative/streamnative"
        }
      }
    }

    provider "streamnative" {
      # Please replace path use your own key file path
      key_file_path = "/path/to/your/service/account/key.json"
    }

    resource "streamnative_rolebinding" "org_admin_user1_role_binding" {
      organization      = "${The ID of your organization}"
      name              = "org-admin-user1"
      cluster_role_name = "org-admin"
      # To bind to a service account, use 'service_account_names' instead
      user_names        = ["user1@example.com"]
    }
    ```
  </Tab>
</Tabs>

### org-admin

The Organization Administrator. Grants complete and unrestricted control over all StreamNative Cloud and Pulsar/Kafka resources within the organization. This role is equivalent to a superuser or root administrator.

* Account Management: Manages all organization-level settings, including billing, subscriptions, users, service accounts, and security policies.
* Identity Management: Full lifecycle management of users and service accounts: invite, delete, and assign any role to any principal.
* Infrastructure Management: Full administrative control over all infrastructure, including creating, viewing, and deleting instances and clusters.
* Data Plane Management: Complete control over all Pulsar and Kafka resources across all clusters, including tenants, namespaces, topics, schemas, Pulsar Functions, and Connectors.

### org-readonly

The Organization Read-Only Observer. Provides comprehensive, read-only visibility across all resources and configurations in the organization, ideal for auditing and monitoring.

* Global Visibility: Can view all organization-level settings, instances, clusters, tenants, namespaces, and topics.
* Monitoring: Can view resource statistics and health metrics across the entire organization.

**Limitations**:

* No Modification Rights: Cannot create, update, or delete any resource.
* No Sensitive Data Access: Cannot view sensitive resources such as secrets or service account credentials.
* No Data Access: Cannot produce and consume messages from any topic.

<Note title="Note">
  The **connectors/functions** currently lack granular permission control, so `*-readonly` role has no access to connectors/functions. For connector access, assign `*-operator` role to access it.

  We’re developing granular control for connectors and will support read-only connector access to `*-readonly` roles in the future.
</Note>

### org-operator

The Organization Operator. Enables principals to perform organization-wide operational and troubleshooting tasks without granting full administrative or destructive permissions.

* Operational Monitoring: Includes all permissions of the org-readonly role for comprehensive visibility.
* Troubleshooting Actions: Can perform safe, non-destructive operational tasks such as unloading topics, change configuration, resetting subscription cursors, and restarting Pulsar Functions or Connector tasks across any cluster in the organization.

**Limitations**:

* No Destructive Actions: Cannot delete critical resources like clusters, tenants, namespaces, or topics.
* No Sensitive Data Access: Cannot view sensitive resources such as organization secrets or service account credentials.
* No Data Access: Cannot produce and consume messages from any topic.
* No Sensitive Configuration Changes: Cannot alter critical policies like data retention or manage user permissions.
* No Account Management: Cannot manage users, service accounts, or billing information.

Assign to a central Site Reliability Engineering (SRE) or platform operations team responsible for maintaining the day-to-day health and stability of the entire StreamNative Cloud deployment.

### metrics-viewer

The Metrics Viewer. This role is specifically designed to grant access to the metrics API, such as those compatible with Prometheus.

Assign exclusively to a service account used by an external monitoring or observability platform (for example, Prometheus, Grafana, Datadog) to collect performance metrics, adhering to the principle of least privilege.

### account-admin

The Account Administrator. Delegates the management of users and service accounts, separating identity management from infrastructure and data plane administration.

* User Management: Can invite new users to the organization, remove existing users, and manage their role assignments.
* Service Account Management: Full lifecycle control over service accounts, including creation, deletion, and API key management.
* Role Assignment: Can bind any role to users or service accounts.

Assign to an IT security or identity management team responsible for controlling access to the platform, allowing them to manage principals without granting them permissions over the data infrastructure itself.

### billing-admin

The Billing Administrator. A specialized role that provides exclusive access to manage the financial aspects of the organization's account.

* Payment Management: Can view and manage payment methods and billing information.
* Invoice Access: Can view and download billing history and invoices.

Assign to members of the finance department or budget owners who need to manage the organization's cloud spending without needing access to the underlying technical platform.

## Catalogs

You can expose your topics as Lakehouse tables by configuring a catalog. You must have permissions on a catalog to use it when enabling the Lakehouse Table feature.

Follow the instructions below to create a role binding for a catalog role.

<Tabs>
  <Tab title="snctl">
    * Ensure you have downloaded [snctl](/tools/cli/snctl/snctl-overview) and [logged in to your StreamNative Cloud organization](/tools/cli/snctl/snctl-overview#sign-in-to-an-organization).
    * To bind a role to a different principal type
      * For [Service Accounts](/cloud/security/authentication/service-accounts/service-accounts), replace `--user` with `--serviceaccount`.
      * For [IdentityPool](/cloud/security/authentication/oidc-identity-providers/manage-oidc-identity-pools), replace `--user` with `--identitypool`.
    * The example below binds the catalog-owner role to a specific catalog.

    ```shell theme={null}
    snctl create rolebinding catalog-owner-binding \
      --clusterrole catalog-owner \
      --user user1@example.com \
      --resource-cloud-catalog-name ${catalog_name}
    ```

    To grant the `catalog-owner` role to all catalogs that start with a prefix:

    ```shell theme={null}
    snctl create rolebinding catalog-owner-prefix-binding \
      --clusterrole catalog-owner \
      --user user1@example.com \
      --resource-cloud-catalog-name "startsWith('${catalog_prefix}')"
    ```

    To grant the catalog-owner role to all catalogs in the organization, omit the resource flag:

    ```shell theme={null}
    snctl create rolebinding catalog-owner-all-binding \
      --clusterrole catalog-owner \
      --user user1@example.com
    ```
  </Tab>

  <Tab title="Terraform">
    * You can refer [StreamNative Terraform](https://github.com/streamnative/terraform-provider-streamnative/blob/main/examples/rolebinding/main.tf) get more details.
    * To bind a role to [Service Accounts](/cloud/security/authentication/service-accounts/service-accounts), use the `service_account_names` argument instead of `user_names`.
    * The example below binds the catalog-owner role to a specific catalog.

    ```hcl theme={null}
    terraform {
      required_providers {
        streamnative = {
          version = "0.10.0"
          source  = "streamnative/streamnative"
        }
      }
    }

    provider "streamnative" {
      # Please replace path use your own key file path
      key_file_path = "/path/to/your/service/account/key.json"
    }

    resource "streamnative_rolebinding" "catalog_owner_all_binding" {
      organization      = "${The ID of your organization}"
      name              = "catalog-owner-user1"
      cluster_role_name = "catalog-owner"
      user_names        = ["user1@example.com"]
      resource_name_restriction {
        cloud_catalog_name = "${catalog_name}"
      }
    }
    ```
  </Tab>
</Tabs>

### catalog-owner

The Catalog Administrator. Grants full administrative control over specified catalogs.

* Lifecycle Management: Can create, view, update, and delete catalogs.

Assign to platform administrators or data architects responsible for managing the organization's data catalogs.

### catalog-operator

The Catalog Operator. Grants permissions to manage and operate specified catalogs without allowing deletion.

* View and Update: Can view and update the configuration of specified catalogs.
* Operational Tasks: Can perform routine operational tasks related to catalogs.
* Limitations: Cannot create or delete catalogs.

Assign to data engineers or operations teams who need to manage the day-to-day configuration of catalogs without having full administrative control.

## Instance

These roles are scoped to a specific StreamNative Cloud Instance. The roles available at this scope are:

* [instance-owner](#instance-owner)
* [instance-readonly](#instance-readonly)
* [instance-operator](#instance-operator)
* [topic-producer](#topic-producer-instance)
* [topic-consumer](#topic-consumer-instance)

Follow the instructions below to create a role binding for an instance role.

<Tabs>
  <Tab title="snctl">
    * Ensure you have downloaded [snctl](/tools/cli/snctl/snctl-overview) and [logged in to your StreamNative Cloud organization](/tools/cli/snctl/snctl-overview#sign-in-to-an-organization).
    * To bind a role to a different principal type
      * For [Service Accounts](/cloud/security/authentication/service-accounts/service-accounts), replace `--user` with `--serviceaccount`.
      * For [IdentityPool](/cloud/security/authentication/oidc-identity-providers/manage-oidc-identity-pools), replace `--user` with `--identitypool`.
    * The example below binds the `instance-owner` role. To bind a different instance role, replace `instance-owner` in the `--clusterrole` argument and in the role binding's name.

    ```shell theme={null}
    snctl create rolebinding instance-owner-user1 \
      --clusterrole instance-owner \
      --user user1@example.com \
      --resource-common-instance ${instance_name}
    ```
  </Tab>

  <Tab title="Terraform">
    * You can refer [StreamNative Terraform](https://github.com/streamnative/terraform-provider-streamnative/blob/main/examples/rolebinding/main.tf) get more details.
    * To bind a role to [Service Accounts](/cloud/security/authentication/service-accounts/service-accounts), use the `service_account_names` argument instead of `user_names`.
    * The example below binds the `instance-owner` role. To bind a different instance role, replace `instance-owner` in the `cluster_role_name` argument and in the role binding's name.

    ```hcl theme={null}
    terraform {
      required_providers {
        streamnative = {
          version = "0.10.0"
          source  = "streamnative/streamnative"
        }
      }
    }

    provider "streamnative" {
      # Please replace path use your own key file path
      key_file_path = "/path/to/your/service/account/key.json"
    }

    resource "streamnative_rolebinding" "instance_admin_role_binding" {
      organization      = "${The ID of your organization}"
      name              = "instance-owner-user1"
      cluster_role_name = "instance-owner"
      # To bind to a service account, use 'service_account_names' instead
      user_names        = ["user1@example.com"]
      resource_name_restriction {
        common_instance = "${instance_name}"
      }
    }
    ```
  </Tab>
</Tabs>

### instance-owner

The Instance Administrator. Grants full administrative control over all resources within a specific instance, including the clusters it contains.

* Cluster Lifecycle Management: Can create, view, update, and delete Pulsar and Kafka clusters within the specified instance.
* Inherited Permissions: Inherits all permissions of cluster-owner for every cluster within the instance.
* Kafka Cluster Level Resource: Full permissions for Kafka cluster level resources, such as [Schema Registry](#schema-registry), [Kafka Consumer Groups](#kafka-consumer-group), and [Kafka Transactional IDs](#kafka-transactional).

Assign to an environment owner (for example dev-admin, prod-admin) who is responsible for managing all clusters and resources within a specific instance.

### instance-readonly

The Instance Read-Only Observer. Provides read-only visibility into a specific instance and all of its resources.

* View All Resources: Can view all clusters, tenants, namespaces, and topics within the specified instance.
* Inspect Configuration: Can view the configuration of the instance and all its sub-resources.

**Limitations:**

* No Data Access: Cannot produce and consume messages from any topic within the binding instance.

<Note title="Note">
  The **connectors/functions** currently lack granular permission control, so `*-readonly` role has no access to connectors/functions. For connector access, assign `*-operator` role to access it.

  We’re developing granular control for connectors and will support read-only connector access to `*-readonly` roles in the future.
</Note>

### instance-operator

The Instance Operator. Permits operational tasks across all clusters within a specific instance.

* Instance-Wide Operations: Includes all instance-readonly permissions, plus the ability to perform operational tasks (for example, unload topics, restart connectors) on any cluster within the instance.

**Limitations:**

* No Destructive Actions: Cannot create or delete clusters or modify critical configurations within the instance.
* No Data Access: Cannot produce and consume messages from any topic within the binding cluster.

Assign to an SRE team responsible for a specific environment (for example, the production instance) to enable troubleshooting and maintenance.

<h3>
  topic-producer
</h3>

Grants produce access to all Pulsar and Kafka topics within the bound instance. When the role binding condition is set to the instance level, the principal can produce messages to every topic across all clusters in that instance.

For example, the following command grants `user1` the `topic-producer` role scoped to `${instance_name}`, which means the principal can produce to all topics in every cluster within that instance:

```shell theme={null}
snctl create rolebinding topic-producer-user1 \
  --clusterrole topic-producer \
  --user user1@example.com \
  --resource-common-instance ${instance_name}
```

**Limitations:**

* No Administrative Rights: Cannot manage topics, policies, or any other resources.
* No Consume Rights: Cannot consume messages.
* Grant appropriate [Schema Registry](#schema-registry) permissions separately if needed.

<h3>
  topic-consumer
</h3>

Grants consume access to all Pulsar and Kafka topics within the bound instance. When the role binding condition is set to the instance level, the principal can consume messages from every topic across all clusters in that instance.

For example, the following command grants `user1` the `topic-consumer` role scoped to `${instance_name}`:

```shell theme={null}
snctl create rolebinding topic-consumer-user1 \
  --clusterrole topic-consumer \
  --user user1@example.com \
  --resource-common-instance ${instance_name}
```

**Limitations:**

* No Administrative Rights: Cannot manage topics, policies, or any other resources.
* No Produce Rights: Cannot produce messages.
* For Kafka consumers, you must also grant the `consumer-group-reader` role to the principal.
* Grant appropriate [Schema Registry](#schema-registry) permissions separately if needed.

## Cluster

These roles are scoped to a specific Pulsar or Kafka cluster within an cluster. The roles available at this scope are:

* [cluster-owner](#cluster-owner)
* [cluster-readonly](#cluster-readonly)
* [cluster-operator](#cluster-operator)
* [topic-producer](#topic-producer-cluster)
* [topic-consumer](#topic-consumer-cluster)

Follow the instructions below to create a role binding for a cluster role.

<Tabs>
  <Tab title="snctl">
    * Ensure you have downloaded [snctl](/tools/cli/snctl/snctl-overview) and [logged in to your StreamNative Cloud organization](/tools/cli/snctl/snctl-overview#sign-in-to-an-organization).
    * To bind a role to a different principal type
      * For [Service Accounts](/cloud/security/authentication/service-accounts/service-accounts), replace `--user` with `--serviceaccount`.
      * For [IdentityPool](/cloud/security/authentication/oidc-identity-providers/manage-oidc-identity-pools), replace `--user` with `--identitypool`.
    * The example below binds the `cluster-owner` role. To bind a different cluster role, replace `cluster-owner` in the `--clusterrole` argument and in the role binding's name.

    ```shell theme={null}
    snctl create rolebinding cluster-owner-user1 \
      --clusterrole cluster-owner \
      --user user1@example.com \
      --resource-common-instance ${instance_name} \
      --resource-common-cluster ${cluster_id}
    ```

    <Note title="Note">
      The `--resource-common-cluster` flag requires the **Cluster ID**, not the **Cluster Name**. The Cluster ID is a unique, randomly generated string (for example, `pc-y7bti`, `c-6mhjbx2`) and can be found on the cluster overview page in the StreamNative Cloud Console.
    </Note>
  </Tab>

  <Tab title="Terraform">
    * You can refer [StreamNative Terraform](https://github.com/streamnative/terraform-provider-streamnative/blob/main/examples/rolebinding/main.tf) get more details.
    * To bind a role to [Service Accounts](/cloud/security/authentication/service-accounts/service-accounts), use the `service_account_names` argument instead of `user_names`.
    * The example below binds the `cluster-owner` role. To bind a different cluster role, replace `cluster-owner` in the `cluster_role_name` argument and in the role binding's name.

    ```hcl theme={null}
    terraform {
      required_providers {
        streamnative = {
          version = "0.10.0"
          source  = "streamnative/streamnative"
        }
      }
    }

    provider "streamnative" {
      # Please replace path use your own key file path
      key_file_path = "/path/to/your/service/account/key.json"
    }

    resource "streamnative_rolebinding" "cluster_admin_role_binding" {
      organization      = "${The ID of your organization}"
      name              = "cluster-owner-user1"
      cluster_role_name = "cluster-owner"
      # To bind to a service account, use 'service_account_names' instead
      user_names        = ["user1@example.com"]
      resource_name_restriction {
        common_instance = "${instance_name}"
        common_cluster  = "${cluster_id}"
      }
    }
    ```

    <Note title="Note">
      The `resource_name_restriction.common_cluster` field requires the **Cluster ID**, not the **Cluster Name**. The Cluster ID is a unique, randomly generated string (for example, `pc-y7bti`, `c-6mhjbx2`) and can be found on the cluster overview page in the StreamNative Cloud Console.
    </Note>
  </Tab>
</Tabs>

### cluster-owner

The Cluster Administrator. Grants full administrative control over all resources within a specific cluster, enabling multi-tenancy management.

* Tenant Lifecycle Management: Can create, view, update, and delete tenants within the specified cluster. This is the key permission that distinguishes it from tenant-owner.
* Cluster-Level Policies: Manages cluster-wide configurations and policies.
* Inherited Permissions: Inherits all permissions of tenant-owner for every tenant within the cluster.
* Kafka Cluster Level Resource: Full permissions for Kafka cluster level resources, such as [Schema Registry](#schema-registry), [Kafka Consumer Groups](#kafka-consumer-group), and [Kafka Transactional IDs](#kafka-transactional).

**Limitations:**

* Scoped to Cluster: Cannot manage other clusters or instance-level configurations.
* No Org-Level Access: Cannot manage users, service accounts, or billing.

Assign to a platform administrator or team responsible for managing a shared, multi-tenant cluster and onboarding new teams by provisioning tenants for them.

### cluster-readonly

The Cluster Read-Only Observer. Provides read-only visibility into a specific cluster and all of its tenants and resources.

* View All Resources: Can view all tenants, namespaces, topics, and their configurations within the specified cluster.

**Limitations:**

* No Modification Rights: Cannot make any changes to the cluster or any resources within it.
* No Data Access: Cannot produce and consume messages from any topic within the binding cluster.

Assign to any user who needs to understand the structure and status of a specific cluster without having permission to alter it.

<Note title="Note">
  The **connectors/functions** currently lack granular permission control, so `*-readonly` role has no access to connectors/functions. For connector access, assign `*-operator` role to access it.

  We’re developing granular control for connectors and will support read-only connector access to `*-readonly` roles in the future.
</Note>

### cluster-operator

The Cluster Operator. Permits operational tasks across all tenants within a specific cluster.

* Cluster-Wide Operations: Includes all cluster-readonly permissions, plus the ability to perform operational tasks (for example, clear backlogs, reset subscriptions, manage function/connector) on any tenant or namespace within the cluster.

**Limitations:**

* No Destructive Actions: Cannot create or delete tenants or modify critical cluster configurations.
* No Data Access: Cannot produce and consume messages from any topic within the binding cluster.

Assign to an SRE or operations team focused on a single cluster, enabling them to perform maintenance and troubleshooting for all teams using that cluster.

<h3>
  topic-producer
</h3>

Grants produce access to all Pulsar and Kafka topics within the bound cluster. When the role binding condition is set to the cluster level, the principal can produce messages to every topic across all tenants and namespaces in that cluster.

For example, the following command grants `user1` the `topic-producer` role scoped to `${cluster_id}`, which means the principal can produce to all topics in every tenant within that cluster:

```shell theme={null}
snctl create rolebinding topic-producer-user1 \
  --clusterrole topic-producer \
  --user user1@example.com \
  --resource-common-instance ${instance_name} \
  --resource-common-cluster ${cluster_id}
```

<Note title="Note">
  The `--resource-common-cluster` flag requires the **Cluster ID**, not the **Cluster Name**. The Cluster ID is a unique, randomly generated string (for example, `pc-y7bti`, `c-6mhjbx2`) and can be found on the cluster overview page in the StreamNative Cloud Console.
</Note>

**Limitations:**

* No Administrative Rights: Cannot manage topics, policies, or any other resources.
* No Consume Rights: Cannot consume messages.
* Grant appropriate [Schema Registry](#schema-registry) permissions separately if needed.

<h3>
  topic-consumer
</h3>

Grants consume access to all Pulsar and Kafka topics within the bound cluster. When the role binding condition is set to the cluster level, the principal can consume messages from every topic across all tenants and namespaces in that cluster.

For example, the following command grants `user1` the `topic-consumer` role scoped to `${cluster_id}`:

```shell theme={null}
snctl create rolebinding topic-consumer-user1 \
  --clusterrole topic-consumer \
  --user user1@example.com \
  --resource-common-instance ${instance_name} \
  --resource-common-cluster ${cluster_id}
```

<Note title="Note">
  The `--resource-common-cluster` flag requires the **Cluster ID**, not the **Cluster Name**. The Cluster ID is a unique, randomly generated string (for example, `pc-y7bti`, `c-6mhjbx2`) and can be found on the cluster overview page in the StreamNative Cloud Console.
</Note>

**Limitations:**

* No Administrative Rights: Cannot manage topics, policies, or any other resources.
* No Produce Rights: Cannot produce messages.
* For Kafka consumers, you must also grant the `consumer-group-reader` role to the principal.
* Grant appropriate [Schema Registry](#schema-registry) permissions separately if needed.

## Tenant

These roles are scoped to a specific Pulsar or Kafka cluster within a tenant. The roles available at this scope are:

* [tenant-owner](#tenant-owner)
* [tenant-readonly](#tenant-readonly)
* [tenant-operator](#tenant-operator)
* [topic-producer](#topic-producer-tenant)
* [topic-consumer](#topic-consumer-tenant)

Follow the instructions below to create a role binding for a tenant role.

<Tabs>
  <Tab title="snctl">
    * Ensure you have downloaded [snctl](/tools/cli/snctl/snctl-overview) and [logged in to your StreamNative Cloud organization](/tools/cli/snctl/snctl-overview#sign-in-to-an-organization).
    * To bind a role to a different principal type
      * For [Service Accounts](/cloud/security/authentication/service-accounts/service-accounts), replace `--user` with `--serviceaccount`.
      * For [IdentityPool](/cloud/security/authentication/oidc-identity-providers/manage-oidc-identity-pools), replace `--user` with `--identitypool`.
    * The example below binds the `tenant-owner` role. To bind a different tenant role, replace `tenant-owner` in the `--clusterrole` argument and in the role binding's name.

    ```shell theme={null}
    snctl create rolebinding tenant-owner-user1 \
      --clusterrole tenant-owner \
      --user user1@example.com \
      --resource-common-instance ${instance_name} \
      --resource-common-cluster ${cluster_id} \
      --resource-common-tenant ${tenant_name}
    ```

    You can use a `startsWith` macros to bind a role to multiple tenants. For example, grants the `tenant-owner` role to all tenants that start with the prefix `${tenant_prefix}`.

    ```shell theme={null}
    snctl create rolebinding tenant-owner-user1 \
      --clusterrole tenant-owner \
      --user user1@example.com \
      --resource-common-instance ${instance_name} \
      --resource-common-cluster ${cluster_id} \
      --resource-common-tenant "startsWith('${tenant_prefix}')"
    ```

    You can remove `--resource-common-tenant` to bind the role to all tenants in the specified cluster:

    ```shell theme={null}
    snctl create rolebinding tenant-owner-user1 \
      --clusterrole tenant-owner \
      --user user1@example.com \
      --resource-common-instance ${instance_name} \
      --resource-common-cluster ${cluster_id}
    ```

    <Note title="Note">
      The `--resource-common-cluster` flag requires the **Cluster ID**, not the **Cluster Name**. The Cluster ID is a unique, randomly generated string (for example, `pc-y7bti`, `c-6mhjbx2`) and can be found on the cluster overview page in the StreamNative Cloud Console.
    </Note>
  </Tab>

  <Tab title="Terraform">
    * You can refer [StreamNative Terraform](https://github.com/streamnative/terraform-provider-streamnative/blob/main/examples/rolebinding/main.tf) get more details.
    * To bind a role to [Service Accounts](/cloud/security/authentication/service-accounts/service-accounts), use the `service_account_names` argument instead of `user_names`.
    * The example below binds the `tenant-owner` role. To bind a different tenant role, replace `tenant-owner` in the `cluster_role_name` argument and in the role binding's name.

    ```hcl theme={null}
    terraform {
      required_providers {
        streamnative = {
          version = "0.10.0"
          source  = "streamnative/streamnative"
        }
      }
    }

    provider "streamnative" {
      # Please replace path use your own key file path
      key_file_path = "/path/to/your/service/account/key.json"
    }

    resource "streamnative_rolebinding" "tenant_admin_role_binding" {
      organization      = "${The ID of your organization}"
      name              = "tenant-owner-user1"
      cluster_role_name = "tenant-owner"
      # To bind to a service account, use 'service_account_names' instead
      user_names        = ["user1@example.com"]
      resource_name_restriction {
        common_instance = "${instance_name}"
        common_cluster  = "${cluster_id}"
        common_tenant   = "${tenant_name}"
      }
    }
    ```

    You can use a `startsWith` macros to bind a role to multiple tenants. For example, grants the `tenant-owner` role to all tenants that start with the prefix `${tenant_prefix}`.

    ```hcl theme={null}
    #... terraform streamnative basic configuration

    resource "streamnative_rolebinding" "tenant_admin_role_binding" {
      organization      = "${The ID of your organization}"
      name              = "tenant-owner-user1"
      cluster_role_name = "tenant-owner"
      # To bind to a service account, use 'service_account_names' instead
      user_names        = ["user1@example.com"]
      resource_name_restriction {
        common_instance = "${instance_name}"
        common_cluster  = "${cluster_id}"
        common_tenant   = "startsWith('${tenant_prefix}')"
      }
    }
    ```

    <Note title="Note">
      The `resource_name_restriction.common_cluster` field requires the **Cluster ID**, not the **Cluster Name**. The Cluster ID is a unique, randomly generated string (for example, `pc-y7bti`, `c-6mhjbx2`) and can be found on the cluster overview page in the StreamNative Cloud Console.
    </Note>
  </Tab>
</Tabs>

### tenant-owner

The Tenant Administrator. Grants full administrative control over a specific tenant and all its sub-resources, enabling self-service for application teams.

* Namespace Management: Full lifecycle management (create, read, update, delete) of namespaces within the assigned tenant.
* Topic and Policy Management: Manages all topics and namespace-level policies, such as retention, message TTL, and backlog quotas.
* Functions and Connectors: Full lifecycle management of Pulsar Functions and Connectors (Sources/Sinks) within the tenant's namespaces.
* Allows creating tenants bound by the role. For example, if you bind resources prefixed with `test-tenant_`, you can create `test-tenant_x`
* Data Access: Can produce and consume messages from any topic within the binding tenant.

**Limitations:**

* Cannot Manage Other Tenants: Creating or deleting tenants that you haven't bound is not allowed.
* Cannot view and manage cluster-level resources, such as [Schema Registry](#schema-registry), [Kafka Consumer Groups](#kafka-consumer-group), and [Kafka Transactional IDs](#kafka-transactional). You need to grant a cluster-level role or set permissions for these resources individually.

Assign to a team lead or a team's primary service account to give them full control over their application's resources in a shared cluster, promoting a multi-tenant, self-service model.

### tenant-readonly

The Tenant Read-Only Observer. Can view the configuration and status of a specific tenant and all its resources without modification rights.

* View Tenant Resources: Can view all namespaces, topics, subscriptions, functions, and connectors within the tenant.
* Inspect Policies: Can view namespace-level policies and topic configurations.
* View Statistics: Can retrieve statistics for topics and subscriptions.

**Limitations:**

* No Modification Rights: Cannot create, update, or delete any resource within the tenant.
* No Data Access: Cannot produce and consume messages from any topic within the binding tenant.

Assign to developers who need to view the state and configuration of their team's applications and resources without being able to modify them.

<Note title="Note">
  The **connectors/functions** currently lack granular permission control, so `*-readonly` role has no access to connectors/functions. For connector access, assign `*-operator` role to access it.

  We’re developing granular control for connectors and will support read-only connector access to `*-readonly` roles in the future.
</Note>

### tenant-operator

The Tenant Operator. Can perform operational tasks within a specific tenant, such as troubleshooting and routine maintenance.

* Operational Control: Includes all tenant-readonly permissions, plus the ability to perform tasks like clearing backlogs, resetting subscription cursors, unloading topics, and restarting functions/connectors within the tenant.
* Policy Changes: Can alter namespace policies like data retention.
* Allows creating tenants bound by the role. For example, if you bind resources prefixed with `test-tenant_`, you can create `test-tenant_x`

**Limitations:**

* No Destructive Actions: Cannot delete namespaces, topics, or other critical resources.
* No Data Access: Cannot produce and consume messages from any topic within the binding tenant.

Assign to a DevOps engineer or an automated operational tool responsible for maintaining the health of applications within a specific tenant.

<h3>
  topic-producer
</h3>

Grants produce access to all Pulsar and Kafka topics within the bound tenant. When the role binding condition is set to the tenant level, the principal can produce messages to every topic across all namespaces in that tenant.

For example, the following command grants `user1` the `topic-producer` role scoped to `${tenant_name}`, which means the principal can produce to all topics in every namespace under that tenant:

```shell theme={null}
snctl create rolebinding topic-producer-user1 \
  --clusterrole topic-producer \
  --user user1@example.com \
  --resource-common-instance ${instance_name} \
  --resource-common-cluster ${cluster_id} \
  --resource-common-tenant ${tenant_name}
```

<Note title="Note">
  The `--resource-common-cluster` flag requires the **Cluster ID**, not the **Cluster Name**. The Cluster ID is a unique, randomly generated string (for example, `pc-y7bti`, `c-6mhjbx2`) and can be found on the cluster overview page in the StreamNative Cloud Console.
</Note>

**Limitations:**

* No Administrative Rights: Cannot manage topics, policies, or any other resources.
* No Consume Rights: Cannot consume messages.
* Grant appropriate [Schema Registry](#schema-registry) permissions separately if needed.

<h3>
  topic-consumer
</h3>

Grants consume access to all Pulsar and Kafka topics within the bound tenant. When the role binding condition is set to the tenant level, the principal can consume messages from every topic across all namespaces in that tenant.

For example, the following command grants `user1` the `topic-consumer` role scoped to `${tenant_name}`:

```shell theme={null}
snctl create rolebinding topic-consumer-user1 \
  --clusterrole topic-consumer \
  --user user1@example.com \
  --resource-common-instance ${instance_name} \
  --resource-common-cluster ${cluster_id} \
  --resource-common-tenant ${tenant_name}
```

<Note title="Note">
  The `--resource-common-cluster` flag requires the **Cluster ID**, not the **Cluster Name**. The Cluster ID is a unique, randomly generated string (for example, `pc-y7bti`, `c-6mhjbx2`) and can be found on the cluster overview page in the StreamNative Cloud Console.
</Note>

**Limitations:**

* No Administrative Rights: Cannot manage topics, policies, or any other resources.
* No Produce Rights: Cannot produce messages.
* For Kafka consumers, you must also grant the `consumer-group-reader` role to the principal.
* Grant appropriate [Schema Registry](#schema-registry) permissions separately if needed.

## Namespace

These roles are scoped to a specific Pulsar or Kafka cluster within a namespace. The roles available at this scope are:

* [namespace-owner](#namespace-owner)
* [namespace-readonly](#namespace-readonly)
* [namespace-operator](#namespace-operator)
* [topic-producer](#topic-producer-namespace)
* [topic-consumer](#topic-consumer-namespace)

Follow the instructions below to create a role binding for a namespace role.

<Tabs>
  <Tab title="snctl">
    * Ensure you have downloaded [snctl](/tools/cli/snctl/snctl-overview) and [logged in to your StreamNative Cloud organization](/tools/cli/snctl/snctl-overview#sign-in-to-an-organization).
    * To bind a role to a different principal type
      * For [Service Accounts](/cloud/security/authentication/service-accounts/service-accounts), replace `--user` with `--serviceaccount`.
      * For [IdentityPool](/cloud/security/authentication/oidc-identity-providers/manage-oidc-identity-pools), replace `--user` with `--identitypool`.
    * The example below binds the `namespace-owner` role. To bind a different namespace role, replace `namespace-owner` in the `--clusterrole` argument and in the role binding's name.

    ```shell theme={null}
    snctl create rolebinding namespace-owner-user1 \
      --clusterrole namespace-owner \
      --user user1@example.com \
      --resource-common-instance ${instance_name} \
      --resource-common-cluster ${cluster_id} \
      --resource-common-tenant ${tenant_name} \
      --resource-common-namespace ${namespace_name}
    ```

    You can use a `startsWith` macros to bind a role to multiple namespace. For example, grants the `namespace-owner` role to all namespaces that start with the prefix `${namespace_prefix}`.

    ```shell theme={null}
    snctl create rolebinding namespace-owner-user1 \
      --clusterrole namespace-owner \
      --user user1@example.com \
      --resource-common-instance ${instance_name} \
      --resource-common-cluster ${cluster_id} \
      --resource-common-tenant ${tenant_name} \
      --resource-common-namespace "startsWith('${namespace_prefix}')"
    ```

    You can remove `--resource-common-namespace` to bind the role to all namespaces in the specified tenant:

    ```shell theme={null}
    snctl create rolebinding tenant-owner-user1 \
      --clusterrole tenant-owner \
      --user user1@example.com \
      --resource-common-instance ${instance_name} \
      --resource-common-cluster ${cluster_id} \
      --resource-common-tenant ${tenant_name}
    ```

    <Note title="Note">
      The `--resource-common-cluster` flag requires the **Cluster ID**, not the **Cluster Name**. The Cluster ID is a unique, randomly generated string (for example, `pc-y7bti`, `c-6mhjbx2`) and can be found on the cluster overview page in the StreamNative Cloud Console.
    </Note>
  </Tab>

  <Tab title="Terraform">
    * You can refer [StreamNative Terraform](https://github.com/streamnative/terraform-provider-streamnative/blob/main/examples/rolebinding/main.tf) get more details.
    * To bind a role to [Service Accounts](/cloud/security/authentication/service-accounts/service-accounts), use the `service_account_names` argument instead of `user_names`.
    * The example below binds the `namespace-owner` role. To bind a different namespace role, replace `namespace-owner` in the `cluster_role_name` argument and in the role binding's name.

    ```hcl theme={null}
    terraform {
      required_providers {
        streamnative = {
          version = "0.10.0"
          source  = "streamnative/streamnative"
        }
      }
    }

    provider "streamnative" {
      # Please replace path use your own key file path
      key_file_path = "/path/to/your/service/account/key.json"
    }

    resource "streamnative_rolebinding" "namespace_admin_role_binding" {
      organization      = "${The ID of your organization}"
      name              = "namespace-owner-user1"
      cluster_role_name = "namespace-owner"
      # To bind to a service account, use 'service_account_names' instead
      user_names        = ["user1@example.com"]
      resource_name_restriction {
        common_instance = "${instance_name}"
        common_cluster  = "${cluster_id}"
        common_tenant   = "${tenant_name}"
        common_namespace = "${namespace_name}"
      }
    }
    ```

    You can use a `startsWith` macros to bind a role to multiple namespace. For example, grants the `namespace-owner` role to all namespaces that start with the prefix `${namespace_prefix}`.

    ```hcl theme={null}
    #... terraform streamnative basic configuration

    resource "streamnative_rolebinding" "namespace_admin_role_binding" {
      organization      = "${The ID of your organization}"
      name              = "namespace-owner-user1"
      cluster_role_name = "namespace-owner"
      # To bind to a service account, use 'service_account_names' instead
      user_names        = ["user1@example.com"]
      resource_name_restriction {
        common_instance = "${instance_name}"
        common_cluster  = "${cluster_id}"
        common_tenant   = "${tenant_name}"
        common_namespace = "startsWith('${namespace_prefix}')"
      }
    }
    ```

    <Note title="Note">
      The `resource_name_restriction.common_cluster` field requires the **Cluster ID**, not the **Cluster Name**. The Cluster ID is a unique, randomly generated string (for example, `pc-y7bti`, `c-6mhjbx2`) and can be found on the cluster overview page in the StreamNative Cloud Console.
    </Note>
  </Tab>
</Tabs>

### namespace-owner

The Namespace Administrator. Grants full administrative control over a specific namespace and all topics and functions within it.

* Topic Management: Full lifecycle management of topics within the namespace.
* Policy Control: Manages all policies for the namespace, such as retention and backlog quotas.
* Functions and Connectors: Full lifecycle management of Pulsar Functions and Connectors deployed to the namespace.
* Data Access: Can produce and consume messages from any topic within the binding tenant.
* Allows creating namespaces bound by the role. For example, if you bind resources prefixed with `test-namepsace_`, you can create `test-namespace_x`

**Limitations:**

* Cannot Manage Other Namespace: Creating or deleting namespace that you haven't bound is not allowed.
* Cannot view and manage cluster-level resources, such as [Schema Registry](#schema-registry), [Kafka Consumer Groups](#kafka-consumer-group), and [Kafka Transactional IDs](#kafka-transactional). You need to grant a cluster-level role or set permissions for these resources individually.

Assign to an application owner or lead developer to provide full control over all resources related to a specific microservice or application component.

### namespace-readonly

The Namespace Read-Only Observer. Can view the configuration and status of a specific namespace and all its

* View Namespace Resources: Can view all topics, subscriptions, functions, and connectors within the namespace.
* Inspect Configuration: Can view namespace policies and topic configurations.

**Limitations:**

* No Modification Rights: Cannot create, update, or delete any resource.
* No Data Access: Cannot produce and consume messages from any topic within the binding namespace.

Assign to a developer who needs visibility into a specific application's resources for debugging or understanding its configuration.

<Note title="Note">
  The **connectors/functions** currently lack granular permission control, so `*-readonly` role has no access to connectors/functions. For connector access, assign `*-operator` role to access it.

  We’re developing granular control for connectors and will support read-only connector access to `*-readonly` roles in the future.
</Note>

### namespace-operator

The Namespace Operator. Can perform operational tasks within a specific namespace.

* Operational Control: Includes all namespace-readonly permissions, plus the ability to perform operational tasks like clearing backlogs, resetting subscriptions, and restarting functions/connectors within the namespace.
* Allows creating namespaces bound by the role. For example, if you bind resources prefixed with `test-namepsace_`, you can create `test-namespace_x`

**Limitations:**

* No Destructive Actions: Cannot delete topics or other resources.
* No Data Access: Cannot produce and consume messages from any topic within the binding namespace.

Assign to a service account for an automated CI/CD pipeline that needs to perform operational tasks like restarting a function after a deployment.

<h3>
  topic-producer
</h3>

Grants produce access to all Pulsar and Kafka topics within the bound namespace. When the role binding condition is set to the namespace level, the principal can produce messages to every existing and future topic in that namespace.

For example, the following command grants `user1` the `topic-producer` role scoped to `${namespace_name}`, which means the principal can produce to all topics in that namespace:

```shell theme={null}
snctl create rolebinding topic-producer-user1 \
  --clusterrole topic-producer \
  --user user1@example.com \
  --resource-common-instance ${instance_name} \
  --resource-common-cluster ${cluster_id} \
  --resource-common-tenant ${tenant_name} \
  --resource-common-namespace ${namespace_name}
```

<Note title="Note">
  The `--resource-common-cluster` flag requires the **Cluster ID**, not the **Cluster Name**. The Cluster ID is a unique, randomly generated string (for example, `pc-y7bti`, `c-6mhjbx2`) and can be found on the cluster overview page in the StreamNative Cloud Console.
</Note>

**Limitations:**

* No Administrative Rights: Cannot manage topics, policies, or any other resources.
* No Consume Rights: Cannot consume messages.
* Grant appropriate [Schema Registry](#schema-registry) permissions separately if needed.

<h3>
  topic-consumer
</h3>

Grants consume access to all Pulsar and Kafka topics within the bound namespace. When the role binding condition is set to the namespace level, the principal can consume messages from every existing and future topic in that namespace.

For example, the following command grants `user1` the `topic-consumer` role scoped to `${namespace_name}`:

```shell theme={null}
snctl create rolebinding topic-consumer-user1 \
  --clusterrole topic-consumer \
  --user user1@example.com \
  --resource-common-instance ${instance_name} \
  --resource-common-cluster ${cluster_id} \
  --resource-common-tenant ${tenant_name} \
  --resource-common-namespace ${namespace_name}
```

<Note title="Note">
  The `--resource-common-cluster` flag requires the **Cluster ID**, not the **Cluster Name**. The Cluster ID is a unique, randomly generated string (for example, `pc-y7bti`, `c-6mhjbx2`) and can be found on the cluster overview page in the StreamNative Cloud Console.
</Note>

**Limitations:**

* No Administrative Rights: Cannot manage topics, policies, or any other resources.
* No Produce Rights: Cannot produce messages.
* For Kafka consumers, you must also grant the `consumer-group-reader` role to the principal.
* Grant appropriate [Schema Registry](#schema-registry) permissions separately if needed.

## Topic

These roles are scoped to a specific Pulsar or Kafka cluster within a topic. The roles available at this scope are:

* [topic-owner](#topic-owner)
* [topic-readonly](#topic-readonly)
* [topic-producer](#topic-producer)
* [topic-consumer](#topic-consumer)

Follow the instructions below to create a role binding for a topic role.

<Tabs>
  <Tab title="snctl">
    * Ensure you have downloaded [snctl](/tools/cli/snctl/snctl-overview) and [logged in to your StreamNative Cloud organization](/tools/cli/snctl/snctl-overview#sign-in-to-an-organization).
    * To bind a role to a different principal type
      * For [Service Accounts](/cloud/security/authentication/service-accounts/service-accounts), replace `--user` with `--serviceaccount`.
      * For [IdentityPool](/cloud/security/authentication/oidc-identity-providers/manage-oidc-identity-pools), replace `--user` with `--identitypool`.
    * The example below binds the `topic-owner` role. To bind a different topic role, replace `topic-owner` in the `--clusterrole` argument and in the role binding's name.

    ```shell theme={null}
    snctl create rolebinding topic-owner-user1 \
      --clusterrole topic-owner \
      --user user1@example.com \
      --resource-common-instance ${instance_name} \
      --resource-common-cluster ${cluster_id} \
      --resource-common-tenant ${tenant_name} \
      --resource-common-namespace ${namespace_name} \
      --resource-common-topic "allPartition('${topic_name}')"
    ```

    <Note title="Note">
      The `allPartition` macro applies the binding to all partitions of the specified topic. To bind to a single partition, provide the specific partition's full name directly. For example: `--resource-common-topic test-topic-partition-0`
    </Note>

    You can use a `startsWith` macros to bind a role to multipl topic. For example, grants the `topic-owner` role to all topics that start with the prefix `${topic_prefix}`.

    ```shell theme={null}
    snctl create rolebinding topic-owner-user1 \
      --clusterrole topic-owner \
      --user user1@example.com \
      --resource-common-instance ${instance_name} \
      --resource-common-cluster ${cluster_id} \
      --resource-common-tenant ${tenant_name} \
      --resource-common-namespace ${namespace_name} \
      --resource-common-topic "startsWith('${topic_prefix}')"
    ```

    You can remove `--resource-common-namespace` to bind the role to all topics in the specified namespace:

    ```shell theme={null}
    snctl create rolebinding topic-owner-user1 \
      --clusterrole topic-owner \
      --user user1@example.com \
      --resource-common-instance ${instance_name} \
      --resource-common-cluster ${cluster_id} \
      --resource-common-tenant ${tenant_name} \
      --resource-common-namespace ${namespace_name}
    ```

    <Note title="Note">
      The `--resource-common-cluster` flag requires the **Cluster ID**, not the **Cluster Name**. The Cluster ID is a unique, randomly generated string (for example, `pc-y7bti`, `c-6mhjbx2`) and can be found on the cluster overview page in the StreamNative Cloud Console.
    </Note>
  </Tab>

  <Tab title="Terraform">
    * You can refer [StreamNative Terraform](https://github.com/streamnative/terraform-provider-streamnative/blob/main/examples/rolebinding/main.tf) get more details.
    * To bind a role to [Service Accounts](/cloud/security/authentication/service-accounts/service-accounts), use the `service_account_names` argument instead of `user_names`.
    * The example below binds the `topic-owner` role. To bind a different topic role, replace `topic-owner` in the `cluster_role_name` argument and in the role binding's name.

    ```hcl theme={null}
    terraform {
      required_providers {
        streamnative = {
          version = "0.10.0"
          source  = "streamnative/streamnative"
        }
      }
    }

    provider "streamnative" {
      # Please replace path use your own key file path
      key_file_path = "/path/to/your/service/account/key.json"
    }

    resource "streamnative_rolebinding" "topic_admin_role_binding" {
      organization      = "${The ID of your organization}"
      name              = "topic-owner-user1"
      cluster_role_name = "topic-owner"
      # To bind to a service account, use 'service_account_names' instead
      user_names        = ["user1@example.com"]
      resource_name_restriction {
        common_instance = "${instance_name}"
        common_cluster  = "${cluster_id}"
        common_tenant   = "${tenant_name}"
        common_topic = "allPartition('${topic_name}')"
      }
    }
    ```

    <Note title="Note">
      The `allPartition` macro applies the binding to all partitions of the specified topic. To bind to a single partition, provide the specific partition's full name directly. For example: `--resource-common-topic test-topic-partition-0`
    </Note>

    You can use a `startsWith` macros to bind a role to multipl topic. For example, grants the `topic-owner` role to all topics that start with the prefix `${topic_prefix}`.

    ```hcl theme={null}
    #... terraform streamnative basic configuration

    resource "streamnative_rolebinding" "topic_admin_role_binding" {
      organization      = "${The ID of your organization}"
      name              = "topic-owner-user1"
      cluster_role_name = "topic-owner"
      # To bind to a service account, use 'service_account_names' instead
      user_names        = ["user1@example.com"]
      resource_name_restriction {
        common_instance = "${instance_name}"
        common_cluster  = "${cluster_id}"
        common_tenant   = "${tenant_name}"
        common_topic = "startsWith('${topic_prefix}')"
      }
    }
    ```

    <Note title="Note">
      The `resource_name_restriction.common_cluster` field requires the **Cluster ID**, not the **Cluster Name**. The Cluster ID is a unique, randomly generated string (for example, `pc-y7bti`, `c-6mhjbx2`) and can be found on the cluster overview page in the StreamNative Cloud Console.
    </Note>
  </Tab>
</Tabs>

### topic-owner

The Topic Administrator. Grants administrative control over a specific topic's configuration and lifecycle.

**Limitations:**

* Does not include data access.
* Cannot view and manage cluster-level resources, such as [Schema Registry](#schema-registry), [Kafka Consumer Groups](#kafka-consumer-group), and [Kafka Transactional IDs](#kafka-transactional). You need to grant a cluster-level role or set permissions for these resources individually.

### topic-readonly

The Topic Read-Only Observer. Allows viewing the configuration and statistics of a specific topic without modification rights or data access.

### topic-producer

The Producer. Allows a principal to produce messages to Pulsar and Kafka topics within the bound resource scope.

This role can be bound at any level of the resource hierarchy — instance, cluster, tenant, namespace, or individual topic. When bound at a broader scope, the principal can produce to all topics within that scope. For example, binding at the namespace scope grants produce access to all topics in that namespace; binding at the tenant scope grants produce access to all topics across all namespaces in that tenant.

For example, the following command grants `user1` the `topic-producer` role scoped to a specific topic:

```shell theme={null}
snctl create rolebinding topic-producer-user1 \
  --clusterrole topic-producer \
  --user user1@example.com \
  --resource-common-instance ${instance_name} \
  --resource-common-cluster ${cluster_id} \
  --resource-common-tenant ${tenant_name} \
  --resource-common-namespace ${namespace_name} \
  --resource-common-topic "allPartition('${topic_name}')"
```

<Note title="Note">
  The `allPartition` macro applies the binding to all partitions of the specified topic. To bind to a single partition, provide the specific partition's full name directly. For example: `--resource-common-topic test-topic-partition-0`

  The `--resource-common-cluster` flag requires the **Cluster ID**, not the **Cluster Name**. The Cluster ID is a unique, randomly generated string (for example, `pc-y7bti`, `c-6mhjbx2`) and can be found on the cluster overview page in the StreamNative Cloud Console.
</Note>

**Limitations:**

* No Administrative Rights: Cannot manage topics, policies, or any other resources.
* No Consume Rights: Cannot consume messages.
* Grant appropriate [Schema Registry](#schema-registry) permissions separately if needed.

Assign to a service account for a producer application. Use a narrower scope (such as a specific topic or namespace) to follow the principle of least privilege.

### topic-consumer

The Consumer. Allows a principal to consume messages from Pulsar and Kafka topics within the bound resource scope.

This role can be bound at any level of the resource hierarchy — instance, cluster, tenant, namespace, or individual topic. When bound at a broader scope, the principal can consume from all topics within that scope. For example, binding at the namespace scope grants consume access to all existing and future topics in that namespace.

For example, the following command grants `user1` the `topic-consumer` role scoped to a specific topic:

```shell theme={null}
snctl create rolebinding topic-consumer-user1 \
  --clusterrole topic-consumer \
  --user user1@example.com \
  --resource-common-instance ${instance_name} \
  --resource-common-cluster ${cluster_id} \
  --resource-common-tenant ${tenant_name} \
  --resource-common-namespace ${namespace_name} \
  --resource-common-topic "allPartition('${topic_name}')"
```

<Note title="Note">
  The `allPartition` macro applies the binding to all partitions of the specified topic. To bind to a single partition, provide the specific partition's full name directly. For example: `--resource-common-topic test-topic-partition-0`

  The `--resource-common-cluster` flag requires the **Cluster ID**, not the **Cluster Name**. The Cluster ID is a unique, randomly generated string (for example, `pc-y7bti`, `c-6mhjbx2`) and can be found on the cluster overview page in the StreamNative Cloud Console.
</Note>

* Consume Data: Can create subscriptions and consume messages from topics within the bound resource scope.

**Limitations:**

* No Administrative Rights: Cannot manage topics, policies, or any other resources.
* No Produce Rights: Cannot produce messages.
* For Kafka consumers, you must also grant the `consumer-group-reader` role to the principal to consume messages from a Kafka topic.
* Grant appropriate [Schema Registry](#schema-registry) permissions separately if needed.

Assign to a service account for a consumer application or stream processing job. Use a narrower scope (such as a specific topic or namespace) to follow the principle of least privilege.

## Schema Registry

These roles are used to manage Pulsar Schemas, ensuring data governance and type safety. The roles available at this scope are:

* [schema-owner](#schema-owner)
* [schema-manager](#schema-manager)
* [schema-reader](#schema-reader)
* [schema-writer](#schema-writer)

The operational resource of the schema is `subject`, which are resources under the `cluster`. Follow the instructions below to create a role binding for a schema subject role.

<Tabs>
  <Tab title="snctl">
    * Ensure you have downloaded [snctl](/tools/cli/snctl/snctl-overview) and [logged in to your StreamNative Cloud organization](/tools/cli/snctl/snctl-overview#sign-in-to-an-organization).
    * To bind a role to a different principal type
      * For [Service Accounts](/cloud/security/authentication/service-accounts/service-accounts), replace `--user` with `--serviceaccount`.
      * For [IdentityPool](/cloud/security/authentication/oidc-identity-providers/manage-oidc-identity-pools), replace `--user` with `--identitypool`.
    * The example below binds the `schema-owner` role. To bind a different schema role, replace `schema-owner` in the `--clusterrole` argument and in the role binding's name.

    ```shell theme={null}
    snctl create rolebinding schema-owner-user1 \
      --clusterrole schema-owner \
      --user user1@example.com \
      --resource-common-instance ${instance_name} \
      --resource-common-cluster ${cluster_id} \
      --resource-schema-subject ${subject}
    ```

    You can use a `startsWith` macros to bind a role to multiple subjects. For example, grants the `schema-owner` role to all subjects that start with the prefix `${subject_prefix}`.

    ```shell theme={null}
    snctl create rolebinding schema-owner-user1 \
      --clusterrole schema-owner \
      --user user1@example.com \
      --resource-common-instance ${instance_name} \
      --resource-common-cluster ${cluster_id} \
      --resource-schema-subject "startsWith('${subject_prefix}')"
    ```

    You can remove `--resource-schema-subject` to bind the role to all subjects in the specified cluster schema registry:

    ```shell theme={null}
    snctl create rolebinding topic-owner-user1 \
      --clusterrole topic-owner \
      --user user1@example.com \
      --resource-common-instance ${instance_name} \
      --resource-common-cluster ${cluster_id} \
    ```

    <Note title="Note">
      The `--resource-common-cluster` flag requires the **Cluster ID**, not the **Cluster Name**. The Cluster ID is a unique, randomly generated string (for example, `pc-y7bti`, `c-6mhjbx2`) and can be found on the cluster overview page in the StreamNative Cloud Console.
    </Note>
  </Tab>

  <Tab title="Terraform">
    * You can refer [StreamNative Terraform](https://github.com/streamnative/terraform-provider-streamnative/blob/main/examples/rolebinding/main.tf) get more details.
    * To bind a role to [Service Accounts](/cloud/security/authentication/service-accounts/service-accounts), use the `service_account_names` argument instead of `user_names`.
    * The example below binds the `schema-owner` role. To bind a different schema role, replace `schema-owner` in the `cluster_role_name` argument and in the role binding's name.

    ```hcl theme={null}
    terraform {
      required_providers {
        streamnative = {
          version = "0.10.0"
          source  = "streamnative/streamnative"
        }
      }
    }

    provider "streamnative" {
      # Please replace path use your own key file path
      key_file_path = "/path/to/your/service/account/key.json"
    }

    resource "streamnative_rolebinding" "schema_owner_role_binding" {
      organization      = "${The ID of your organization}"
      name              = "schema-owner-user1"
      cluster_role_name = "schema-owner"
      # To bind to a service account, use 'service_account_names' instead
      user_names        = ["user1@example.com"]
      resource_name_restriction {
        common_instance = "${instance_name}"
        common_cluster  = "${cluster_id}"
        schema_subject  = "${subject}"
      }
    }
    ```

    You can use a `startsWith` macros to bind a role to multiple subjects. For example, grants the `schema-owner` role to all subjects that start with the prefix `${subject_prefix}`.

    ```hcl theme={null}
    #... terraform streamnative basic configuration

    resource "streamnative_rolebinding" "schema_owner_role_binding" {
      organization      = "${The ID of your organization}"
      name              = "schema-owner-user1"
      cluster_role_name = "schema-owner"
      # To bind to a service account, use 'service_account_names' instead
      user_names        = ["user1@example.com"]
      resource_name_restriction {
        common_instance = "${instance_name}"
        common_cluster  = "${cluster_id}"
        schema_subject  = "startsWith('${subject_prefix}')"
      }
    }
    ```

    <Note title="Note">
      The `srn.cluster` field requires the **Cluster ID**, not the **Cluster Name**. The Cluster ID is a unique, randomly generated string (for example, `pc-y7bti`, `c-6mhjbx2`) and can be found on the cluster overview page in the StreamNative Cloud Console.
    </Note>
  </Tab>
</Tabs>

### schema-owner

The Schema Administrator. Grants full administrative control over specified schema subjects, including destructive actions.

* Full Lifecycle Management: Can create, update, and delete schemas and schema versions.
* Destructive Actions: Can delete entire schema subjects, which is an irreversible action.
* Policy Control: Includes all permissions of the schema-manager role, such as managing compatibility policies.

**Limitations:**

* Scoped to Subject: Permissions are limited to the specified schema subjects.

### schema-manager

The Schema Manager. Allows for the management of schema evolution and compatibility policies for specified subjects.

* Compatibility Management: Can get and update the schema compatibility policy for a subject (for example, BACKWARD, FORWARD, FULL).
* Schema Evolution: Can test the compatibility of new schema versions.
* Includes Writer Permissions: Can create and update schemas, inheriting permissions from the schema-writer role.

**Limitations:**

* No Deletion: Cannot delete schemas or entire schema subjects.

### schema-reader

The Schema Reader. Provides read-only access to view schema definitions for specified subjects.

* Read Schemas: Can fetch schema definitions and all their versions for a given subject.
* Check Compatibility: Can check if a schema is compatible with the latest version for a subject.

**Limitations:**

* No Modification Rights: Cannot post new schemas, update compatibility settings, or delete any schema information.

### schema-writer

The Schema Writer. Allows for the creation and updating of schemas for specified subjects.

* Create and Update Schemas: Can post new schema versions for a subject.

**Limitations:**

* No Policy Management: Cannot change the compatibility mode for the subject.
* No Deletion: Cannot delete schemas or subjects.

## Kafka Consumer Group

These roles are designed for managing Kafka consumer group resources on StreamNative Cloud, applicable to Kafka-on-Pulsar (KoP) use cases. The roles available at this scope are:

* [consumer-group-owner](#consumer-group-owner)
* [consumer-group-reader](#consumer-group-reader)

Follow the instructions below to create a role binding for a schema subject role.

<Tabs>
  <Tab title="snctl">
    * Ensure you have downloaded [snctl](/tools/cli/snctl/snctl-overview) and [logged in to your StreamNative Cloud organization](/tools/cli/snctl/snctl-overview#sign-in-to-an-organization).
    * To bind a role to a different principal type
      * For [Service Accounts](/cloud/security/authentication/service-accounts/service-accounts), replace `--user` with `--serviceaccount`.
      * For [IdentityPool](/cloud/security/authentication/oidc-identity-providers/manage-oidc-identity-pools), replace `--user` with `--identitypool`.
    * The example below binds the `consumer-group-owner` role. To bind a different consumer group role, replace `consumer-group-owner` in the `--clusterrole` argument and in the role binding's name.

    ```shell theme={null}
    snctl create rolebinding consumer-group-owner-user1 \
      --clusterrole consumer-group-owner \
      --user user1@example.com \
      --resource-common-instance ${instance_name} \
      --resource-common-cluster ${cluster_id} \
      --resource-kafka-consumerGroup-name ${group_name}
    ```

    You can use a `startsWith` macros to bind a role to multiple consumer group. For example, grants the `consumer-group-owner` role to all consumer group that start with the prefix `${group_name_prefix}`.

    ```shell theme={null}
    snctl create rolebinding consumer-group-owner-user1 \
      --clusterrole consumer-group-owner \
      --user user1@example.com \
      --resource-common-instance ${instance_name} \
      --resource-common-cluster ${cluster_id} \
      --resource-kafka-consumerGroup-name "startsWith('${group_name_prefix}')"
    ```

    You can remove `--resource-kafka-consumerGroup-name` to bind the role to all consumer group in the specified cluster:

    ```shell theme={null}
    snctl create rolebinding consumer-group-owner-user1 \
      --clusterrole consumer-group-owner \
      --user user1@example.com \
      --resource-common-instance ${instance_name} \
      --resource-common-cluster ${cluster_id} \
    ```

    <Note title="Note">
      The `--resource-common-cluster` flag requires the **Cluster ID**, not the **Cluster Name**. The Cluster ID is a unique, randomly generated string (for example, `pc-y7bti`, `c-6mhjbx2`) and can be found on the cluster overview page in the StreamNative Cloud Console.
    </Note>
  </Tab>

  <Tab title="Terraform">
    * You can refer [StreamNative Terraform](https://github.com/streamnative/terraform-provider-streamnative/blob/main/examples/rolebinding/main.tf) get more details.
    * To bind a role to [Service Accounts](/cloud/security/authentication/service-accounts/service-accounts), use the `service_account_names` argument instead of `user_names`.
    * The example below binds the `consumer-group-owner` role. To bind a different consumer group role, replace `consumer-group-owner` in the `cluster_role_name` argument and in the role binding's name.

    ```hcl theme={null}
    terraform {
      required_providers {
        streamnative = {
          version = "0.10.0"
          source  = "streamnative/streamnative"
        }
      }
    }

    provider "streamnative" {
      # Please replace path use your own key file path
      key_file_path = "/path/to/your/service/account/key.json"
    }

    resource "streamnative_rolebinding" "consumer_group_admin_role_binding" {
      organization      = "${The ID of your organization}"
      name              = "consumer-group-ownerr-user1"
      cluster_role_name = "consumer-group-owner"
      # To bind to a service account, use 'service_account_names' instead
      user_names        = ["user1@example.com"]
      resource_name_restriction {
        common_instance = "${instance_name}"
        common_cluster  = "${cluster_id}"
        kafka_consumergroup_name  = "${group_name}"
      }
    }
    ```

    You can use a `startsWith` macros to bind a role to multiple consumer group. For example, grants the `consumer-group-owner` role to all consumer group that start with the prefix `${group_name_prefix}`.

    ```hcl theme={null}
    #... terraform streamnative basic configuration

    resource "streamnative_rolebinding" "consumer_group_admin_role_binding" {
      organization      = "${The ID of your organization}"
      name              = "consumer-group-ownerr-user1"
      cluster_role_name = "consumer-group-owner"
      # To bind to a service account, use 'service_account_names' instead
      user_names        = ["user1@example.com"]
      resource_name_restriction {
        common_instance = "${instance_name}"
        common_cluster  = "${cluster_id}"
        kafka_consumergroup_name  = "startsWith('${group_name_prefix}')"
      }
    }
    ```

    <Note title="Note">
      The `srn.cluster` field requires the **Cluster ID**, not the **Cluster Name**. The Cluster ID is a unique, randomly generated string (for example, `pc-y7bti`, `c-6mhjbx2`) and can be found on the cluster overview page in the StreamNative Cloud Console.
    </Note>
  </Tab>
</Tabs>

### consumer-group-owner

The Consumer Group Administrator. Grants full administrative control over specific Kafka consumer groups, including the ability to delete them.

* Delete Group: Can delete a consumer group, which removes its committed offsets.
* Alter Offsets: Can alter committed offsets for any partition.
* View Group: Can describe the consumer group, view its members, and monitor consumer lag.
* Includes Reader Permissions: Inherits all permissions from the consumer-group-reader role.

**Limitations:**

* Scoped to Group: Permissions are limited to the specified consumer groups.

### consumer-group-reader

The Consumer Group Reader. Allows for monitoring and performing safe operational tasks on a specific Kafka consumer group.

* Reset Offsets: Can perform the common operational task of resetting consumer offsets to a specific point in time (for example, earliest, latest) to reprocess or skip messages.
* View Group: Can describe the consumer group, view its members, and monitor consumer lag.

**Limitations:**

* No Deletion: Cannot delete the consumer group.

## Kafka Transactional

These roles are designed for managing Kafka transactional ID resources on StreamNative Cloud, applicable to Kafka-on-Pulsar (KoP) use cases. The roles available at this scope are:

* [transactional-id-owner](#transactional-id-owner)

Follow the instructions below to create a role binding for a schema subject role.

<Tabs>
  <Tab title="snctl">
    * Ensure you have downloaded [snctl](/tools/cli/snctl/snctl-overview) and [logged in to your StreamNative Cloud organization](/tools/cli/snctl/snctl-overview#sign-in-to-an-organization).
    * To bind a role to a different principal type
      * For [Service Accounts](/cloud/security/authentication/service-accounts/service-accounts), replace `--user` with `--serviceaccount`.
      * For [IdentityPool](/cloud/security/authentication/oidc-identity-providers/manage-oidc-identity-pools), replace `--user` with `--identitypool`.
    * The example below binds the `transactional-id-owner` role. To bind a different transactional role, replace `transactional-id-owner` in the `--clusterrole` argument and in the role binding's name.

    ```shell theme={null}
    snctl create rolebinding transactional-id-owner-user1 \
      --clusterrole transactional-id-owner \
      --user user1@example.com \
      --resource-common-instance ${instance_name} \
      --resource-common-cluster ${cluster_id} \
      --resource-kafka-transaction-id ${transactional_id}
    ```

    You can use a `startsWith` macros to bind a role to multiple transactional ID. For example, grants the `transactional-id-owner` role to all transactional ID that start with the prefix `${transactional_id_prefix}`.

    ```shell theme={null}
    snctl create rolebinding transactional-id-owner-user1 \
      --clusterrole transactional-id-owner \
      --user user1@example.com \
      --resource-common-instance ${instance_name} \
      --resource-common-cluster ${cluster_id} \
      --resource-kafka-transaction-id "startsWith('${transactional_id_prefix}')"
    ```

    You can remove `--resource-kafka-transaction-id` to bind the role to all transactional ID in the specified cluster:

    ```shell theme={null}
    snctl create rolebinding transactional-id-owner-user1 \
      --clusterrole transactional-id-owner \
      --user user1@example.com \
      --resource-common-instance ${instance_name} \
      --resource-common-cluster ${cluster_id}
    ```

    <Note title="Note">
      The `srn.cluster` flag requires the **Cluster ID**, not the **Cluster Name**. The Cluster ID is a unique, randomly generated string (for example, `pc-y7bti`, `c-6mhjbx2`) and can be found on the cluster overview page in the StreamNative Cloud Console.
    </Note>
  </Tab>

  <Tab title="Terraform">
    * You can refer [StreamNative Terraform](https://github.com/streamnative/terraform-provider-streamnative/blob/main/examples/rolebinding/main.tf) get more details.
    * To bind a role to [Service Accounts](/cloud/security/authentication/service-accounts/service-accounts), use the `service_account_names` argument instead of `user_names`.
    * The example below binds the `transactional-id-owner` role. To bind a different transactional role, replace `transactional-id-owner` in the `cluster_role_name` argument and in the role binding's name.

    ```hcl theme={null}
    terraform {
      required_providers {
        streamnative = {
          version = "0.10.0"
          source  = "streamnative/streamnative"
        }
      }
    }

    provider "streamnative" {
      # Please replace path use your own key file path
      key_file_path = "/path/to/your/service/account/key.json"
    }

    resource "streamnative_rolebinding" "transactional_id_admin_role_binding" {
      organization      = "${The ID of your organization}"
      name              = "transactional-id-owner-user1"
      cluster_role_name = "transactional-id-owner"
      # To bind to a service account, use 'service_account_names' instead
      user_names        = ["user1@example.com"]
      resource_name_restriction {
        common_instance = "${instance_name}"
        common_cluster  = "${cluster_id}"
        kafka_transaction_id  = "${transactional_id}"
      }
    }
    ```

    You can use a `startsWith` macros to bind a role to multiple transactional ID. For example, grants the `transactional-id-owner` role to all transactional ID that start with the prefix `${transactional_id_prefix}`.

    ```hcl theme={null}
    #... terraform streamnative basic configuration

    resource "streamnative_rolebinding" "transactional_id_admin_role_binding" {
      organization      = "${The ID of your organization}"
      name              = "transactional-id-owner-user1"
      cluster_role_name = "transactional-id-owner"
      # To bind to a service account, use 'service_account_names' instead
      user_names        = ["user1@example.com"]
      resource_name_restriction {
        common_instance = "${instance_name}"
        common_cluster  = "${cluster_id}"
        kafka_transaction_id  = "startsWith('${transactional_id_prefix}')"
      }
    }
    ```

    <Note title="Note">
      The `srn.cluster` field requires the **Cluster ID**, not the **Cluster Name**. The Cluster ID is a unique, randomly generated string (for example, `pc-y7bti`, `c-6mhjbx2`) and can be found on the cluster overview page in the StreamNative Cloud Console.
    </Note>
  </Tab>
</Tabs>

### transactional-id-owner

The Transactional ID Administrator. Grants administrative control over specific Kafka transactional IDs, which are persistent identities used for exactly once semantics.

* Describe Transactions: Can view the state of active transactions associated with a given transactional ID.
* Manage ID Lifecycle: Can manage the lifecycle of the transactional ID resource itself. This is an administrative function for platform health, not for application use.


# Role-Based Access Control (RBAC) on StreamNative Cloud
Source: https://docs.streamnative.io/cloud/security/access/rbac/rbac-overview



<Note title="Prerequisite">
  RBAC requires Pulsar 3.3.2.5, 4.0.0.9, or later.
</Note>

Role-based access control (RBAC) allows you to control what level of access users have to your organization's resources, including but not limited to instances, clusters, tenants, namespaces, topics, schemas, service accounts, and more. Use RBAC to protect your StreamNative Cloud resources and data by authorizing and restricting access to **principals** and by delegating access authorization to the appropriate business units and teams in your organization.

## Prerequisites

Before using RBAC, it is important to understand the following concepts:

* [Resources on StreamNative Cloud](/cloud/security/access/resource-hierarchy/resources-overview)
* [Organizations](/cloud/security/access/resource-hierarchy/organizations)
* [Instances](/cloud/clusters/manage-instances/instance)
* [Infrastructure Pools](/cloud/clusters/streamnative-cluster-overview#infrastructure-pools)
* [User Accounts](/cloud/security/authentication/user-accounts)
* [Service Accounts](/cloud/security/authentication/service-accounts/service-accounts)
* [Identity Pools](/cloud/security/authentication/oidc-identity-providers/manage-oidc-identity-pools)

## Concepts

In RBAC, there are a few key concepts:

### Principal

A principal is an entity that can be granted access to resources. Principals can be [User Accounts](/cloud/security/authentication/user-accounts), [Service Accounts](/cloud/security/authentication/service-accounts/service-accounts), or [Identity Pools](/cloud/security/authentication/oidc-identity-providers/manage-oidc-identity-pools).

### [Predefined Role](/cloud/security/access/rbac/manage-rbac-roles)

A predefined role defines the boundary of permissions it can operate, and a predefined role can be assigned to principals.

### [Role binding](/cloud/security/access/rbac/manage-rbac-role-bindings)

A role binding assigns a predefined role to a principal. Role bindings are used to grant permissions to principals.

## RBAC Workflow

The RBAC workflow consists of two main parts:

### RBAC Management

Users can manage (`create`, `update`, or `delete`) RBAC roles and role bindings using:

* [Terraform](https://github.com/streamnative/terraform-provider-streamnative/blob/main/examples/rolebinding/main.tf)
* [StreamNative CLI](/tools/cli/snctl/snctl-overview)
* [Cloud API](/api-references/cloudapi/cloud-api)
* [Cloud Console](/cloud/get-started/cloud-console)

Once resources are created, the StreamNative Cloud control plane automatically monitors them and propagates any changes to different components and resources for validation.

### RBAC Validation

Each component in both the control plane and data plane receives the RBAC settings (roles and bindings) and uses them for validation. For example, when a user attempts to produce to a topic, the action is validated against the RBAC settings. If the user does not have the necessary permissions, the action is rejected.


# Use Pulsar ACLs with RBAC on StreamNative Cloud
Source: https://docs.streamnative.io/cloud/security/access/rbac/use-pulsar-acls-with-rbac



<Note title="Prerequisite">
  RBAC requires Pulsar 3.3.2.5, 4.0.0.9, or later.
</Note>

You can use RBAC role bindings together with [Pulsar ACLs](/cloud/security/access/access-control-lists/authorization-and-acls) to control access to Pulsar resources. Principals (users and service accounts) can be granted ACLs, RBAC role bindings, or both. The system considers all granted permissions when determining whether a principal can perform a specific action.

When RBAC is enabled, the following changes apply:

* Users no longer have implicit **Super Admin (Super User)** access to Pulsar clusters. They only have permissions that are explicitly granted.
* You can grant granular permissions to resources by applying ACLs or RBAC role bindings to principals.
* Both ACLs and RBAC role bindings can be used with users and service accounts to grant fine-grained access to resources.

## ACLs vs RBAC

The following table summarizes which principals can be granted each type of access control:

| Principal Type  | ACLs | RBAC Role Bindings |
| --------------- | ---- | ------------------ |
| User            | Yes  | Yes                |
| Service Account | Yes  | Yes                |
| Identity Pool   | No   | Yes                |

## ACLs + RBAC Role Bindings

When used together, ACLs and RBAC role bindings are combined using a logical AND operation:

* ACLs evaluate whether a principal has permission to perform a specific action
* RBAC role bindings evaluate whether a principal has a role that grants permission for an action

For example, if a user has:

* An ACL that allows them to produce to a topic
* An RBAC role binding that allows them to consume from that topic

Then that user will be able to both produce to and consume from the topic, since they have both permissions explicitly granted through the different mechanisms.


# Organizations in StreamNative Cloud
Source: https://docs.streamnative.io/cloud/security/access/resource-hierarchy/organizations



In StreamNative Cloud, the Organization resource is the root node in the StreamNative Cloud resource hierarchy. If you have an [annual commitment](/cloud/billing/billing-overview#annual-commitments), there is a 1:1 relationship between one organization and one annual commitment.

Organizations might contain:

* One to many [users](/cloud/security/authentication/user-accounts)
* Zero to many [service accounts](/cloud/security/authentication/service-accounts/service-accounts)
* Zero to many [instances](/cloud/clusters/manage-instances/instance)

When you first sign up for StreamNative Cloud, a new organization is created simultaneously with the first user account that belongs to that organization. When you invite another user to StreamNative Cloud, the invited user is added to the existing organization. You can create other organizations as needed.

## Organization settings

In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Organizations** to check the organizations you belong to.

<img alt="screenshot of profile menu" />

In addition, you can perform the following operations:

* Create more organizations: click **Create organization** to add one or more organizations.
* Search for existing organizations: enter an organization's name in the **Search organization** field, and then press **Enter** to search for a specific organization.

### Cloud Organization ID

Each organization in StreamNative Cloud is uniquely identified by a Cloud Organization ID.

You can find the Cloud Organization ID from the query parameter `org` of any StreamNative Cloud Console URL.

### Organization profile

Organization profile page allows you to change the organization name, set the contact emails for billing and technical. You can get into the Organization profile page from the upper-right corner of the StreamNative Cloud Console:

<img alt="Organization profile path" />

On the Organization profile page, you can update the below informations:

* **Organization name**: this will change the organization display name on the StreamNative Cloud.
* **Billing contact email**: organization's billing email is where StreamNative Cloud sends billing notification and other billing-related communication.
* **Technical contact email**: organization's technical email is where StreamNative Cloud sends cluster notification and other technical-related communication.

<img alt="Organization profile page" />

## Manage Multiple Organizations

Usually, a combination of instances and role-based access control (RBAC) should be used to isolate different projects, teams, or other use cases instead of creating separate organizations. Depending on your requirements, you can optionally create multiple organizations in StreamNative Cloud. Some of the benefits include the following:

* You can create separate organizations to provide isolation for different business units (for example, projects and teams) without requiring the sharing of billing, adminstration, or anything else between them in the future.

* If your current organization pays using a cloud service provider's marketplace, and you need to create resources in a different cloud provider, then creating a new organization is an option.

  Note that an alternatively to consider is that you might be able to convert your existing organization to paying StreamNative directly, which allows creation of resources in any mix of cloud providers. To learn more, contact your StreamNative sales representative.

* Users can seamlessly switch between organizations they belong to.

### Limitations

The current implementation of multiple organizations support has the following limitations:

* To create an organization, you need to use the StreamNative Cloud console. For details, see [Create an organization](#create-an-organization).

* StreamNative Cloud resources cannot be moved between organizations. Resources (for example, clusters, connectors, and functions) cannot be moved from one organization to a different one.

### Create an organization

<Tabs>
  <Tab title="StreamNative Console">
    To create an organization, follow these steps.

    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Organizations**.

    2. Click **Create organization** and a dialog box displays.

    3. Enter the organization name and then click **Confirm**. An organization name must be less than 12 characters and can contain any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-). An organization is created successfully, as shown below.

       <img alt="screenshot of created organization" />
  </Tab>
</Tabs>

### Sign in to an organization

<Tabs>
  <Tab title="StreamNative Console">
    To sign in to a specific organization,

    1. Go to the StreamNative Cloud Console at [https://console.streamnative.cloud](https://console.streamnative.cloud).

    2. Sign in to StreamNative Cloud.

       You are signed in to the last organization you signed in to unless this is the first time signing in on your web browser. For the first time, you are signed in to your default organization, which is the first organization you become a member of.

    To sign in to a different organization, follow the procedure in [Switch between organizations](#switch-between-organizations).
  </Tab>

  <Tab title="StreamNative CLI">
    To sign in to a specific organization using the [StreamNative CLI (`snctl`)](/tools/cli/snctl/snctl-overview), use the following commands:

    1. Configure the target organization as the default organization.

       ```bash theme={null}
       snctl config set --organization <organization-id>
       ```

    2. Log in as a user

       ```bash theme={null}
       snctl auth login
       ```

    To find the Cloud Organization ID, see [Cloud Organization ID](/cloud/security/access/resource-hierarchy/organizations#cloud-organization-id).
  </Tab>
</Tabs>

<Note title="Quickly sign in to a specific organization using a bookmark">
  Create a bookmark in your web browswer for each organization you belong to. For the link, save the StreamNative Cloud Console URL with the query parameter for the organization ID (`org`), like this:

  ```
  https://console.streamnative.cloud?org=<organization-id>
  ```

  You can find the organization ID (`org`) value from the query parameter `org` of any StreamNative Cloud Console URL.
</Note>

### Switch between organizations

If you are a user who is a member of two or more organizations, you can switch between organizations using the StreamNative Cloud Console or StreamNative Cloud CLI. When you switch between organizations, you are signed out the current organization and signed in to the new organization.

<Tabs>
  <Tab title="StreamNative Console">
    To switch to a different organization using the StreamNative Cloud Console:

    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Organizations**.

    2. Find the name of the organization that you want to switch and click the Organization name. You are signed in to the organization you selected.

    You have switched to the organization you selected.
  </Tab>

  <Tab title="StreamNative CLI">
    To switch between organizations using the StreamNative CLI (`snctl`), reconfigure `snctl` to specify the organization you want to switch to as the default organization.

    1. Configure the target organization as the default organization.

       ```bash theme={null}
       snctl config set --organization <organization-id>
       ```

    2. Log in as a user

       ```bash theme={null}
       snctl auth login
       ```

    To find the Cloud Organization ID, see [Cloud Organization ID](/cloud/security/access/resource-hierarchy/organizations#cloud-organization-id).
  </Tab>
</Tabs>

### Leave an organization

If you no longer need to belong to an organization, you can be removed from an organization. Please reach out to the other users in the organization you want to leave, they can remove you from it. If you are the only user, you cannot remove yourself from the organization - contact [StreamNative Support](https://support.streamnative.io/hc/en-us/requests/new) and they can remove you from the organization.

### Delete an organization

Currently, you can't delete an organization through either the StreamNative Cloud Console or CLI. If you need to delete an organization, please [submit a ticket](https://support.streamnative.io/hc/en-us/requests/new).

### Manage users across organizations

To collaborate in `Example Org 1` with a team member who belongs to a different StreamNative Cloud organization (`Example Org 2`), you can invite the user to your organization. Users can switch back and forth between organizations they belong to.


# Resources on StreamNative Cloud
Source: https://docs.streamnative.io/cloud/security/access/resource-hierarchy/resources-overview



StreamNative Cloud offers a variety of resources to help you build and manage data streaming applications. This section provides information about the top-level resources, organizations, instances, and workspaces, that you can use to organize your Pulsar/Kafka clusters, compute jobs (connectors, functions, and Flink jobs), and other StreamNative Cloud resources.

## Organizations

The top-level resource in StreamNative Cloud is the organization. An organization is a logical grouping of resources that you can use to manage access to your StreamNative Cloud resources and to organize your resources in a way that makes sense for your organization. Most users will only need one organization, but you can create multiple organizations if you need to separate resources for different departments, teams, or customers.

To learn more about organizations, see [Organizations](/cloud/security/access/resource-hierarchy/organizations).

## Pools

**Infrastructure Pools** (abbreviated as **Pools**) are the underlying infrastructure resources that run workloads. Each pool encompasses a collection of infrastructure environments, known as **Pool Members** (abbreviated as **PMs**), distributed across multiple regions within a cloud provider. These **Pool Members** are equivalent to Kubernetes clusters dedicated to running workloads (such as Pulsar/Kafka clusters, Flink jobs, and other cloud resources). Organizations can have multiple pools to support different environments, such as development, testing, and production.

To learn more about pools, see [Infrastructure Pools](/cloud/clusters/streamnative-cluster-overview#infrastructure-pools).

## Workloads

StreamNative Cloud supports two major types of workloads: data workloads and compute workloads. Data workloads represents the running Kafka/Pulsar clusters that provides data storage and streaming capabilities. Compute workloads represents all the running computing jobs such as Flink jobs that can process data in real-time.

These workloads are logically grouped into instances and workspaces, respectively. They are allocated to different infrastructure pools to run.

### Instances

Within each StreamNative Cloud organization, you can create one or more instances. Each instance can contain multiple clusters and deployed components, such as connectors and functions. Different departments or teams can use separate instances to isolate their resources and avoid interfering with each other. Organizations also often create multiple instances to support different environments, such as development, testing, and production.

To learn more about instances, see [Instances](/cloud/clusters/manage-instances/instance).

### Workspaces

<Note title="Note">
  Workspaces are introduced as part of Managed Flink service. It is currently in Private Preview. Please join our [Early Access Program](https://hs.streamnative.io/early-access-program-for-streamnative) to try it out.
</Note>

While an instance is a logical grouping of data workloads (i.e., Pulsar/Kafka clusters), a workspace is a logical grouping of compute resources. Currently, workspaces are only supported for Flink jobs.


# Manage Authentication on StreamNative Cloud
Source: https://docs.streamnative.io/cloud/security/authentication/authentication-overview



StreamNative Cloud offers multiple authentication methods to ensure secure access to your resources. This includes managing user accounts and workload identities, which encompass service accounts, API keys, and identity providers (OIDC and mTLS).

## User accounts

User accounts are the primary means of authentication for individuals accessing StreamNative Cloud resources. These accounts can be managed through the StreamNative Cloud Console or `snctl`, where administrators can invite users, assign roles, and manage permissions. User accounts can also be integrated with Single Sign-On (SSO) providers for enhanced security and convenience.

See [Manage User Accounts](/cloud/security/authentication/user-accounts) for more details.

## AuthV2 organization audience

Some StreamNative Cloud features use AuthV2 organization-scoped tokens. An AuthV2 organization token includes an organization audience in the form `urn:sn:cloud:<organization>`. Services use this audience to bind a request to one organization before evaluating RBAC.

For example, the Remote StreamNative MCP Server root endpoint (`https://mcp.streamnative.cloud/mcp`) and organization endpoint (`https://mcp.streamnative.cloud/mcp/x/<organization>`) require AuthV2 organization authentication. Non-interactive clients should use [API Key v2](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#api-key-v1-vs-api-key-v2) when they need organization-scoped access.

## Service accounts

Service accounts represent applications or services that need to access StreamNative Cloud resources programmatically. They are not tied to individual users, making them ideal for automated workflows and integrations. Service accounts can own API keys and have specific permissions assigned through ACLs or role bindings.

See [Manage Service Accounts](/cloud/security/authentication/service-accounts/service-accounts) for more details.

### API Keys

API Keys are used to authenticate service accounts to StreamNative Cloud components and resources. Each API key is a JWT compliant token that contains the service account's identity and credentials, and can be scoped to specific StreamNative Cloud resources. API keys can be managed using StreamNative Cloud Console, `snctl`, or the StreamNative Cloud API.

See [Manage API Keys](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview) for more details.

## Identity providers

<Note title="Note">
  Identity providers support is currently in Private Preview. If you are interested in this feature, please join our [Early Access Program](https://hs.streamnative.io/early-access-program-for-streamnative).
</Note>

Identity providers enable applications and services to authenticate to StreamNative Cloud using external identity providers.

### Available identity providers

StreamNative Cloud supports the following identity providers:

* [Use OAuth/OIDC to Authenticate to StreamNative Cloud](/cloud/security/authentication/oidc-identity-providers/oidc-federation-overview)
* [Use Mutual TLS (mTLS) to Authenticate to StreamNative Cloud](/cloud/security/authentication/mtls-identity-providers/mtls-overview)


# Use Mutual TLS (mTLS) to Authenticate to StreamNative Cloud
Source: https://docs.streamnative.io/cloud/security/authentication/mtls-identity-providers/mtls-overview



<Note title="Note">
  mTLS authentication is currently in Private Preview and is only available for **BYOC Pro** clusters using the MQTT protocol. If you are interested in this feature, please join our [Early Access Program](https://hs.streamnative.io/early-access-program-for-streamnative).
</Note>

You can use mutual TLS (mTLS) for X.509 client certificate authentication and granular access control to **BYOC Pro** clusters.

## Key benefits

* **Two-way authentication**: Both the client and server must verify each other's identity using X.509 certificates, providing stronger security than one-way TLS.
* **Certificate-based access**: Access to StreamNative Cloud resources requires valid X.509 certificates, making unauthorized access more difficult.
* **Secure data transmission**: TLS encryption protects data in transit from tampering and interception.
* **Fine-grained control**: Manage access at a granular level by issuing certificates to specific clients and applications.
* **Regulatory compliance**: Meet industry security standards and compliance requirements with strong authentication.
* **Alternative to passwords**: Reduce security risks by using certificates instead of traditional username/password authentication.

## How to access this feature

Currently, this feature is only available for **BYOC Pro** clusters using the MQTT protocol. If you are interested in this feature, please join our [Early Access Program](https://hs.streamnative.io/early-access-program-for-streamnative).


# Best Practices for OAuth/OIDC Identity Providers
Source: https://docs.streamnative.io/cloud/security/authentication/oidc-identity-providers/best-practices-for-oidc-identity-providers



When using OIDC Federation on StreamNative Cloud, your OAuth/OIDC identity provider handles all identity management. While this provides flexibility in managing users and their resource access, the overall security depends heavily on how well your identity provider is configured and protected. This guide outlines key best practices for securely managing your OAuth/OIDC identity provider.

## Managing a Single Identity Provider Across Multiple Organizations

When using the same OAuth identity provider (sharing the same Issuer URI and JWKS URI) across multiple organizations under your control, be aware that access tokens issued by this provider can potentially be used across all organizations. To maintain proper access control:

* Implement pool filters to restrict token usage to specific operations or purposes within each organization
* Carefully plan and document your token usage strategy across organizations
* Regularly audit access patterns to detect any unauthorized cross-organization usage

## Implementing Identity Pool Filters

Always configure [identity pool filters](/cloud/security/authentication/oidc-identity-providers/manage-oidc-identity-pools#identity-pool-filters) for your identity pools. These filters provide granular access control by:

* Using claims like `aud` (audience) and `scp` (scope) to define precise access boundaries
* Restricting operations to specific purposes or functions
* Creating logical groupings of permissions based on business needs

## Restricting Access Using Claims

To limit token usage to specific organizations, configure your identity provider to issue tokens with organization-specific claims. Here's how:

1. Configure your identity provider to issue tokens with specific claim values (such as `aud`, `scp`, or custom claims)
2. Set up identity pool filters in each organization to match these claims
3. Even though the token signature remains valid across all organizations, access is granted only when token claims match an organization's identity pool filters

This approach ensures that tokens are only valid within their intended organizational context, despite sharing a common identity provider.


# Configure OAuth Clients
Source: https://docs.streamnative.io/cloud/security/authentication/oidc-identity-providers/configure-oauth-clients



Follow the instructions below to configure Pulsar or Kafka clients to use OAuth 2.0 with your OIDC-compliant identity provider to connect to StreamNative Cloud clusters.

* [Configure Pulsar Clients with OAuth 2.0](/cloud/security/authentication/service-accounts/use-oauth/configure-pulsar-clients-with-oauth-20)
* [Configure Kafka Clients with OAuth 2.0](/cloud/security/authentication/service-accounts/use-oauth/configure-kafka-clients-with-oauth-20)


# Manage Identity Pools on StreamNative Cloud
Source: https://docs.streamnative.io/cloud/security/authentication/oidc-identity-providers/manage-oidc-identity-pools



<Note title="Note">
  This feature is currently in **Private Preview**. To access this feature, you need to join our [Early Access Program](https://hs.streamnative.io/early-access-program-for-streamnative).

  Please note that currently, StreamNative Cloud only supports managing identity pools through [`snctl`](/tools/cli/snctl/snctl-overview). Support for the Console and Terraform will be available soon.
</Note>

Once you have registered an OAuth/OIDC identity provider, you can use identity pools to manage the access of external application identities. An identity pool is a group of external application identities that are assigned a certain level of access based on a claims-based policy. The use of identity pools is defined by the pool filter expression. Access is controlled by using [role-based access control (RBAC)](/cloud/security/access/rbac/cloud-rbac).

You can use the following instructions to create, read, update, list, and delete identity pools.

## Prerequisites

Before managing identity pools, ensure you have:

* A StreamNative Cloud account with **Super Admin** privileges
* An OAuth/OIDC identity provider [registered](/cloud/security/authentication/oidc-identity-providers/manage-oidc-identity-providers#register-an-o-auth-oidc-identity-provider)

## Create an identity pool

<Tabs>
  <Tab title="snctl">
    You can create an identity pool by running the following command. Note that the `provider-name` must be the name of an existing OAuth/OIDC identity provider. The `expression` parameter specifies which identities can authenticate using this pool based on their claims. See [identity pool filters](#identity-pool-filters) for more information about the expression syntax.

    ```bash theme={null}
    snctl create identitypool <pool-name> \
      --description '<pool description>' \
      --provider-name '<provider name>' \
      --expression 'claims.sub==abc'
    ```

    Alternatively, you can prepare a manifest file `identitypool.yaml` as follows:

    ```yaml theme={null}
    apiVersion: cloud.streamnative.io/v1alpha1
    kind: IdentityPool
    metadata:
      name: <pool-name>
      namespace: <namespace>
    spec:
      description: '<pool-description>'
      expression: 'claims.sub==abc'
      providerName: '<provider-name>'
    ```

    Then, you can create an identity pool by running the following command:

    ```bash theme={null}
    snctl create identitypool -f identitypool.yaml
    ```

    You can use `snctl get identitypool` to check the status of the identity pool.

    ```bash theme={null}
    snctl get identitypool <pool-name> -o yaml
    ```

    You should be able to see the status of the identity pool as `Ready`.
  </Tab>
</Tabs>

## Update an identity pool

You can update the description and filter expression of an identity pool. However, when you update the filter expression, ensure that the change does not disrupt applications and services that use the identity pool.

To update an identity pool, follow these steps:

<Tabs>
  <Tab title="snctl">
    You can update an identity pool by editing the identity pool object with `snctl edit` or by editing the manifest file `identitypool.yaml` and then applying the changes by running the following command:

    ```bash theme={null}
    snctl apply -f identitypool.yaml
    ```

    You can check the status of the identity pool by running the following command:

    ```bash theme={null}
    snctl get identitypool <pool-name> -o yaml
    ```
  </Tab>
</Tabs>

## Delete an identity pool

<Warning title="Warning">
  Deleting an identity pool is irreversible and will remove all the information associated with the identity pool. This can cause disruption to the applications and services that use the identity pool.
</Warning>

<Tabs>
  <Tab title="snctl">
    You can delete an identity pool by running the following command:

    ```bash theme={null}
    snctl delete identitypool <pool-name>
    ```
  </Tab>
</Tabs>

## List identity pools

<Tabs>
  <Tab title="snctl">
    You can list all identity pools by running the following command:

    ```bash theme={null}
    snctl get identitypool
    ```

    You should be able to see the list of identity pools.
  </Tab>
</Tabs>

## Identity pool filters

Identity pool filters allow you to control which identities can authenticate using an identity pool by evaluating their OIDC claims. Each identity pool requires at least one filter to determine which identities are allowed access.

The filters are written using [Common Expression Language (CEL)](https://github.com/google/cel-spec), a simple but powerful expression language that evaluates claims from the OIDC provider.

### CEL filter expressions

The following table lists the supported CEL filter expressions and how to use them:

All token fields used in filter definitions must be prefixed with `claims`. For development purposes, you can temporarily set the filter to `true` to allow all identities with a valid token to authenticate.

| Use case        | CEL expression                        |
| --------------- | ------------------------------------- |
| Equality        | `claims.iss == "google"`              |
| Inclusion       | `claims.appid in ["app1", "app2"]`    |
|                 | `!(claims.appid in ["app1", "app2"])` |
|                 | `'admins' in claims.groups`           |
|                 | `!('admins' in claims.groups)`        |
| Presence check  | `has(claims.iss)`                     |
|                 | `!has(claims.iss)`                    |
| Prefix matching | `claims.principal.startsWith("user")` |
| Suffix matching | `claims.principal.endsWith("user")`   |

For more complex use cases, use the following operators:

| Use case    | Operator precedence | CEL expression                                            |
| ----------- | ------------------- | --------------------------------------------------------- |
| Logical NOT | 1                   | `!(claims.iss == "google")`                               |
| Logical AND | 2                   | `claims.iss == "google" && claims.principal == "user1"`   |
| Logical OR  | 3                   | `claims.iss == "google" \|\| claims.principal == "user1"` |

The rules can grouped into parentheses to form more complex expressions, like this `Expression && (Expression || Expression)`. For example:

```yaml theme={null}
claims.iss == "google" && (claims.principal == "user1" || claims.principal == "user2")
```

## Grant access to an identity pool

<Note title="Note">
  To grant access to an identity pool, you need to have the RBAC feature enabled for your StreamNative organization.
</Note>

After creating an identity pool, you can grant access to it by creating a role binding.

```bash theme={null}
snctl create rolebinding <binding-name> --role <role-name> --identitypool <identitypool-name>
```

Learn more about role bindings in [RBAC](/cloud/security/access/rbac/cloud-rbac).

### Verify OIDC Token

Now, you can verify whether an exchanged OIDC token from your OAuth/OIDC identity provider is able to match the identity pool. You can use the following command to verify it:

```bash theme={null}
curl -X GET <broker-admin-service-url>/admin/sn/oidc/v1/oidc-test/getPrincipal -H "Authorization: Bearer <super-user-api-key>" -H "Content-Type: application/json" -H "token: '<OIDC-token>'"
```

Please replace the following placeholders with your actual values:

* `<broker-admin-service-url>`: The URL of the broker admin service of your StreamNative Cloud cluster.
* `<super-user-api-key>`: The API key of a **Service Account** with the **Super Admin** permission.
* `<OIDC-token>`: The OIDC token exchanged from your OAuth/OIDC identity provider to verify.

### Verify the role binding of an identity pool

You can verify the role binding of an identity pool by running the following command:

```bash theme={null}
curl -X GET <broker-admin-service-url>/admin/sn/rbac/v1/role-bindings/<identitypool-name> -H "Authorization: Bearer <super-user-api-key>" -H "Content-Type: application/json"
```

Please replace the following placeholders with your actual values:

* `<broker-admin-service-url>`: The URL of the broker admin service of your StreamNative Cloud cluster.
* `<super-user-api-key>`: The API key of a **Service Account** with the **Super Admin** permission.
* `<identitypool-name>`: The name of the identity pool to verify.


# Manage OAuth/OIDC Identity Providers on StreamNative Cloud
Source: https://docs.streamnative.io/cloud/security/authentication/oidc-identity-providers/manage-oidc-identity-providers



<Note title="Note">
  This feature is currently in **Private Preview**. To access this feature, you need to join our [Early Access Program](https://hs.streamnative.io/early-access-program-for-streamnative).

  Please note that currently, StreamNative Cloud only supports managing OAuth/OIDC identity providers through [`snctl`](/tools/cli/snctl/snctl-overview). Support for the Console and Terraform will be available soon.
</Note>

You can register an OAuth/OIDC identity provider on StreamNative Cloud to grant applications and services access to StreamNative Cloud resources. A registered OAuth/OIDC identity provider uses the industry standard OAuth 2.0 and OpenID Connect (OIDC) protocols to authenticate users and services, reduce operational overhead, and improve security.

## Prerequisites

Before registering an OAuth/OIDC identity provider, ensure you have:

* An account with the identity provider you want to register
* A StreamNative Cloud account with **Super Admin** privileges
* The following information from your identity provider:
  * **OIDC Discovery URL**: This URL is used to import the metadata needed for configuring your OIDC identity provider. The Discovery URL automatically provides both the JWKS URI and Issuer URI information. Note that StreamNative Cloud currently only supports configuration via Discovery URL.
* Configured your identity provider to allow access from StreamNative Cloud

## Obtain the OIDC Discovery URL

To obtain the OIDC discovery URL for an OIDC provider, you typically need to know the base URL of the provider. The discovery URL is constructed by appending `/.well-known/openid-configuration` to the base URL. Here's how you can find it:

### Check the OIDC Provider's Documentation

Many OIDC providers specify their discovery URLs in their documentation. Common examples include:

* **Google**: `https://accounts.google.com/.well-known/openid-configuration`
* **Auth0**: `https://<your-auth0-domain>/.well-known/openid-configuration`
* **Okta**: `https://<your-okta-domain>/.well-known/openid-configuration`
* **Azure AD**: `https://login.microsoftonline.com/<your-tenant-id>/v2.0/.well-known/openid-configuration` (where `<your-tenant-id>` is your Azure AD tenant ID or "common" for multi-tenant)

### Manually Construct the Discovery URL

If you know the base domain of your OIDC provider, construct the discovery URL like this:

```bash theme={null}
<base-url>/.well-known/openid-configuration
```

Examples:

* `https://example-oidc-provider.com/.well-known/openid-configuration`

### Test the URL

Once you have the discovery URL:

* Open it in a browser or use a tool like `curl` or `wget` to ensure it returns a JSON configuration.

  ```bash theme={null}
  curl https://example-oidc-provider.com/.well-known/openid-configuration
  ```

### Contact Your Provider

If you cannot find the base URL or documentation, contact your provider's support team or administrator to obtain the correct discovery URL.

For organizations using a custom or private OIDC implementation, the discovery URL will be specific to your deployment. Contact your OIDC provider administrator to obtain the appropriate URL.

## Register an OAuth/OIDC identity provider

<Tabs>
  <Tab title="snctl">
    You can register an OAuth/OIDC identity provider by running the following command:

    ```bash theme={null}
    snctl create oidcprovider <provider-name> \
      --description '<provider-description>' \
      --discovery-url '<discovery-url>'
    ```

    Alternatively, you can prepare a manifest file `oidc-provider.yaml` for the identity provider as follows:

    ```yaml theme={null}
    apiVersion: cloud.streamnative.io/v1alpha1
    kind: OIDCProvider
    metadata:
      name: <provider-name>
      namespace: <organization-id>
    spec:
      description: '<provider-description>'
      discoveryUrl: '<discovery-url>'
    ```

    Then, create the identity provider by running the following command:

    ```bash theme={null}
    snctl create oidcprovider -f oidc-provider.yaml
    ```

    Once the identity provider is created, you can check the status of the identity provider by running the following command:

    ```bash theme={null}
    snctl get oidcprovider <provider-name> -o yaml
    ```

    You should be able to see the status of the identity provider as `Ready`.
  </Tab>
</Tabs>

## Update an OAuth/OIDC identity provider

You can update the description and discovery URL of an OAuth/OIDC identity provider. When updating the discovery URL, ensure that StreamNative Cloud can still access the old discovery URL during the update process to avoid disrupting applications and services that use the identity provider.

To update an OAuth/OIDC identity provider, follow these steps:

<Tabs>
  <Tab title="snctl">
    You can use `snctl edit <oidc-provider-name>` to update the description and discovery URL of an OAuth/OIDC identity provider.

    Alternatively, you can edit the manifest file `oidc-provider.yaml` and then apply the changes by running the following command:

    ```bash theme={null}
    snctl apply -f oidc-provider.yaml
    ```

    You can also check the status of the identity provider by running the following command:

    ```bash theme={null}
    snctl get oidcprovider <oidc-provider-name> -o yaml
    ```
  </Tab>
</Tabs>

## Delete an OAuth/OIDC identity provider

<Warning title="Warning">
  Deleting an OAuth/OIDC identity provider is irreversible and will remove all the information associated with the identity provider. This can cause disruption to the applications and services that use the identity provider.
</Warning>

<Tabs>
  <Tab title="snctl">
    You can delete an OAuth/OIDC identity provider by running the following command:

    ```bash theme={null}
    snctl delete oidcprovider <oidc-provider-name>
    ```
  </Tab>
</Tabs>


# Use OAuth/OIDC to Authenticate to StreamNative Cloud
Source: https://docs.streamnative.io/cloud/security/authentication/oidc-identity-providers/oidc-federation-overview



<Note title="Note">
  OAuth/OIDC Federation is currently in Private Preview and is only available for **BYOC Pro** clusters. If you are interested in this feature, please join our [Early Access Program](https://hs.streamnative.io/early-access-program-for-streamnative).
</Note>

StreamNative Cloud uses its own OAuth2 provider for authenticating [User Accounts](/cloud/security/authentication/user-accounts) and [Service Accounts](/cloud/security/authentication/service-accounts/service-accounts). However, you may want to use your existing OAuth/OIDC-compliant identity provider (IdP) for authentication instead.

OAuth/OIDC Federation enables this capability by allowing you to configure StreamNative Cloud to authenticate users through your organization's existing OAuth/OIDC-compliant identity provider.

## Key Features

* Manage application identities and credentials through your own identity provider
* Authenticate to StreamNative Cloud resources using secure, short-lived JSON Web Tokens (JWTs)
* Leverage StreamNative's OIDC Federation service to securely integrate with your identity provider using standards-compliant tokens based on the [OAuth 2.0 Authorization Framework \[RFC 6749\]](https://tools.ietf.org/html/rfc6749) and [OpenID Connect (OIDC)](https://openid.net/connect/)
* Configure identity pools to map group memberships and other attributes to access policies (RBAC or ACLs)

## Feature availability and limitations

* This feature is currently available only for **BYOC Pro** clusters for Private Preview. To get access, please join our [Early Access Program](https://hs.streamnative.io/early-access-program-for-streamnative).
* Identity pools and the ACLs for identity pools can be managed only by using [`snctl`](/tools/cli/snctl/snctl-overview) or the [Cloud API](/api-references/cloudapi/cloud-api).

## OIDC Federation Flow

The following diagram illustrates the OIDC Federation flow for an organization.

<img alt="OIDC Federation flow" />

The OIDC Federation process consists of these key steps:

1. **Configure OAuth/OIDC Identity Provider**:

   To use OIDC Federation, you must first establish trust between StreamNative Cloud and your identity provider by adding it as a trusted identity provider in StreamNative Cloud. This involves:

   * Defining the identity provider type
   * Creating a trust relationship between StreamNative Cloud and your identity provider
   * Adding the claims needed for authentication and authorization

2. **Create an Identity Pool and Access Policies**:

   Create an identity pool to represent a group of external identities. This allows you to assign appropriate access levels through policies.

3. **Configure Clients to Use Your Identity Provider**:

   Obtain the **Client ID** and **Client Secret** from your identity provider to configure clients for OAuth2 authentication. For more details, see [Use OAuth to Authenticate to StreamNative Cloud](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview).

4. **Token Validation and Authorization**:

   StreamNative Cloud validates incoming tokens using the trust JSON Web Key Set (JWKS), extracts the authenticated ID (`sub`) or other configured claims, and matches the authenticated identity to the appropriate identity pool for authorization.


# Manage Service Accounts for StreamNative Cloud
Source: https://docs.streamnative.io/cloud/security/authentication/service-accounts/manage-service-accounts



The sections below provide detailed instructions on managing service accounts (SAs) in StreamNative Cloud, including creating and managing service accounts using StreamNative Cloud Console, `snctl`, and StreamNative Terraform Provider.

## Create a service account

When creating a service account, you can optionally enable **Super Admin** access. A service account with **Super Admin** privileges has full management capabilities over all resources within the organization. By default, service accounts are created without **Super Admin** access.

<Note title="Note">
  Service accounts cannot be edited after creation. If you need a service account to have Super Admin access, make sure to enable it during the initial creation.
</Note>

You can create a service account using the following methods:

<Tabs>
  <Tab title="Console">
    1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Accounts & Accesses**.

    2. On the left navigation pane, click **Service Accounts**.

    3. On the **Service Account** page, click **+ New Service Account**.

    4. On the **Create Service Account** dialog, enter a name for the service account.

    5. Click **Create**.

    6. Click the `Access` button to grant roles to the service account. For details on available roles, see [Predefined Role](/cloud/security/access/rbac/manage-rbac-roles)

    After you have created a service account, you can check the details of the service account.

    The **Service Accounts** page displays all the created service accounts.

    The table below describes the details that you can view about the service account.

    | Item                    | Description                                                                                      |
    | ----------------------- | ------------------------------------------------------------------------------------------------ |
    | Name                    | The name of the service account.                                                                 |
    | Principal Name          | The principal name of the service account used for identification during authentication.         |
    | Active API tokens       | The number of active API keys for the service account.                                           |
    | Create Time             | The time when the service account was created.                                                   |
    | ServiceAccount Bindings | The list of service account bindings.                                                            |
    | Status                  | The status of the service account.                                                               |
    | Admin                   | Whether the service account has **Super Admin** enabled or not.                                  |
    | Access                  | Click to manage the roles bound to this service account (add new roles or remove existing ones). |
  </Tab>

  <Tab title="snctl">
    You can create a service account using the [`snctl` CLI](/tools/cli/snctl/snctl-overview). Please make sure that you have installed and configured the `snctl` CLI.

    Create the service account.

    ```bash theme={null}
    snctl create serviceaccount ${service_account_name}
    ```

    After creating a service account, you must bind a role to it to grant permissions. For use all predefined roles, see [Predefined Role](/cloud/security/access/rbac/manage-rbac-roles). The following example shows how to bind the `admin` role.

    Make sure that the `${service_account_name}` is the one you created in the previous step. The `${rolebinding_name}` should be a unique identifier for the role binding within your organization. For ease of reference, it is recommended to use the format `${service_account_name}_${role}` as the identifier.

    ```
    snctl create rolebinding ${rolebinding_name} \
      --clusterrole admin \
      --serviceaccount ${service_account_name}
    ```
  </Tab>
</Tabs>

After you have created a service account, you need to:

1. Grant your service account with necessary permissions. See [Control Access to StreamNative Cloud](/cloud/security/access/access-control-overview).
2. Configure your applications to use the service account to authenticate to StreamNative Cloud:
   * [Use OAuth to Authenticate to StreamNative Cloud](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview)
   * [Use API Keys to Authenticate to StreamNative Cloud](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview)

## Delete a service account

<Tabs>
  <Tab title="Console">
    To delete a service account, follow these steps.

    1. On the left navigation pane, click **Service Accounts**.

    2. Click the ellipsis at the end of the row of the service account you want to delete, and then select **Delete**.

       <img alt="screenshot showing the ellipsis at the end of the service account details row" />

    3. On the dialog box asking, *Are you sure you want to delete this service account?*, click **Confirm**.
  </Tab>
</Tabs>


# Service Accounts for StreamNative Cloud
Source: https://docs.streamnative.io/cloud/security/authentication/service-accounts/service-accounts



Each service account represents an application programmatically accessing StreamNative Cloud.

You can manage application access to StreamNative Cloud by using service accounts. Permissions can be specified using [ACLs](/cloud/security/access/access-control-lists/authorization-and-acls) and role bindings tied to a specific service account. ACLs and role bindings for service accounts are set by an administrator or another user with a similar role within the organization.

Service accounts are an [organization-level](/cloud/security/access/resource-hierarchy/organizations) resource.

Service accounts span the entire organization and can own API keys for many different resources, including development and production clusters. A typical use case has one team administering the StreamNative Cloud platform and issuing service accounts (with ACLs applied) to various application teams that use the data streaming platform. While service accounts cannot sign in to StreamNative Cloud Console, they can own any type of [API keys](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview) that can be used for CLI or API access. Keep in mind the following:

* Although users can leave or change roles within a company, applications continue to operate independently of the users, service accounts are especially useful in organizations requiring special identifiers for applications or services not be tied to a specific user.

* You can create service accounts using any of the following methods:
  * [StreamNative Cloud Console](https://console.streamnative.cloud): When creating service accounts using StreamNative Cloud Console, you can create API keys for the service account.
  * [StreamNative CLI](/tools/cli/snctl/snctl-tutorials) command `snctl create serviceaccount`.
  * StreamNative Cloud API
  * [StreamNative Terraform Provider](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/resources/apikey)

<Warning title="Warning">
  When you delete a service account, all associated API keys will also be deleted. Any client applications using a deleted API key will lose access, which may cause an outage for your streaming application. Always confirm that none of the API keys owned by an account are in active use before deleting a service account.
</Warning>

## Authentication Methods

Service accounts can authenticate to StreamNative Cloud using the following methods:

* [Use OAuth to Authenticate to StreamNative Cloud](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview)
* [Use API Keys to Authenticate to StreamNative Cloud](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview)


# Best Practices for Using API Keys in StreamNative Cloud
Source: https://docs.streamnative.io/cloud/security/authentication/service-accounts/use-api-keys/api-keys-best-practices



Review the following recommendations for best practices when using StreamNative Cloud API keys and incorporate them into your security strategy.

## Delete unneeded API keys and service accounts

As a standard practice of your security strategy, you should regularly review and clean up your existing API keys and service accounts.

To better understand the which API keys are being used, you can review and monitor authorization and authentication events in [StreamNative Cloud audit logs](/cloud/security/monitor-activity/cloud-audit-log).

## Rotate API keys regularly

Access to your StreamNative Cloud resources is controlled by API keys associated with service accounts, which have access controls determining what the service account has access to. API keys can be created and destroyed without affecting the service account ACLs and RBAC role bindings. Rotating API keys is a good security practice that provides access to a resource and limits the potential impact of an API key that is leaked.

When you rotate API keys, you perform the following steps:

1. Create a new API key,
2. Update the resource or application to use the new API key.
3. Delete the old API key.

Because service accounts can have multiple active API keys, you can create a new API key without having to remove the old key. This short time period of overlap enables applications to continue running until they can be updated to the new API key.

<Note title="Important">
  To immediately block access to a service account, changing the associated ACLs and RBAC role bindings is quicker and more effective than API key rotation or deletion.
</Note>


# Use API Keys to Authenticate to StreamNative Cloud
Source: https://docs.streamnative.io/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview



<Note title="Note">
  You need to upgrade the Pulsar cluster to the following minimum versions (`2.9.5.2`, `2.10.4.4`, or `3.0.0.3`) in order to access this feature. If you want to turn this feature on, please [submit a ticket](https://support.streamnative.io/hc/en-us/requests/new) to the support team.
</Note>

StreamNative Cloud API keys are used to control access to StreamNative Cloud components and resources.

API keys are JSON Web Tokens (JWTs) issued and managed in StreamNative Cloud. They allow you to create long-lived tokens and revoke them. These API keys are associated with the service accounts and the organization from which you create them. For details about using user and service accounts and their ownership of API keys, see [Ownership of API keys](#ownership-of-api-keys).

Use the [API keys](#cluster-api-keys) to control access to StreamNative Cloud clusters. Each API key is scoped for an entire organization.

## API Key v1 vs API Key v2

API Key v2 introduces organization-level authentication to simplify credential management and expand API key usage beyond Pulsar clusters to StreamNative Cloud APIs.

### Comparison

| Feature                                 | API Key v1                              | API Key v2                                         |
| --------------------------------------- | --------------------------------------- | -------------------------------------------------- |
| Scope                                   | Instance-level                          | Organization-level                                 |
| Key management                          | Separate API keys required per instance | Single API key can be used across the organization |
| Authorization model                     | Instance-scoped permissions             | Role-based access control (RBAC)                   |
| Pulsar cluster authentication           | Supported                               | Supported                                          |
| StreamNative Cloud API authentication   | Not supported (OAuth required)          | Supported                                          |
| CLI and Terraform support for Cloud API | Not supported                           | Planned in upcoming release                        |

### Key improvements in API Key v2

* **Organization-level scope** – Manage authentication at the organization level instead of per instance
* **Simplified access management** – No need to create or manage instance-specific API keys
* **RBAC-based authorization** – Fine-grained permissions without requiring multiple keys
* **Expanded authentication coverage** – API keys can be used for both Pulsar clusters and StreamNative Cloud APIs
* **Future extensibility** – Enables upcoming platform security and automation capabilities

> **Important**
> API Key v2 keys cannot be used with Pulsar clusters that have not been upgraded to support API Key v2.

> **Migration considerations**
>
> * Existing API Key v1 credentials will always continue to work even after the migration.
> * Organizations must be upgraded to API Key v2 before new v2 keys can be used.
> * Existing clusters may require upgrades to fully support API Key v2 authentication.

> **Note**
> Support for API key authentication to StreamNative Cloud APIs via **snctl** and the **Terraform provider** will be available in a future release.

To create and manage API keys in StreamNative Cloud, you can use the following tools:

* [StreamNative Cloud Console](https://console.streamnative.cloud)

For recommendations on using API keys, see [Best Practices for Using API Keys in StreamNative Cloud](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-best-practices).

## Ownership of API keys

Each API key is associated with a specific \[service account]\[id:service-accounts].

* A best practice is to create separate service accounts associated with an API keys for each applications or use case.

* Using API keys associated with a user account is currently not supported yet.

* API keys are immutable and cannot be modified. If you need to change the permissions associated with an API key, you must delete the key and create a new one.

* Access control lists (ACLs) and role bindings are associated with **principals**, not with API keys. For details, see [Authorization and ACLs](/cloud/security/access/access-control-lists/authorization-and-acls).

* Restric access to an application that uses an API key associated with a service account:
  * For resource API keys, you can use [access control lists (ACLs)](/cloud/security/access/access-control-lists/authorization-and-acls).

<Warning title="Warning">
  When you delete a service account, all associated API keys will also be deleted. Any client applications using a deleted API key will lose access, which many cause an outage for your streaming application. Always confirm that none of the API keys owned by an account are in active use before deleting a service account.
</Warning>

## API keys

API Keys are used to control access to specific StreamNative Cloud clusters. Each Cluster API key is valid for one specific Organization. You can view the API keys of your organization by going to the Service Accounts page, and clicking on a Service Account's name, or by using `snctl get apikey -O your-org-name`.

<Note title="Important">
  API keys propagate quickly after creation, usually within a few minutes. If you try to use an API key before propagation completes, authentication failures occur. Depending on workloads, you might need to wait a few minutes more and try again.
</Note>

### Prerequisites

Ensure your Pulsar cluster has been upgraded to the following minimum versions:

* 2.9.5.2
* 2.10.4.4
* 3.0.0.3

To use snctl for managing API keys, you will need snctl version 0.16.0 or later.

### Create an API key

<span />

<Tabs>
  <Tab title="StreamNative Console">
    To create an API key for a specific service account in StreamNative Cloud Console, follow these steps.

    1. On the left navigation pane, click **Service Accounts**.

    2. On the Service Accounts page, select a specific service account.

    <Note title="Tip">
      You can also click **New** on the top-right corner and select **Create API Key** to create the API key for any service account.
    </Note>

    3. On the API Key page of this service account, click **New API Key**.

    4. Specify the required attributes for the API key.
       | Field                  | Description                                                                                                        |
       | ---------------------- | ------------------------------------------------------------------------------------------------------------------ |
       | Name                   | The name of the API key.                                                                                           |
       | Expiration date        | By default, each API key expires in 30 days. If you want it to be long-lived, you can select `No expiration date`. |
       | Service Account        | The service account you want to associate the API key with. The one selected in step 2 is auto-populated.          |
       | Description (optional) | The descriptive text to introduce the API key.                                                                     |

    5. Click **Confirm**.

    6. Click **Copy and close** to copy the generated API key and save it in a safe location for future use.

    <Warning title="Warning">
      Make sure the generated API key is securely saved since you won't be able to see it again after closing this window.
    </Warning>
  </Tab>

  <Tab title="snctl">
    To create an API key using snctl, use the `snctl create apikey` command:

    ```bash theme={null}
    snctl create apikey APIKEY_NAME --service-account-name SERVICE_ACCOUNT_NAME --expiration-time EXPIRE_TIME --description DESCRIPTION
    ```

    \| Flags | Description |
    \|--description | The description of API Key |
    \|--expiration-time | The expiration time of an API Key, the value is time in days to expire to with --expiration-time (or minutes, hours, days e.g.: 10m, 3h, 2d) or a time value with format like '2006-01-02T15:04:05Z'or 0 means never expire. |
    \|--service-account-name | The service account name of API key. |

    ## Examples

    To create an API key that never expires, do the following:

    ```bash theme={null}
    snctl create apikey test-apikey --service-account-name test-api-key --expiration-time 0 --description 'this is test' -O sndev
    ```

    To create an API key that expires in 30 days, do the following:

    ```bash theme={null}
    snctl create apikey test-apikey2 --service-account-name test-api-key --expiration-time 30d --description 'this is test' -O sndev
    ```

    To create an API key that expires in at a specific time, use the following:

    ```bash theme={null}
    snctl create apikey my-api-key --service-account-name my-service-account --expiration-time "2025-02-08T15:38:40Z" --description 'API Key for development team' -O my-organization
    ```
  </Tab>

  <Tab title="terraform">
    To create an API key using the StreamNative Terraform module, refer to the [Terraform documentation](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/resources/apikey).
  </Tab>
</Tabs>

## Using API keys to connect to your cluster

After creating an API key in StreamNative Cloud Console, you can use it to authenticate Kafka, Pulsar, and MQTT clients.

### Kafka Clients

<span />

You can use API keys with [SASL/PLAIN authentication](https://kafka.apache.org/documentation/#security_sasl_plain) to connect Kafka clients to StreamNative Cloud clusters.

To configure SASL authentication on the clients:

1. Configure the JAAS configuration property for each client in `producer.properties` or `consumer.properties`. The login module describes how the clients like producers and consumers can connect to the cluster. Below is an example configuration for a client for the **PLAIN** mechanism:

   ```yaml theme={null}
   sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
   username="<tenant>/<namespace>" \
   password="token:<API KEY>";
   ```

   Use `<tenant>/<namespace>` as `username` and `<API KEY>` in `password`.

2. Configure the following properties in `producer.properties` or `consumer.properties`:
   ```yaml theme={null}
   security.protocol=SASL_SSL
   sasl.mechanism=PLAIN
   ```

For configuring Kafka clients in different languages to use API keys, you can find examples on ["Build Applications > Kafka Clients"](/cloud/build/kafka-clients/kafka-on-cloud#kafka-clients).

### Pulsar Clients

<span />

You can use API keys with [JWT authentication](https://pulsar.apache.org/docs/security-jwt/) to connect Pulsar clients to StreamNative Cloud clusters.

For configuring Pulsar clients in different languages to use API keys, you can find examples on ["Build Applications > Pulsar Clients"](/cloud/build/pulsar-clients/qs-connect).

For more examples of configuring Pulsar clients using API keys, see [JWT Authentication](https://pulsar.apache.org/docs/security-jwt/#configure-jwt-authentication-in-pulsar-clients) in Pulsar documentation.

### MQTT Clients

You can use API keys as `password` to connect MQTT clients to StreamNative Cloud clusters.

An example is shown below, where `<username>` can be any string and `password` is the API key.

```java theme={null}
Mqtt5SimpleAuth simpleAuth = Mqtt5SimpleAuth.builder().username("<username>")
                .password("<API KEY>")
                .build();
```

## Deleting and Revoking API Keys

<Tabs>
  <Tab title="StreamNative Console">
    To delete or revoke an API key for a specific service account in StreamNative Cloud Console, follow these steps.

    1. On the left navigation pane, click **Service Accounts**.

    2. On the Service Accounts page, select a specific service account.

    3. On the API Key page of this service account, click the Ellipsis (...) icon in the row of the API key that you want to revoke/delete, and select **Revoke** or **Delete**.

    4. Type the API key’s name to confirm and then click **Revoke** or **Confirm**.
  </Tab>

  <Tab title="snctl">
    To revoke an API key for a specific service account using snctl, use `snctl revoke apikey`.

    ```bash theme={null}
    snctl revoke apikey name-of-apikey -O your-org-name
    ```

    To delete an API key using snctl, use this `snctl delete apikey`:

    ```bash theme={null}
    snctl delete apikey test-apikey-name -O your-org-name
    ```
  </Tab>

  <Tab title="terraform">
    To delete or revoke an API key using the StreamNative Terraform module, refer to the [Terraform documentation](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/resources/apikey).
  </Tab>
</Tabs>

<Note title="What's expected after a revocation?">
  * Revoked API keys will be removed from their service accounts in 15 days.
  * If an API key gets revoked when being used by Kafka clients, the connection will stay alive. For Pulsar clients, the connection will be terminated in 1 minute.
</Note>

<Note title="What's expected after a deletion?">
  If an API key is deleted, it will simultaneously be revoked.
</Note>

## Cloud API keys

StreamNative Cloud doesn't support using API keys to access StreamNative Cloud API. Instead, you can use [OAuth2](/cloud/security/authentication/service-accounts/use-oauth/access-cloud-api-oauth) to connect to the StreamNative Cloud API.


# Access Cloud API using OAuth 2.0
Source: https://docs.streamnative.io/cloud/security/authentication/service-accounts/use-oauth/access-cloud-api-oauth



Use the following information to configure your applications to use the OAuth2 authentication mechanism for connecting to StreamNative Cloud API.

## Configure StreamNative CLI

### Use StreamNative CLI as a user account

You can follow the [instructions](/cloud/security/access/resource-hierarchy/organizations#sign-in-to-an-organization) in [Organizations in StreamNative Cloud](/cloud/security/access/resource-hierarchy/organizations) to sign in to an organization as a user account to use StreamNative CLI.

### Use StreamNative CLI as a service account

In order to use StreamNative CLI as a service account, you need to download a credentials file for the service account you want to use. Please follow the [instructions](/cloud/security/authentication/service-accounts/use-oauth/configure-pulsar-clients-with-oauth-20#credentials-file) in [Access Cloud Clusters](/cloud/security/authentication/service-accounts/use-oauth/configure-pulsar-clients-with-oauth-20) to download the credentials file. We assume that the download credentials file is saved in `/path/to/credentials.json`.

Once you have the credentials file, you can activate the service account by running the following command:

```bash theme={null}
snctl auth activate-service-account --key-file /path/to/credentials.json
```

If the service account is successfully activated, you will see a similar message below in your terminal running the command.

```bash theme={null}
Logged in as sa@my_org.auth.streamnative.cloud.
Welcome to StreamNative Cloud!
```

You can also run `snctl auth whoami` to check which service account that `snctl` is using. It will return the princial name of this service account. Example message is shown below:

```bash theme={null}
sa@my_org.auth.streamnative.cloud
```


# Access Cloud Clusters using Kafka Clients with OAuth 2.0
Source: https://docs.streamnative.io/cloud/security/authentication/service-accounts/use-oauth/configure-kafka-clients-with-oauth-20



Use the following information to configure your Kafka clients to use the OAuth2 authentication mechanism for connecting to StreamNative Cloud clusters.

<Note title="Note">
  The OAuth2 authentication mechanism is currently only validated for Java clients with `io.streamnative.pulsar.handlers.kop.security.oauth.OauthLoginCallbackHandler` callback handler.

  [OIDC Federation](/cloud/security/authentication/oidc-identity-providers/oidc-federation-overview) is not fully supported for Kafka clients yet. The support is under development.
</Note>

## Prerequisites

* [Apache Kafka client](https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients): 3.2.1 or later
* Include the following dependencies in your `pom.xml` file:

  ```xml theme={null}
  <dependency>
    <groupId>io.streamnative.pulsar.handlers</groupId>
    <artifactId>oauth-client</artifactId>
    <version>3.1.0.1</version>
  </dependency>
  ```

## Service URLs

In order to connect to a StreamNative Cloud cluster, you need to get its service URLs.

<Tabs>
  <Tab title="Console">
    You can get a cluster's service URLs by following the steps below:

    1. [Navigate to the **Cluster Workspace** page](/cloud/get-started/cloud-console#switch-a-cluster).

    2. Navigate to the **Details** tab, and in the **Access Points** section, you can find all the available service URLs of this cluster. Click **Copy** at the end of the row of the service URL to copy the URL.

    * `Kafka Service URL (TCP)`: The URL of Kafka service.
    * `Kafka Schema Registry URL (HTTPS)`: The URL of Kafka schema registry service.
  </Tab>
</Tabs>

## JAAS configuration options

Before configuring your Kafka clients to use OAuth 2.0 for connecting to StreamNative Cloud clusters, you need to prepare a JAAS configuration for your clients by following the steps below.

Several configuration options are available for the callback handler. Sensitive configuration options and SASL extensions are included in the JAAS configuration file (`sasl.jaas.config`) while the others are top-level configurations.

<Tabs>
  <Tab title="StreamNative OAuth2">
    | JAAS Configuration Option | Description                                                                                                                                                                                                                                      |
    | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
    | `oauth.issuer.url`        | The URL of the authentication provider which allows the Kafka client to obtain an access token. Currently, StreamNative Cloud only support Auth0 as the identity provider. So the value here should be `https://auth.streamnative.cloud`.        |
    | `oauth.audience`          | The OAuth 2.0 resource server identifier for a Pulsar cluster. In StreamNative Cloud, a Pulsar cluster is identified by a Uniform Resource Name (URN), which is in the following format `urn:sn:pulsar:${your_orgnization_id}:${instance_name}`. |
    | `oauth.credentials.url`   | The URL to the JSON credentials file. It supports the following pattern formats: <br /> <li> `file:///path/to/file` </li><li> `data:application/json;base64,<base64-encoded value>` </li>                                                        |
  </Tab>
</Tabs>

## Configure Kafka Clients

The section describes how to configure Kafka clients to use OAuth 2.0 for connecting to StreamNative Cloud clusters. In this doc, we use `urn:sn:pulsar:my_org:my_instance` as the instance for an example.

* For **StreamNative OAuth2**, `audience` is required.

<Tabs>
  <Tab title="Java">
    ```Java theme={null}
    import io.streamnative.pulsar.handlers.kop.security.oauth.OauthLoginCallbackHandler;

    // replace these configs with your cluster
    String serverUrl = "YOUR-KAFKA-SERVICE-URL";
    String keyPath = "YOUR-KEY-FILE-ABSOLUTE-PATH";
    String audience = "YOUR-AUDIENCE-STRING";

    final Properties props = new Properties();
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, serverUrl);
    ...
    props.setProperty("sasl.login.callback.handler.class", OauthLoginCallbackHandler.class.getName());
    props.setProperty("security.protocol", "SASL_SSL");
    props.setProperty("sasl.mechanism", "OAUTHBEARER");
    final String jaasTemplate = "org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required"
        + " oauth.issuer.url=\"%s\""
        + " oauth.credentials.url=\"%s\""
        + " oauth.audience=\"%s\";";
    props.setProperty("sasl.jaas.config", String.format(jaasTemplate,
        "https://auth.streamnative.cloud/",
        "file://" + keyPath,
        audience
    ));
    ```
  </Tab>
</Tabs>

## Examples

* See [Connect to your cluster using the Kafka Java client](/cloud/build/kafka-clients/quick-starts/cloud-connect-kafka-java) for how to connect your Kafka clients to StreamNative Cloud clusters using OAuth 2.0.


# Access Cloud Clusters using Pulsar Clients with OAuth 2.0
Source: https://docs.streamnative.io/cloud/security/authentication/service-accounts/use-oauth/configure-pulsar-clients-with-oauth-20



Use the following information to configure your Pulsar clients to use the OAuth2 authentication mechanism for connecting to StreamNative Cloud clusters.

## Service URLs

In order to connect to a StreamNative Cloud cluster, you need to get its service URLs.

<Tabs>
  <Tab title="Console">
    You can get a cluster's service URLs by following the steps below:

    1. [Navigate to the **Cluster Workspace** page](/cloud/get-started/cloud-console#switch-a-cluster).

    2. Navigate to the **Details** tab, and in the **Access Points** section, you can find all the available service URLs of this cluster. Click **Copy** at the end of the row of the service URL to copy the URL.

    * `HTTP Service URL (TLS)`: The URL of Pulsar Admin HTTP service.
    * `Broker Service URL (TLS)`: The URL of Pulsar broker service.
  </Tab>
</Tabs>

## OAuth 2.0 Credentials

Before configuring your Pulsar clients to use OAuth 2.0 for connecting to StreamNative Cloud clusters, you need to prepare an OAuth 2.0 credential file for your clients by following the steps below.

### StreamNative OAuth2

If you are using StreamNative's built-in OAuth2 service, you can download the OAuth2 credential file for the service account that you want to use for your clients by following these steps:

<Tabs>
  <Tab title="Console">
    1. Navigate to the **Accounts & Accesses** page.

    2. On the left navigation pane, click **Service Accounts**.

    3. In the row of the service account you want to use, click `...`.

    4. In the dropdown menu, click **Download OAuth2 Key** to download the OAuth2 credential file to your local directory.

    The downloaded credentials file contains the service account credentials used for client authentication. The following is an example of the credentials file for a service account `sa` of organization `my_org`. Both `client_id` and `client_secret` are required while the other fields are optional. Since the credentials file contains `client_secret`, please make sure the credentials file is stored in a safe place.

    ```JSON theme={null}
    {
      "type":"sn_service_account",
      "client_id":"PZWBM2tMCVDFQ1lQInIaYgG4k1OSqwIO",
      "client_secret":"EZr...Kgv",
      "client_email":"sa@my_org.auth.streamnative.cloud",
      "issuer_url":"https://auth.streamnative.cloud"
    }
    ```
  </Tab>
</Tabs>

### OIDC Federation

If you are using an OIDC-compliant identity provider via [OIDC Federation](/cloud/security/authentication/oidc-identity-providers/oidc-federation-overview), you can prepare the credentials information to configure your Pulsar clients to use OAuth 2.0 for connecting to StreamNative Cloud clusters. Create a credentials file containing the `client_id` and `client_secret` of your service account:

```JSON theme={null}
{
  "client_id":"PZWBM2tMCVDFQ1lQInIaYgG4k1OSqwIO",
  "client_secret":"EZr...Kgv"
}
```

## Configure Pulsar Applications

Once you have the credentials file, you can follow the steps below to configure your Pulsar applications to use OAuth2 authentication.

First of all, since Pulsar clients support different authentication plugins, you need to configure the Pulsar clients to use OAuth2 authentication plugin. For example, you can configure the Pulsar Java clients to use `org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2`.

Secondly, you need to prepare the authentication parameters for OAuth2 authentication. The following table outlines the parameters required for configuring OAuth2 authentication.

| Parameter        | Description                                                                                                                                                                                                                                      | Example                                       | Required or not                                                     |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------- | ------------------------------------------------------------------- |
| `type`           | OAuth 2.0 authentication type. Currently, Pulsar clients only support the `client_credentials` authentication type.                                                                                                                              | `client_credentials` (default)                | Optional                                                            |
| `issuerUrl`      | The URL of the authentication provider which allows the Pulsar client to obtain an access token. Currently, StreamNative Cloud only support Auth0 as the identity provider. So the value here should be `https://auth.streamnative.cloud`.       | `https://auth.streamnative.cloud`             | Required                                                            |
| `credentialsUrl` | The URL to the JSON credentials file. It supports the following pattern formats: <br /> <li> `file:///path/to/file` </li><li> `data:application/json;base64,<base64-encoded value>` </li>                                                        | `file:///path/to/my_service_account_key.json` | Required                                                            |
| `audience`       | The OAuth 2.0 resource server identifier for a Pulsar cluster. In StreamNative Cloud, a Pulsar cluster is identified by a Uniform Resource Name (URN), which is in the following format `urn:sn:pulsar:${your_orgnization_id}:${instance_name}`. | `urn:sn:pulsar:my_org:my_instance`            | Required for `StreamNative OAuth2` but optional for OIDC Federation |
| `scope`          | The scope of an access request. For more information, see [access token scope](https://datatracker.ietf.org/doc/html/rfc6749#section-3.3)                                                                                                        | api://pulsar-cluster-1/.default               | Optional                                                            |

### Configure Pulsar Clients

This section describes how to configure Pulsar clients to connect to a StreamNative Cloud cluster `urn:sn:pulsar:my_org:my_instance` using OAuth2. Please notes:

* For **StreamNative OAuth2**, `audience` is required.
* For **OIDC Federation**, based on your identity provider, you may need to specify the `scope` and/or `audience` parameters accordingly.

<Tabs>
  <Tab title="Java">
    ```Java theme={null}
    import org.apache.pulsar.client.impl.auth.oauth2.AuthenticationFactoryOAuth2;

    URL issuerUrl = new URL("https://auth.streamnative.cloud");
    URL credentialsUrl = new URL("file:///path/to/credentials.json");
    String audience = "urn:sn:pulsar:my_org:my_instance";

    PulsarClient client = PulsarClient.builder()
        .serviceUrl("<PULSAR_BROKER_URL>")
        .authentication(
            AuthenticationFactoryOAuth2.clientCredentials(issuerUrl, credentialsUrl, audience))
        .build();
    ```
  </Tab>

  <Tab title="Python">
    ```Python theme={null}
    from pulsar import Client, AuthenticationOauth2

    params = '''
    {
        "issuer_url": "https://auth.streamnative.cloud",
        "private_key": "/path/to/credentials.json",
        "audience": "urn:sn:pulsar:my_org:my_instance"
    }
    '''

    client = Client("<PULSAR_BROKER_URL>", authentication=AuthenticationOauth2(params))
    ```
  </Tab>

  <Tab title="C++">
    ```CPP theme={null}
    #include <pulsar/Client.h>

    pulsar::ClientConfiguration config;
    std::string params = R"({
        "issuer_url": "https://auth.streamnative.cloud",
        "private_key": "/path/to/credentials.json",
        "audience": "urn:sn:pulsar:my_org:my_instance"})";

    config.setAuth(pulsar::AuthOauth2::create(params));

    pulsar::Client client("<PULSAR_BROKER_URL>", config);
    ```
  </Tab>

  <Tab title="Node.js">
    ```javascript theme={null}
    const Pulsar = require('pulsar-client')
    const issuer_url = 'https://auth.streamnative.cloud/'
    const private_key = '/path/to/credentials.json'
    const audience = 'urn:sn:pulsar:my_org:my_instance'
    const service_url = '<PULSAR_BROKER_URL>'

    ;(async () => {
      const params = {
        issuer_url: issuer_url,
        private_key: private_key,
        audience: audience,
      }

      const auth = new Pulsar.AuthenticationOauth2(params)

      const client = new Pulsar.Client({
        serviceUrl: service_url,
        authentication: auth,
        operationTimeoutSeconds: 30,
      })
      await client.close()
    })()
    ```
  </Tab>

  <Tab title="Go">
    ```Go theme={null}
    oauth := pulsar.NewAuthenticationOAuth2(map[string]string{
            "type":       "client_credentials",
            "issuerUrl":  "https://auth.streamnative.cloud/",
            "audience":   "urn:sn:pulsar:my_org:my_instance",
            "privateKey": "/path/to/credentials.json",
        })
    client, err := pulsar.NewClient(pulsar.ClientOptions{
            URL:              "<PULSAR_BROKER_URL>",
            Authentication:   oauth,
    })
    ```
  </Tab>

  <Tab title="Rust">
    ```Rust theme={null}
    let addr = "<PULSAR_BROKER_URL>".to_string();
    let mut builder = Pulsar::builder(addr, TokioExecutor);
    builder = builder.with_auth_provider(OAuth2Authentication::client_credentials(OAuth2Params {
        issuer_url: "https://auth.streamnative.cloud/".to_string(),
        credentials_url: "file:///path/to/credentials.json".to_string(),
        audience: "urn:sn:pulsar:my_org:my_instance".to_string(),
        scope: None,
    }));

    let pulsar: Pulsar<_> = builder.build().await?;
    ```
  </Tab>

  <Tab title="C#">
    ```C# theme={null}
    var fileUri = new Uri("file:///path/to/credentials.json");
    var issuerUrl = new Uri("https://auth.streamnative.cloud/");
    var audience = "urn:sn:pulsar:my_org:my_instance";
    const string serviceUrl = "<PULSAR_BROKER_URL>";

    var client = await new PulsarClientBuilder()
        .ServiceUrl(serviceUrl)
        .Authentication(AuthenticationFactoryOAuth2.ClientCredentials(issuerUrl, audience, fileUri))
        .BuildAsync();
    ```
  </Tab>
</Tabs>

In the example above:

* Replace `<PULSAR_ADMIN_URL>` and/or `<PULSAR_BROKER_URL>` with the right admin and/or broker URL of your Pulsar cluster. You can get the service URLs in the [Cluster Details](#service-urls) page of the StreamNative Cloud Console.

* Replace `urn:sn:pulsar:my_org:my_instance` with the right URN of your Pulsar cluster.

* Replace `file:///path/to/credentials.json` with the right file path of your downloaded credentials file from the StreamNative Cloud Console.

You can find detailed examples in [Build Applications with Pulsar](/cloud/build/pulsar-clients/qs-connect).

### Configure Pulsar Command-line tools

This section describes how to use Pulsar command-line tools to connect to a StreamNative Cloud cluster `urn:sn:pulsar:my_org:my_instance` using OAuth2.

<Tabs>
  <Tab title="pulsar-admin">
    ```bash theme={null}
    bin/pulsar-admin \
        --admin-url <PULSAR_ADMIN_URL> \
        --auth-plugin org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 \
        --auth-params '{"privateKey":"file:///path/to/credentials.json", \
            "issuerUrl":"https://auth.streamnative.cloud", \
            "audience":"urn:sn:pulsar:my_org:my_instance"}' \
        tenants list
    ```
  </Tab>

  <Tab title="pulsar-client">
    ```bash theme={null}
    bin/pulsar-client \
        --url <PULSAR_BROKER_URL> \
        --auth-plugin org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 \
        --auth-params '{"privateKey":"file:///path/to/credentials.json", \
            "issuerUrl":"https://auth.streamnative.cloud", \
            "audience":"urn:sn:pulsar:my_org:my_instance"}' \
        produce test-topic -m "test-message" -n 10
    ```
  </Tab>

  <Tab title="pulsar-perf">
    ```bash theme={null}
    bin/pulsar-perf produce \
        --service-url <PULSAR_BROKER_URL> \
        --auth-plugin org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 \
        --auth-params '{"privateKey":"file:///path/to/credentials.json", \
            "issuerUrl":"https://auth.streamnative.cloud", \
            "audience":"urn:sn:pulsar:my_org:my_instance"}' \
        -r 1000 -s 1024 test-topic
    ```
  </Tab>

  <Tab title="snctl">
    In order to use `snctl` to connect to a StreamNative Cloud cluster using OAuth2, you can follow the steps below:

    #### 1. Make your target StreamNative Cloud cluster as `snctl` current service context

    ```bash theme={null}
    snctl context use --organization <ORGANIZATION> --pulsar-instance <SNCLOUD_INSTANCE> --pulsar-cluster <SNCLOUD_CLUSTER>
    ```

    #### 2. Use `snctl` provided client and admin tools

    You will be able to use `snctl pulsar admin` or `snctl pulsar client` to access to the StreamNative Cloud cluster.

    ```bash theme={null}
    snctl pulsar admin tenants list # list all tenants
    ```
  </Tab>

  <Tab title="pulsarctl">
    In order to use `pulsarctl` to connect to a StreamNative Cloud cluster using OAuth2, you can follow the steps below:

    #### 1. Create a context to point to a StreamNative Cloud cluster

    ```bash theme={null}
    pulsarctl context set <context-name> \
        --admin-service-url <PULSAR_ADMIN_URL> \
        --issuer-endpoint https://auth.streamnative.cloud \
        --audience urn:sn:pulsar:my_org:my_instance \
        --key-file /path/to/credentials.json
    ```

    #### 2. Activate the service account

    You need to activate the service account used in the context before running other Pulsar commands.

    ```bash theme={null}
    pulsarctl oauth2 activate
    ```

    After successfully running the command, you should see a similar output as below:

    ```
    Logged in as sa@my_org.auth.streamnative.cloud.
    Welcome to Pulsar!
    ```
  </Tab>
</Tabs>

In the example above:

* Replace `<PULSAR_ADMIN_URL>` and/or `<PULSAR_BROKER_URL>` with the right admin and/or broker URL of your Pulsar cluster. You can get the service URLs in the [Cluster Details](#service-urls) page of the StreamNative Cloud Console.

* Replace `urn:sn:pulsar:my_org:my_instance` with the right URN of your Pulsar cluster.

* Replace `file:///path/to/credentials.json` with the right file path of your downloaded credentials file from the StreamNative Cloud Console.


# Use OAuth to Authenticate to StreamNative Cloud
Source: https://docs.streamnative.io/cloud/security/authentication/service-accounts/use-oauth/oauth-overview



StreamNative Cloud supports the [OAuth 2.0](https://oauth.net/2/) protocol for authentication and authorization. OAuth is an open-standard protocol that grants access to supported clients using a temporary access token. Supported clients use delegated authorization to access and use StreamNative Cloud resources and data on the behalf of a user or application.

Summary of key features provided by OAuth 2.0 support in StreamNative Cloud:

* Manage application identities and credentials through Auth0.

* Authenticate with StreamNative Cloud resources using short-lived credentails (JSON Web Tokens).

* StreamNative Cloud's OAuth 2.0 service provides OIDC-based tokens for authentication and authorization that are based on the [OAuth 2.0 Authorization Framework \[RFC 6746\]](https://tools.ietf.org/html/rfc6749) and is compliant with [OpenID Connect (OIDC)](https://openid.net/connect/).

* You can configure OAuth using the StreamNative Cloud Console and StreamNative CLI.

## Limitations

OAuth 2.0 for StreamNative Cloud includes the following limitations:

* StreamNative Cloud only uses StreamNative's Auth0 service as identity provider. It doesn't support using your own identity provider yet.

## Access token format

StreamNative Cloud only accepts JSON Web Token (JWT) access tokens, based on an open, industry standard for representing claims to be transferred securely between two parties. A JWT is a string that represents a set of claims as a JSON object in a JSON Web Signature (JWS) or JSON Web Encryption (JWE) structure, enabling the claims to be signed or encrypted.

Each JWT includes a header, body, and signature that is formatted like this:

```
header.body.signature
```

For details about how JWT crednetials, see:

* [JWT (JSON Web Tokens)](https://jwt.io/) website, provided by Auth0
  * [Introduction to JSON Web Tokens](https://jwt.io/introduction)
  * [JWT Debugger](https://jwt.io/#debugger-io)
  * [JWT Handbook](https://auth0.com/resources/ebooks/jwt-handbook): a free ebook
* [JSON Web Token (JWT)\[RFC 7519\]](https://www.rfc-editor.org/rfc/rfc7519)
* [JSON Web Token (JWT) Profile for OAuth 2.0 Access Tokens\[RFC 9068\]](https://datatracker.ietf.org/doc/html/rfc9068)


# User accounts for StreamNative Cloud
Source: https://docs.streamnative.io/cloud/security/authentication/user-accounts



In StreamNative Cloud, users are identified by their email address and authenticated through one of the following: Google login, username and password, or through SSO. Organization administrators can invite users to an organization, and they will receive an email to complete the registration.

## Authentication Methods

StreamNative Cloud provides three authentication methods (username/password, Google, and SSO), as summarized in the following table.

| Authentication method | Description                                                                                          |
| --------------------- | ---------------------------------------------------------------------------------------------------- |
| Username/password     | A user that authenticates using a combination of username and password.                              |
| Google                | A user that authenticates using a user's Google account.                                             |
| SSO                   | A user that authenticates using single sign-on (SSO) with an organization's identity provider (IdP). |

Note that StreamNative Cloud users have the following conditions and limitations:

* Each user account represents one user and allows management of their access to StreamNative Cloud.
* User accounts are organization-level resources. An organization can have only one identity provider (IdP).
* You can sign in to a user account using the StreamNative Cloud Console or StreamNative CLI.
* Principals (users and service accounts) can be granted ACLs. For details, see [Control Access to StreamNative Cloud](/cloud/security/access/access-control-overview).
* You can create and manage users using the StreamNative Cloud Console.
* A user account can be a member of one or more organizations. When a user is a member of multiple organizations, their authentication types are the same across all organizations.
* If your email provider supports creating multiple accounts or aliases by adding a plus sign (`+`) and a tag or word before the `@` sign in an email address, each alias that is used to sign up for StreamNative Cloud will be its own separate account. This doesn't work for organizations enabled SSO login.
* A user account can have multiple authentication methods.

## Users using username/password

### Create a user (initial)

If you don't have a StreamNative Cloud user account, you can create one with a username and password.

To create a user in StreamNative Cloud:

1. Go to the [StreamNative Cloud Console](https://console.streamnative.cloud/?defaultMethod=signup).

2. The **Create Your Account** page appears.

3. To sign up for a new account, type your email and click **Continue** button.

4. Enter a password and click the **Continue** button. A verification link will be sent to your email address.

5. Check your email account for a **Your account with StreamNative Cloud** email.

6. In the email, click "Verify email address". You will be redirected to an **Email Verified** page on StreamNative Cloud. Then click "Back to cloud console for login".

7. The **Log in to StreamNative's cloud console** page appears.

8. Type in your email and password to continue. Follow the instruction to complete the sign-up form to create your first organization.

9. You are signed in to StreamNative Cloud and can begin using the StreamNative Cloud Console.

### Password requirements

Passwords must conform to the following restrictions:

* At least 8 characters
* At least 3 of the following:
  * Lower case letters (a-z)
  * Upper case letters (A-Z)
  * Numbers (0-9)
  * Special characters (e.g. `! @ # $ % ^ & *`)

StreamNative Cloud user accounts use [Auth0](https://auth0.com/) for authentication. For details about password lengths, see [Password Strength in Auth0 Database Connections](https://auth0.com/docs/authenticate/database-connections/password-strength).

### Resetting your password

If you forget your password, reset it by doing the following:

* Go to the Cloud Console. Enter your email address into the login prompt as though you were going to sign in, and click **Continue**. <img alt="Click Continue" />
* On the next screen, you will be prompted to enter your password. Below the password field is a link titled **Forgot password?**. Click it. <img alt="Forgot Password" />
* The next screen, titled **Forgot your password?**, contains a field to enter your email address to receive instructions on resetting your password. Confirm the email address in the field is correct, and click **Continue**.
* Check your inbox for an email from StreamNative Cloud titled "Reset your password." If you do not see it, check your Spam folder.
* Navigate to the link specified in the email, and you will see a page where you can reset your password. Enter in your new password, re-enter it to confirm it, and click **Reset password**. <img alt="New Password" />

## Signing In with Google

Users can create a user account for StreamNative Cloud using Google as their social identity provider (IdP). This simplifies user registration and sign-in and is a convenient alternative to mandatory account creation.

If your organization starts on StreamNative Cloud using the "Continue with Google" option, you can migrate later to use SAML-based single sign-on (SSO).

<Note title="Note">
  You cannot currently disable Google authentication to use username/password authentication.
</Note>

### Use Continue with Google to authenticate

You can sign up for a StreamNative Cloud user account with Google and then you will be able to use **Continue with Google** on every future visit.

To use **Continue with Google**:

1. Go to [StreamNative Cloud Console](https://console.streamnative.cloud/?defaultMethod=signup).

2. Click **Continue with Google**.

3. On the **Choose an account** page, click on your Google account.

4. After you're authenticated with Google, you will be redirected to StreamNative Cloud console. Follow the instruction to complete the sign-up form to create your first organization.

5. You are signed in to StreamNative Cloud and can begin exploring and using the StreamNative Cloud Console.

After registering your Google account with StreamNative Cloud, you can sign in to StreamNative Cloud by going to the StreamNative Cloud Console and clicking **Continue with Google**.

## Single sign-on (SSO) users

User accounts created after enabling single sign-on (SSO) for your organization provide acccess to StreamNative Cloud using an existing SAML-based identity provider (IdP). If you would like to enable SSO for your organization, please reach out to your StreamNative Account Manager.

## Manage user accounts

As an administrator, you can invite users to your organization and delete the current users. This section describes how to invite, view, and delete users in your organization.

You can perform all these operations using the StreamNative Cloud console. Alternatively, you can do them using [StreamNative CLI](/tools/cli/snctl/snctl-overview). Before executing any commands, ensure that `snctl` is correctly configured for the appropriate organization. For guidance on signing in to an organization with `snctl`, refer to the section on [Sign in to an organization](/cloud/security/access/resource-hierarchy/organizations#sign-in-to-an-organization).

## Invite a user

### Invite a user using the StreamNative Cloud Console

You can use the StreamNative Cloud Console to invite a user.

<Tabs>
  <Tab title="Through the Dashboard page">
    1. In the upper-right corner of the StreamNative Console, click your Profile and select **Organizations** to check your created organizations.

    2. Click the name of your organization.

    3. On the **Users** card of the **Dashboard** page, click the **Add** icon and a dialog box appears.

    4. Enter the user's email address and then click **Invite**.

    5. Click the `Access` button to grant roles to the user account. For details on available roles, see [Predefined Role](/cloud/security/access/rbac/manage-rbac-roles)
  </Tab>

  <Tab title="Through the Organization page">
    1. In the upper-right corner of the StreamNative Console, click your Profile and select **Organizations** to check your created organizations.

    2. Click the name of your organization.

    3. On the **Organization** page, click **Invite Teammates** at the end of the target organization.

    4. On the **User** page, click **Invite User** and a dialog box displays.

    5. Enter the user's email address and then click **Invite**.

    6. Click the `Access` button to grant roles to the user account. For details on available roles, see [Predefined Role](/cloud/security/access/rbac/manage-rbac-roles)
  </Tab>

  <Tab title="Through the Users page">
    1. In the upper-right corner of the StreamNative Console, click your Profile and select **Organizations** to check your created organizations.

    2. Click the name of your organization.

    3. In the upper-right corner of the StreamNative Console, click your Profile and select **Users** from the drop-down list

    4. On the **User** page, click **Invite User** and a dialog box displays.

    5. Enter the user's email address and then click **Invite**.

    6. Click the `Access` button to grant roles to the user account. For details on available roles, see [Predefined Role](/cloud/security/access/rbac/manage-rbac-roles)
  </Tab>
</Tabs>

The new user is sent a **Welcome to StreamNative Cloud** message. The new user can click "Get Started" link to sign up or log in to the StreamNative Cloud Console to access the organization that he is invited to.

### Invite a user using `snctl`

<Tabs>
  <Tab title="Command-line">
    To invite a new user to your organization, follow these two steps:

    1. Create a User account
    2. Bind the User account to a Role

    #### Create a User account in an organization

    To create a new User account in an organization, follow the command below. Ensure that `${email}` is replaced with a valid email address where the new user will receive an invitation from StreamNative Cloud. The `${user_account_name}` is a unique identifier for the user account within your organization. It is recommended to use the user's email address as the identifier for simplicity and clarity.

    ```bash theme={null}
    snctl create user ${user_account_name} --email ${email}
    ```

    For example, to invite a user with the email `john.doe@test.local`, you would use:

    ```bash theme={null}
    snctl create user john.doe@test.local --email john.doe@test.local
    ```

    #### Bind the User account to a Role

    After creating a user account, you must bind a role to it to grant permissions. For use all predefined roles, see [Predefined Role](/cloud/security/access/rbac/manage-rbac-roles). The following example shows how to bind the `admin` role.

    To bind a User account to a role, use the following command to create a role binding. Make sure that the `${user_account_name}` is the one you created in the previous step. The `${rolebinding_name}` should be a unique identifier for the role binding within your organization. For ease of reference, it is recommended to use the format `${user_account_name}_${role}` as the identifer.

    ```bash theme={null}
    snctl create rolebinding ${rolebinding_name} --clusterrole admin --user ${user_account_name}
    ```

    For example, to create a role binding for `john.doe@test.local`, use:

    ```bash theme={null}
    snctl create rolebinding john.doe@test.local_admin --clusterrole admin --user john.doe@test.local
    ```

    At this point, the new user is fully prepared to access the invited organization on StreamNative Cloud.

    However, if the new user has not yet registered on StreamNative Cloud, they will **NOT** receive an invitation email if invited through `snctl`. In this case, the new user should follow the [sign-up instructions](#create-a-user-initial) to create a new account. Once signed in, they will be able to locate the organization they were invited to on the 'Organizations' page.

    Conversely, if the new user is already a registered member of StreamNative Cloud, they will immediately find the organization they have been invited to on their 'Organizations' page.
  </Tab>

  <Tab title="YAML">
    To invite a new user to your organization, follow these two steps:

    1. Create a User account
    2. Bind the User account to a Role

    You can achieve this by defining a User account object and a Rolebinding object to bind the User account object to a role, following the YAML template below. Then you can use `snctl apply` to create those objects.

    Please replace the following variables with the right values.

    * User account variables:
      * `${user_account_name}`: a unique identifier for the user account within your organization. It is recommended to use the user's email address as the identifier for simplicity and clarity.
      * `${organization_id}`: the id of the organization that you would like to invite user to.
      * `${email}`: a valid email address where the new user will receive emails from StreamNative Cloud.
      * `${user_first_name}`: the first name of the invited user.
      * `${user_last_name}`: the last name of the invited user.
    * Role binding variables:
      * `${rolebinding_name}`: a unique identifier for the role binding within your organization. For ease of reference, it is recommended to use the format `${user_account_name}_${role}` as the identifer.
      * `${organization_id}`: the id of the organization that you would like to invite user to.
      * `${user_account_name}`: the name of the user account object

    ```yaml theme={null}
    apiVersion: cloud.streamnative.io/v1alpha1
    kind: User
    metadata:
      name: ${user_account_name}
      namespace: ${organization_id}
    spec:
      email: ${email}
      name:
        first: ${user_first_name}
        last: ${user_last_name}
      type: external

    ---
    apiVersion: cloud.streamnative.io/v1alpha1
    kind: RoleBinding
    metadata:
      name: ${rolebinding_name}
      namespace: ${organization_id}
    spec:
      roleRef:
        apiGroup: cloud.streamnative.io
        kind: ClusterRole
        name: admin
      subjects:
        - apiGroup: cloud.streamnative.io
          kind: User
          name: ${user_account_name}
    ```

    For example, to invite a user with the email `john.doe@test.local` to organization `myorg`, you would create a YAML file named `user.yaml`:

    ```YAML theme={null}
    apiVersion: cloud.streamnative.io/v1alpha1
    kind: User
    metadata:
      name: john.doe@test.local
      namespace: myorg
    spec:
      email: john.doe@test.local
      name:
        first: John
        last: Doe
      type: external

    sidebarTitle: User Accounts
    ---
    apiVersion: cloud.streamnative.io/v1alpha1
    kind: RoleBinding
    metadata:
      name: john.doe@test.local_admin
      namespace: myorg
    spec:
      roleRef:
        apiGroup: cloud.streamnative.io
        kind: ClusterRole
        name: admin
      subjects:
        - apiGroup: cloud.streamnative.io
          kind: User
          name: john.doe@test.local
    ```

    Then you can use `snctl apply` to apply the configurations to create those 2 objects on StreamNative Cloud.

    ```bash theme={null}
    snctl apply -f user.yaml
    ```

    You will see following output in your console after executing the `snctl apply` command.

    ```bash theme={null}
    user.cloud.streamnative.io/john.doe@test.local created
    rolebinding.cloud.streamnative.io/john.doe@test.local_admin created
    ```

    At this point, the new user is fully prepared to access the invited organization on StreamNative Cloud.

    However, if the new user has not yet registered on StreamNative Cloud, they will **NOT** receive an invitation email if invited through `snctl`. In this case, the new user should follow the [sign-up instructions](#create-a-user-initial) to create a new account. Once signed in, they will be able to locate the organization they were invited to on the 'Organizations' page.

    Conversely, if the new user is already a registered member of StreamNative Cloud, they will immediately find the organization they have been invited to on their 'Organizations' page.
  </Tab>
</Tabs>

## Delete a user account

### Delete a user using the StreamNative Cloud Console

To delete a user, follow these steps.

1. In the upper-right corner of StreamNative Console, click your Profile and select **Users** from the drop-down list.

2. Click the ellipsis at the end of the row of the user that you want to delete, and then click **Delete**. A dialog box displays.

3. Click **Confirm**.

### Delete a user using `snctl`

<Tabs>
  <Tab title="Command-line">
    To delete a new User account in an organization, follow the command below.

    ```bash theme={null}
    snctl delete user ${user_account_name}
    ```

    For example, to delete the user account `john.doe@test.local`, you would use:

    ```bash theme={null}
    snctl delete user john.doe@test.local
    ```
  </Tab>

  <Tab title="YAML">
    If you create the User account using `snctl apply`, you can also use `snctl delete` to delete the objects created by `snctl apply`.

    For example, to delete the user account and role binding objects created by applying config file `/path/to/config.yaml`, you can use:

    ```bash theme={null}
    snctl delete -f /path/to/config.yaml
    ```
  </Tab>
</Tabs>

## View user accounts

### View user accounts using the StreamNative Cloud Console

In the upper-right corner of StreamNative Console, click your Profile and select **Users** from the drop-down list to view the users for an organization, including the email address and status of the users.

### View user accounts using `snctl`

You can view the list of user accounts by running the following command:

```
snctl get users
```

If you want to view the details of a specific user `${user_account_name}`, you can use:

```bash theme={null}
snctl get users ${user_account_name}
```

For example, to view the user account `john.doe@test.local`, you would use:

```bash theme={null}
snctl get users john.doe@test.local -o yaml
```


# Encrypt and Protect Data on StreamNative Cloud
Source: https://docs.streamnative.io/cloud/security/encrypt-and-protect-data/encryption-overview



This section provides how does StreamNative Cloud encrypt and protect your data. This includes securing data at rest, in transit, and in use, to help ensure that your sensitive information is safeguarded against unauthorized access.

## Encrypt data at rest

StreamNative Cloud automatically encrypts all data at rest across all clusters using the encrypted storage volumes provided by your cloud provider. This ensures your data is protected when stored on disk.

For BYOC Pro clusters, you can also use your own encryption keys through customer-managed encryption keys (BYOK - Bring Your Own Key). This gives you full control over the encryption keys used to protect your data. BYOK must be configured during cluster provisioning and is currently only available for BYOC Pro clusters. To enable BYOK for your BYOC Pro cluster, contact [StreamNative Support](https://support.streamnative.io) before provisioning your cluster.

## End-to-end data encryption

Applications using the Pulsar protocol can leverage Pulsar's [end-to-end encryption (E2EE)](https://pulsar.apache.org/docs/security-encryption/) to encrypt messages on the producer side and decrypt them on the consumer side. This encryption uses public and private key pairs configured by your application to perform the encryption and decryption. Since these operations happen within the application itself, the data remains encrypted while passing through the broker and can only be decrypted by authorized consumers with the correct keys. This ensures your data remains protected even if the broker is compromised, and StreamNative has no access to the encrypted content.

For more information, see [End-to-end encryption](https://pulsar.apache.org/docs/security-encryption/).


# Audit log
Source: https://docs.streamnative.io/cloud/security/monitor-activity/cloud-audit-log



Audit logs track and store authorization activities in Pulsar clusters, tenants, namespaces, and topics. After a Pulsar cluster is up and running within a large team, it's critical to keep an eye on who is touching data and what they're doing with it. Structured audit logs provide an easy way to track user/application access, so you can identify potential anomalies and bad actors.

Structured audit logs enable you to capture audit logs in a set of dedicated Pulsar topics, either on a local or a remote cluster, including:

* low-volume, management-related activities, such as creating or deleting tenants, namespaces or topics
* high-volume activities, such as produce, consume, and acknowledge events

<Note title="Note">
  Because the audit logs are stored in a Pulsar topic, you should configure the cleanup and backlog policy for the namespace to avoid having the Pulsar retention policies inadvertently clean up the audit logs. For more information, see [Configure policies for a namespace](/cloud/manage-data-streams/namespace#configure-policies-for-a-namespace).
</Note>

## Configure audit log

If you did not enable audit log when you set up your cluster, or if you want to change the settings, follow these steps.

1. Log in to [StreamNative Cloud Console](https://console.streamnative.cloud).
2. On the left navigation pane, in the **Admin** area, click **Pulsar Clusters**.
3. Click **Edit Cluster**.
4. If it is not already selected, click **Advanced**.
5. Toggle the **Audit Log** option on.
6. Select the types of audit logs you want to enable. For details about the types of events that you can enable, see [event type](#event-type).
7. Click **Finish** to apply your changes and return to the Clusters page.

## Check the audit log status

To verify that audit log is running, on the left navigation pane of StreamNative Cloud Console, in the **Admin** area, click **Pulsar Clusters**. You can verify that the audit log is running, as shown in the following figure.

<img alt="screenshot of running audit log successfully" />

## Event type

Each audit log includes information about the event, event time, and permission status. The supported audit event types include:

| Category   | Type         | Event type               | Description                                         | Default value |
| ---------- | ------------ | ------------------------ | --------------------------------------------------- | ------------- |
| Management | Cluster      | `CreateCluster`          | Create a Pulsar cluster.                            | true          |
|            | Cluster      | `UpdateCluster`          | Update Pulsar cluster information.                  | true          |
|            | Cluster      | `DeleteCluster`          | Delete Pulsar cluster information.                  | true          |
|            | Tenant       | `CreateTenant`           | Create a Pulsar tenant.                             | true          |
|            | Tenant       | `UpdateTenant`           | Update tenant information.                          | true          |
|            | Tenant       | `DeleteTenant`           | Delete a Pulsar tenant.                             | true          |
|            | Namespace    | `CreateNamespace`        | Create a namespace.                                 | true          |
|            | Namespace    | `DeleteNamespace`        | Delete a namespace.                                 | true          |
|            | Topic        | `CreatePartitionedTopic` | Create a partitioned topic.                         | true          |
|            | Topic        | `UpdatePartitions`       | Update partitions for a partitioned topic.          | true          |
|            | Topic        | `DeletePartitionedTopic` | Delete a partitioned topic.                         | true          |
|            | Subscription | `CreateSubscription`     | Create a subscription.                              | true          |
|            | Subscription | `DeleteSubscription`     | Delete a subscription.                              | true          |
| Describe   | Cluster      | `ListClusters`           | List Pulsar clusters.                               | false         |
|            | Cluster      | `GetCluster`             | Get cluster information.                            | false         |
|            | Tenant       | `ListTenants`            | List Pulsar tenants.                                | false         |
|            | Tenant       | `GetTenant`              | Get tenant information.                             | false         |
|            | Namespace    | `ListNamespaces`         | List Pulsar namespaces.                             | false         |
|            | Namespace    | `GetNamespace`           | Get namespace information.                          | false         |
|            | Topic        | `ListTopics`             | List Pulsar topics.                                 | false         |
|            | Topic        | `ListPartitionedTopics`  | List partitioned Pulsar topics.                     | false         |
|            | Topic        | `GetPartitions`          | Get partitions of a partitioned topic.              | false         |
|            | Subscription | `ListSubscriptions`      | List subscriptions of a topic.                      | false         |
| Produce    | Producer     | `NewProducer`            | Create a producer to produce messages to the topic. | false         |
|            | Producer     | `CloseProducer`          | Close a producer.                                   | false         |
| Consume    | Consumer     | `NewConsumer`            | Create a consumer to subscribe to the topic.        | false         |
|            | Consumer     | `CloseConsumer`          | Close a consumer.                                   | false         |

## Work with audit log

You can use Pulsar clients, Pulsar CLI, Rest API, and sink connectors to process and analyze the audit events stored in the Pulsar topics.

### Consume the audit log topic with Pulsar clients

You can get data from the audit log topic with Pulsar clients. Refer to the [Connect](/cloud/build/pulsar-clients/qs-connect) section to learn how to configure Pulsar clients for StreamNative Cloud cluster and consume data from the audit log topic.

### Consume the audit log topic with Pulsar CLI

The `pulsar-client` is also a helpful tool to get data from the audit log topic.

### Consume the audit log topic with Rest API

StreamNative Cloud supports Rest API, which provides a RESTful interface to a Pulsar cluster. For more information, see [Rest API](/cloud/build/pulsar-clients/connect-restapi) to learn how to leverage the Rest API to consume data from the audit log topic.

### Sink audit log topic with connectors

If you want to integrate audit log data with your other data systems like Google BigQuery, AWS SQS, and AWS Kinesis, use a sink connector. For more information, see [Deploy connectors](/cloud/connect/pulsar-io/deploy-connectors/deploy-connector-index) to learn how to leverage connectors to integrate and synchronize audit log data.


# Work with Secrets
Source: https://docs.streamnative.io/cloud/security/secret



StreamNative Cloud Secrets allow you to store and manage sensitive data such as passwords, tokens, and private keys. A Secret may contain numerous keys. You can create Secrets and refer to them for computing purposes (such as Pulsar connectors and Pulsar Functions).

## Create Secrets

<Note title="Note">
  The per-secret size is up to 1 Mebibyte (MiB).
</Note>

<Tabs>
  <Tab title="StreamNative Cloud Console">
    To create a Secret using the StreamNative Cloud Console, follow these steps.

    1. On the left navigation pane, in the **Admin** area, click **Secrets**.

    2. Click **Create Secret**.

       <img alt="screenshot of creating secrets" />

    3. Configure the Secret.

       * Name: enter the Secret name. The Secret name is unique across an organization. A secret name can contain any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-).|
       * Instance Name : select an Instance
       * Location: select a Pulsar cluster location for the Secret.
       * Key: enter the key for the Secret. Each key must consist of alphanumeric characters, '-', '\_' or '.'. The serialized form of the Secret data is a [base64 encoded string](https://tools.ietf.org/html/rfc4648#section-4), representing the arbitrary (possibly non-string) data value here.
       * Value: enter the value for the Secret. Each value must consist of alphanumeric characters, '-', '\_' or '.'.

    4. Click **Confirm**.
  </Tab>

  <Tab title="REST API">
    To create a Secret using the REST API, follow these steps.

    1. [Create a service account](/cloud/security/authentication/service-accounts/service-accounts).

    2. Get a token of the service account.
       import GetToken from '/snippets/get-token.mdx';

    <GetToken />

    3. [Get names of your organization, instance, and cluster](/cloud/clusters/manage-clusters/cluster#check-cluster-details-through-streamnative-cloud-console).

    4. Create a Secret.

       This example creates a Secret named `secret-test`, substituting the token, instance name, cluster location, and the organization name respectively.

       ```bash theme={null}
       curl -X "POST" 'https://console.streamnative.cloud/cloud-api/apis/cloud.streamnative.io/v1alpha1/namespaces/<your_organization_name>/secrets/' \
       -H 'Authorization: Bearer <YOUR_TOKEN>' \
       -H 'Content-Type: application/json' \
       --data '{
           "instanceName":"your_instance_name",
           "location":"your_cluster_location",
           "data":{"just":"a test"},
           "metadata": {
               "name": "secret-test",
               "namespace": "your_organization_name"
           }
       }'
       ```
  </Tab>
</Tabs>

## Use Secrets

After [creating a Secret](/cloud/security/secret#create-secrets), you can use it when submitting a function or connector.

1. Enable your function/connector to access the Secret.

<Tabs>
  <Tab title="Java">
    ```java theme={null}
    public class ExampleFunction implements Function<String, Void> {
        @Override
        public String process(String input, Context context) {
            String secretValue = context.getSecret("SECRET1"); # access secret value with the name you need; this name will be set during submission
            System.out.println(secretValue) # You should never log or print the secret value in a production environment.
        }
    }
    ```
  </Tab>

  <Tab title="Python">
    ```python theme={null}
    from pulsar import Function

    class GetSecretValueFunction(Function):
        def process(self, input, context):
            secret_value = context.get_secret("SECRET1")
    		     print(secret_value)
    ```
  </Tab>
</Tabs>

2. Submit the function/connector referring to the Secret.

   The following is an example of using the `pulsar-admin` CLI tool.

   ```bash theme={null}
   ./bin/pulsar-admin functions create \
   --jar /pf-examples/pf-examples-jar-with-dependencies.jar \
   --classname io.streamnative.function.SecretFunction \
   --inputs public/default/secret-test \
   --output public/default/test-output \
   --name SecretTest \
   --secrets '{"SECRET1": {"path": "lambda-sink-secret", "key": "awsAccessKey"}}'
   ```

<Note title="Note">
  * The `SECRET1` in the `--secrets` parameter is the name you used in your function or connector code to access the Secret value.
  * The `path` in the `--secrets` parameter is the Secret name you created.
  * The `key` in the `--secrets` parameter is the key you used in the Secret.
</Note>

## Delete Secrets

To delete a Secret, follow these steps.

1. On the left navigation pane, in the **Admin** area, click **Secrets**.

2. Click **Delete Secret**. A dialog box displays, asking *Are you sure you want to delete?*

3. Enter the Secret name and then click **Delete Secret**.


# Manage Security in StreamNative Cloud
Source: https://docs.streamnative.io/cloud/security/security-overview



StreamNative Cloud provides comprehensive security features to help you manage access control, data protection, and network security, ensuring that your data is secure and your resources are protected. This section describes the key areas of managing security on StreamNative Cloud, including authentication, access control, data protection, and activity monitoring.

## Manage authentication

Authentication ensures that only authorized users and applications can access your StreamNative Cloud resources. This includes managing user accounts, service accounts, and integrating with external identity providers for Single Sign-On (SSO), OAuth/OIDC, mTLS, and more.

See [Authenticate to StreamNative Cloud](/cloud/security/authentication/authentication-overview) for more details.

## Control access to StreamNative Cloud resources

Use authorization mechanisms, such as role-based access control (RBAC), to control access to StreamNative Cloud resources. This ensures only authorized entities can access specific resources and perform certain actions.

See [Control access to StreamNative Cloud](/cloud/security/access/access-control-overview) for more details.

## Encrypt and protect data

Data encryption protects your data at rest and in transit, to help you comply with data protection regulations and ensure the security of your sensitive information. StreamNative Cloud supports various encryption methods, including TLS encryption for data in transit, encryption at rest for data stored, and end-to-end encryption to protect your most sensitive data.

See [Encrypt and protect data on StreamNative Cloud](/cloud/security/encrypt-and-protect-data/encryption-overview) for more details.

## Monitor and audit activities

Monitoring activities helps tracking and auditing access and actions performed on your StreamNative Cloud resources. This helps you detect and respond to potential security incidents and ensure compliance with security policies.

See [Audit log](/cloud/security/monitor-activity/cloud-audit-log) for more details.

## Manage secrets

Secret management enables you to securely store and manage sensitive data such as passwords, tokens, and private keys. You can create secrets and reference them in your computing workloads, including connectors, functions, and Flink jobs, without exposing the sensitive values directly in your configurations.

See [Work with Secrets](/cloud/security/secret) for more details.


# Delete apikeys
Source: https://docs.streamnative.io/api-references/cloudapi/apikey/cloudstreamnativeio_v1alpha1/delete-apiscloudstreamnativeiov1alpha1namespaces-apikeys

delete /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/apikeys/{name}
delete an APIKey



# List apikeyss
Source: https://docs.streamnative.io/api-references/cloudapi/apikey/cloudstreamnativeio_v1alpha1/get-apiscloudstreamnativeiov1alpha1namespaces-apikeys

get /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/apikeys
list or watch objects of kind APIKey



# Get apikeys
Source: https://docs.streamnative.io/api-references/cloudapi/apikey/cloudstreamnativeio_v1alpha1/get-apiscloudstreamnativeiov1alpha1namespaces-apikeys-1

get /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/apikeys/{name}
read the specified APIKey



# Patch apikeys
Source: https://docs.streamnative.io/api-references/cloudapi/apikey/cloudstreamnativeio_v1alpha1/patch-apiscloudstreamnativeiov1alpha1namespaces-apikeys

patch /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/apikeys/{name}
partially update the specified APIKey



# Update apikeys
Source: https://docs.streamnative.io/api-references/cloudapi/apikey/cloudstreamnativeio_v1alpha1/post-apiscloudstreamnativeiov1alpha1namespaces-apikeys

post /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/apikeys/{name}
replace the specified APIKey



# Cloud API
Source: https://docs.streamnative.io/api-references/cloudapi/cloud-api



The StreamNative cloud API is an extension of the [kubernetes API](https://kubernetes.io/docs/concepts/overview/kubernetes-api/) that you can use to manage resources on the StreamNative cloud.

You can use the StreamNative cloud API to manage serivce accounts, users, Pulsar Instances, Pulsar Clusters, and ApiKeys programatically.

## Prerequisites

* You need to have an [organization](https://docs.streamnative.io/docs/organizations)
* You need a [service account](https://docs.streamnative.io/docs/service-accounts) with admin permission

## Authentication

StreamNative cloud supports OAuth2 authentication to managing cloud resources. You can use any language http OAuth2 client to connect to the SN cloud.

Run the OAuth2 client credential flow using the curl command to get an access token to access the cloud. Download the service account with admin permissions from the console, then run the following command.

```bash theme={null}
curl --request POST -v \
   --url https://auth.streamnative.cloud/oauth/token \
   --header 'content-type: application/json' \
   --data '{"client_id": "<your-client-id>", "client_secret": "<your-client-secret>", "audience": "https://api.streamnative.cloud", "grant_type": "client_credentials"}'
```

Use go http module run OAuth2 client credential flow to get access token

```go theme={null}
package main

import (
	"bytes"
	"io"
	"net/http"
)

func main() {

	url := "https://auth.streamnative.cloud/oauth/token"
	var jsonStr = []byte(`{
	"client_id":"<your-client-id>",
	"client_secret":"<your-client-secret>",
	"audience":"https://api.streamnative.cloud",
	"grant_type":"client_credentials"
}`)
	req, err := http.NewRequest("POST", url, bytes.NewBuffer(jsonStr))
	req.Header.Set("Content-Type", "application/json")

	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()

	_, _ = io.ReadAll(resp.Body)
}

```

Response:

```jsx theme={null}
{
  "access_token": "<your-access-token>",
  "expires_in": 84086,
  "token_type": "Bearer"
}
```

## Management ServiceAccount

### List ServiceAccounts

```go theme={null}
 	host := "https://api.streamnative.cloud"
	pathPrefix := "/apis/cloud.streamnative.io/v1alpha1/namespaces/"
	organization := "<your-organization-name>"
	token := "<your-access-token>"
	req, err := http.NewRequest(
		"GET",
		fmt.Sprintf("%s%s%s%s", host, pathPrefix, organization, "/serviceaccounts"),
		nil)
	if err != nil {
		panic(err)
	}
	req.Header.Set("Authorization", "bearer "+token)
	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()
	res, _ := io.ReadAll(resp.Body)
	fmt.Println(string(res))
```

Response:

```go theme={null}
{
  "kind": "ServiceAccountList",
  "apiVersion": "cloud.streamnative.io/v1alpha1",
  "metadata": {
    "resourceVersion": "3110513"
  },
  "items": [
  ]
}

```

### Create ServiceAccount

Request Body:

```json theme={null}
{
  "apiVersion": "cloud.streamnative.io/v1alpha1",
  "kind": "ServiceAccount",
  "metadata": {
    "name": "<your-service-account-name>",
    "namespace": "<your-organization-name>",
    "annotations": {
      "annotations.cloud.streamnative.io/service-account-role": "admin"
    }
  }
}
```

If you set `annotations.cloud.streamnative.io/service-account-role` to `""`, it will create a service account with non-admin permissions

Example:

```go theme={null}
	var jsonStr = []byte(`{
	  "apiVersion": "cloud.streamnative.io/v1alpha1",
	  "kind": "ServiceAccount",
	  "metadata": {
		"name": "<your-service-account-name>",
		"namespace": "<your-organization-name>",
		"annotations": {
		  "annotations.cloud.streamnative.io/service-account-role": "admin"
		}
	  }
	}
`)

 	host := "https://api.streamnative.cloud"
	pathPrefix := "/apis/cloud.streamnative.io/v1alpha1/namespaces/"
	organization := "<your-organization-name>"
	token := "<your-access-token>"
	req, err := http.NewRequest(
		"POST", fmt.Sprintf("%s%s%s%s", host, pathPrefix, organization, "/serviceaccounts"),
		bytes.NewBuffer(jsonStr))
	req.Header.Set("Content-Type", "application/json")
	req.Header.Set("Authorization", "bearer "+token)

	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()

	res, _ := io.ReadAll(resp.Body)
	fmt.Println(string(res))
```

Response:

```jsx theme={null}
{
  "kind": "ServiceAccount",
  "apiVersion": "cloud.streamnative.io/v1alpha1",
  "metadata": {
    "name": "<your-service-account-name>",
    "namespace": "<your-organization-name>",
    "creationTimestamp": "2024-05-11T09:12:14Z",
    "annotations": {
      "annotations.cloud.streamnative.io/service-account-role": "admin"
    },
  },
  "spec": {},
  "status": {}
}
```

### Get Service Account

```go theme={null}
	host := "https://api.streamnative.cloud"
	pathPrefix := "/apis/cloud.streamnative.io/v1alpha1/namespaces/"
	organization := "<your-organization-name>"
	token := "<your-access-token>"
	req, err := http.NewRequest(
		"GET",
		fmt.Sprintf("%s%s%s%s%s", host, pathPrefix, organization, "/serviceaccounts/", "<your-serviceaccount-name>"),
		nil)
	if err != nil {
		panic(err)
	}
	req.Header.Set("Authorization", "bearer "+token)
	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()
	res, _ := io.ReadAll(resp.Body)
	fmt.Println(string(res))
```

Response:

```jsx theme={null}
{
  "kind": "ServiceAccount",
  "apiVersion": "cloud.streamnative.io/v1alpha1",
  "metadata": {
    "name": "<your-service-account-name>",
    "namespace": "<your-organization-name>",
    "creationTimestamp": "2024-05-11T09:12:14Z",
    "annotations": {
      "annotations.cloud.streamnative.io/service-account-role": "admin",
    },
  },
  "spec": {},
  "status": {
    "privateKeyType": "TYPE_SN_CREDENTIALS_FILE",
    "privateKeyData": "<base64-encode-data>",
    "conditions": [
      {
        "type": "Ready",
        "status": "True",
        "reason": "Provisioned",
        "lastTransitionTime": "2024-05-11T09:12:15Z"
      }
    ]
  }
}
```

conditions status is ready means that the service account was created successfully

privateKeyData is the base64-encoded credentials to the service account

### Delete Service Account

```go theme={null}
	host := "https://api.streamnative.cloud"
	pathPrefix := "/apis/cloud.streamnative.io/v1alpha1/namespaces/"
	organization := "<your-organization-name>"
	token := "<your-access-token>"
	req, err := http.NewRequest(
		"DELETE",
		fmt.Sprintf("%s%s%s%s%s", host, pathPrefix, organization, "/serviceaccounts/", "test-sa"),
		nil)
	if err != nil {
		panic(err)
	}
	req.Header.Set("Authorization", "bearer "+token)
	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()
	res, _ := io.ReadAll(resp.Body)
	fmt.Println(string(res))
```

Response:

```go theme={null}
{
  "kind": "ServiceAccount",
  "apiVersion": "cloud.streamnative.io/v1alpha1",
  "metadata": {
     "name": "<your-service-account-name>",
    "namespace": "<your-organization-name>",
    "uid": "ef7e24e8-536b-48b4-aeeb-7333531f12b2",
    "resourceVersion": "3110507",
    "generation": 2,
    "creationTimestamp": "2024-05-11T10:47:15Z",
    "deletionTimestamp": "2024-05-11T10:51:12Z",
    "deletionGracePeriodSeconds": 0,
    "annotations": {
      "annotations.cloud.streamnative.io/service-account-role": "admin",
      "cloud.streamnative.io/allowed-origin-set": "true"
    },
    "finalizers": [
      "serviceaccount.finalizers.cloud.streamnative.io"
    ],
  },
	......
}

```

`deletionTimestamp` means this resource will be deleted

## Management PulsarInstance

### List PulsarInstances

See list service accounts, just replace `serviceaccounts` with `pulsarinstances`

### Create PulsarInstance

```go theme={null}
var jsonStr = []byte(`{
		  "apiVersion": "cloud.streamnative.io/v1alpha1",
		  "kind": "PulsarInstance",
		  "metadata": {
			"name": "<your-instance-name>",
			"namespace": "<your-organization-name>"
		  },
		  "spec": {
				"availabilityMode": "zonal",
				"poolRef": {
				  "name": "shared-aws",
	 			  "namespace": "streamnative"
				}
		 }
}`)

	host := "https://api.streamnative.cloud"
	pathPrefix := "/apis/cloud.streamnative.io/v1alpha1/namespaces/"
	organization := "<your-organization-name>"
	token := "<your-access-token>"
	req, err := http.NewRequest(
		"POST", fmt.Sprintf("%s%s%s%s", host, pathPrefix, organization, "/pulsarinstances"),
		bytes.NewBuffer(jsonStr))
	req.Header.Set("Content-Type", "application/json")
	req.Header.Set("Authorization", "bearer "+token)

	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()

	res, _ := io.ReadAll(resp.Body)
	fmt.Println(string(res))
```

`availabilityMode` and `poolRef` Please take a look at the [pulsarinsatnce](https://docs.streamnative.io/docs/snctl-tutorials#create-an-instance) section of snctl

Response:

```go theme={null}
{
  "kind": "PulsarInstance",
  "apiVersion": "cloud.streamnative.io/v1alpha1",
  "metadata": {
		"name": "<your-instance-name>",
		"namespace": "<your-organization-name>",
    "uid": "d9036adf-ba1f-4f7b-a199-de4159bc7e47",
    "resourceVersion": "3110630",
    "generation": 1,
    "creationTimestamp": "2024-05-11T11:55:19Z",
    "annotations": {
      "annotations.cloud.streamnative.io/istio-enabled": "true"
    },
  },
  "spec": {
    "poolRef": {
      "namespace": "streamnative",
      "name": "shared-aws"
    },
    "availabilityMode": "zonal",
    "type": "standard",
    "auth": {
      "apikey": {}
    }
  },
  "status": {
    "auth": null
  }
}

```

### Get PulsarInstance

See get service account, just replace `serviceaccounts` with `pulsarinstances`

### Delete PulsarInstance

See delete service account, just replace `serviceaccounts` with `pulsarinstances`

## Management User

### List User

See list service accounts, just replace `serviceaccounts` with `users`

### Get User

See get service account, just replace `serviceaccounts` with `users`

### Create User

```go theme={null}
var jsonStr = []byte(`{
		  "apiVersion": "cloud.streamnative.io/v1alpha1",
		  "kind": "User",
		  "metadata": {
			"name": "<your-user-name>",
			"namespace": "<your-organization-name>"
		  },
		  "spec": {
				"email": "<user-email>"
		 }
}`)

	host := "https://api.streamnative.cloud"
	pathPrefix := "/apis/cloud.streamnative.io/v1alpha1/namespaces/"
	organization := "<your-organization-name>"
	token := "<your-access-token>"
	req, err := http.NewRequest(
		"POST", fmt.Sprintf("%s%s%s%s", host, pathPrefix, organization, "/users"),
		bytes.NewBuffer(jsonStr))
	req.Header.Set("Content-Type", "application/json")
	req.Header.Set("Authorization", "bearer "+token)

	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()

	res, _ := io.ReadAll(resp.Body)
	fmt.Println(string(res))
```

Response:

```go theme={null}
{
  "kind": "User",
  "apiVersion": "cloud.streamnative.io/v1alpha1",
  "metadata": {
		"name": "<your-instance-name>",
		"namespace": "<your-organization-name>",
    "uid": "ff7fbc4f-4037-4d6d-afb4-4304d57b33e8",
    "resourceVersion": "3110657",
    "generation": 1,
    "creationTimestamp": "2024-05-11T12:06:02Z",
  },
  "spec": {
    "email": "<user-email>"
  },
  "status": {}
}

```

### Delete User

See delete service account, just replace `serviceaccounts` with `users`

## Management PulsarCluster

### List PulsarCluster

See list service accounts, just replace `serviceaccounts` with `pulsarclusters`

### Create PulsarCluster

```go theme={null}
var jsonStr = []byte(`{
		  "apiVersion": "cloud.streamnative.io/v1alpha1",
		  "kind": "PulsarCluster",
		  "metadata": {
			"name": "<your-user-name>",
			"namespace": "<your-organization-name>"
		  },
		  "spec": {
				"instanceName": "test-instance",
				"location": "us-east-2",
				"bookkeeper": {
				  "replicas": 3,
					"resources": {
						"cpu":"400m",
						"memory":"1.6Gi"
					}
				},
				"broker": {
					"replicas": 2,
					"resources": {
						"cpu":"400m",
						"memory":"1.6Gi"
					}
				},
	 			"config": {
					"custom": {
						"backlogQuotaDefaultLimitBytes": "1000000000"
					},
					"functionEnabled": true
				}
		 }
}`)

	host := "https://api.streamnative.cloud"
	pathPrefix := "/apis/cloud.streamnative.io/v1alpha1/namespaces/"
	organization := "<your-organization-name>"
	token := "<your-access-token>"
	req, err := http.NewRequest(
		"POST", fmt.Sprintf("%s%s%s%s", host, pathPrefix, organization, "/pulsarclusters"),
		bytes.NewBuffer(jsonStr))
	req.Header.Set("Content-Type", "application/json")
	req.Header.Set("Authorization", "bearer "+token)

	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()

	res, _ := io.ReadAll(resp.Body)
	fmt.Println(string(res))
```

`Spec` : Please take a look at the [pulsarcluster](https://docs.streamnative.io/docs/snctl-tutorials#work-with-clusters) section of snctl

Response:

```go theme={null}
{
  "kind": "PulsarCluster",
  "apiVersion": "cloud.streamnative.io/v1alpha1",
  "metadata": {
		"name": "<your-user-name>",
		"namespace": "<your-organization-name>"
    "uid": "d05258aa-ed83-4d61-ac61-d3f040b48204",
    "resourceVersion": "3110692",
    "generation": 1,
    "creationTimestamp": "2024-05-11T12:21:41Z",
  },
  "spec": {
    "instanceName": "test-instance",
    "location": "us-east-2",
    "poolMemberRef": {
      "namespace": "streamnative",
      "name": "<poolmember-name>"
    },
    "serviceEndpoints": [
      {
        "dnsName": "<cluster-dns-name>",
        "type": "service"
      }
    ],
    "broker": {
      "replicas": 2,
      "image": "docker-proxy.streamnative.io/streamnative/pulsar-cloud:3.2.2.5",
      "resources": {
        "cpu": "400m",
        "memory": "1717986918400m",
        "heapPercentage": 0,
        "directPercentage": 0
      }
    },
    "bookkeeper": {
      "replicas": 3,
      "image": "docker-proxy.streamnative.io/streamnative/pulsar-cloud:3.2.2.5",
      "resources": {
        "cpu": "400m",
        "memory": "1717986918400m",
        "heapPercentage": 0,
        "directPercentage": 0,
        "ledgerDisk": "16Gi",
        "journalDisk": "3Gi"
      }
    },
    "config": {
      "functionEnabled": true,
      "protocols": {
        "kafka": {}
      },
      "custom": {
        "backlogQuotaDefaultLimitBytes": "1000000000",
        "managedLedgerOffloadAutoTriggerSizeThresholdBytes": "0"
      }
    },
    "releaseChannel": "rapid"
  },
  "status": {}
}

```

### Get PulsarClustser

See get service account, just replace `serviceaccounts` with `pulsarclusters`

### Update PulsarCluster

```go theme={null}
var jsonStr = []byte(`{
			  "spec": {
					"broker": {
							"replicas": 3
					}
			 }
	}`)

	host := "https://api.streamnative.cloud"
	pathPrefix := "/apis/cloud.streamnative.io/v1alpha1/namespaces/"
	organization := "<your-organization-name>"
	token := "<your-access-token>"
	req, err := http.NewRequest(
		"PATCH", fmt.Sprintf("%s%s%s%s%s", host, pathPrefix, organization, "/pulsarclusters/", "<your-pulsar-cluster-name>"),
		bytes.NewBuffer(jsonStr))
	req.Header.Set("Content-Type", "application/merge-patch+json")
	req.Header.Set("Authorization", "bearer "+token)

	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()

	res, _ := io.ReadAll(resp.Body)
	fmt.Println(string(res))
```

### Delete PulsarCluster

See delete service account, just replace `serviceaccounts` with `pulsarclusters`

## Management Apikey

### List Apikey

See list service accounts, just replace `serviceaccounts` with `apikeys`

### Create Apikey

Generate public key and private key, please refer to [this module](https://github.com/streamnative/terraform-provider-streamnative/blob/main/cloud/util/crypto.go) to generate it. Please save the pem and private key in a secure place, it will be used to encrypt and decrypt your apikey token in the future

```go theme={null}
 	privateKey, err := GenerateEncryptionKey()
	if err != nil {
		panic(err)
	}
	encryptionKey, err := ExportPublicKey(privateKey)
	if err != nil {
		panic(err)
	}
	fmt.Println(encryptionKey.PEM)
	fmt.Println(base64.StdEncoding.EncodeToString([]byte(ExportPrivateKey(privateKey))))
```

```go theme={null}
 	pem := `-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAoONi2FfPcpddBKZCxpXD\nBfEylQeoaOWQhFEnnMnAE2teeDgz92XUVj5xQj/ldqkIakje7RfPiGOpCYwoJxwf\nrahB+FpQTrF28POASEEJctNDUkp5DI/Dxx64zUP8gF+z8djsOB3YhSlrR6zZm4jS\nuseF5l97njzJeiZhd40of0LbvfwuEklaQwuze0KuY4Kq6xACiOH91bI8mxh6qDDP\nyUUPr5nbUyMxgyNPke0oqUDexY3RtzB7wYJksPEPFPnyRQ5QMqMoTy2ydCoULiZW\n6FvolFwjgPWgcIEkDjYo+x0x6CHAvILiNSzN47Ooq6vVnILuJGbZkhsHT6EU68NJ\nNwIDAQAB\n-----END PUBLIC KEY-----`
	var jsonStr = `{
			  "apiVersion": "cloud.streamnative.io/v1alpha1",
			  "kind": "APIKey",
			  "metadata": {
		    "name": "<your-apikey-name>",
		    "namespace": "<your-organization-name>"
			  },
			  "spec": {
				"encryptionKey": {
					"pem": "%s"
				},
				"instanceName": "test-instance",
				"serviceAccountName": "admin",
				"expirationTime": "2024-06-10T13:08:42Z"
			 }
	}`
	host := "https://api.streamnative.cloud"
	pathPrefix := "/apis/cloud.streamnative.io/v1alpha1/namespaces/"
	organization := "<your-organization-name>"
	token := "<your-access-token>"
	req, err := http.NewRequest(
		"POST", fmt.Sprintf("%s%s%s%s", host, pathPrefix, organization, "/apikeys"),
		bytes.NewBuffer([]byte(fmt.Sprintf(jsonStr, pem))))
	req.Header.Set("Content-Type", "application/json")
	req.Header.Set("Authorization", "bearer "+token)

	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()
	res, _ := io.ReadAll(resp.Body)
	fmt.Println(string(res))
```

Response:

```go theme={null}
{
  "kind": "APIKey",
  "apiVersion": "cloud.streamnative.io/v1alpha1",
  "metadata": {
    "name": "<your-apikey-name>",
    "namespace": "<your-organization-name>",
    "uid": "502f340f-7316-4541-ba5f-37cca9aa0162",
    "resourceVersion": "3110875",
    "generation": 1,
    "creationTimestamp": "2024-05-11T14:04:56Z",
    "ownerReferences": [
      {
        "apiVersion": "cloud.streamnative.io/v1alpha1",
        "kind": "ServiceAccount",
        "name": "admin",
        "uid": "9867d06e-d81b-4901-aee7-25bd2dffcf63",
        "controller": true,
        "blockOwnerDeletion": true
      }
    ]
  },
  "spec": {
    "instanceName": "test-instance",
    "serviceAccountName": "admin",
    "expirationTime": "2024-06-10T13:08:42Z",
    "encryptionKey": {
      "pem": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAoONi2FfPcpddBKZCxpXD\nBfEylQeoaOWQhFEnnMnAE2teeDgz92XUVj5xQj/ldqkIakje7RfPiGOpCYwoJxwf\nrahB+FpQTrF28POASEEJctNDUkp5DI/Dxx64zUP8gF+z8djsOB3YhSlrR6zZm4jS\nuseF5l97njzJeiZhd40of0LbvfwuEklaQwuze0KuY4Kq6xACiOH91bI8mxh6qDDP\nyUUPr5nbUyMxgyNPke0oqUDexY3RtzB7wYJksPEPFPnyRQ5QMqMoTy2ydCoULiZW\n6FvolFwjgPWgcIEkDjYo+x0x6CHAvILiNSzN47Ooq6vVnILuJGbZkhsHT6EU68NJ\nNwIDAQAB\n-----END PUBLIC KEY-----"
    }
  },
  "status": {}
}
```

### Get Apikey

See get service account, just replace `serviceaccounts` with `apikeys`

If you save the privateKey from above, you can, you can convert `status.encryptedToken.jwe` to original token, `ImportPrivateKey` from this [module](https://github.com/streamnative/terraform-provider-streamnative/blob/main/cloud/util/crypto.go). If you lost the privateKey, you will no longer be able to get the original token

```go theme={null}
	privateKey := "<your-private-key>"
	key, err := base64.StdEncoding.DecodeString(privateKey)
	if err != nil {
		panic(err)
	}
	pk, err := ImportPrivateKey(string(key))
	if err != nil {
		panic(err)
	}
	token, err := jwe.Decrypt([]byte("<your-api-key-status-encryptedToken-jwe>"), jwe.WithKey(jwa.RSA_OAEP, pk))
	if err != nil {
		panic(err)
	}
	fmt.Println(string(token))
```

### Revoke ApiKey

```go theme={null}
var jsonStr = []byte(`{
			  "spec": {
					"revoke": true
			 }
	}`)

	host := "https://api.streamnative.cloud"
	pathPrefix := "/apis/cloud.streamnative.io/v1alpha1/namespaces/"
	organization := "<your-organization-name>"
	token := "<your-access-token>"
	req, err := http.NewRequest(
		"PATCH", fmt.Sprintf("%s%s%s%s%s", host, pathPrefix, organization, "/apikeys/", "<your-apikey-name>"),
		bytes.NewBuffer(jsonStr))
	req.Header.Set("Content-Type", "application/merge-patch+json")
	req.Header.Set("Authorization", "bearer "+token)

	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()

	res, _ := io.ReadAll(resp.Body)
	fmt.Println(string(res))
```

Response:

```json theme={null}

	{
		"type": "Revoked",
		"status": "True",
		"lastTransitionTime": "2024-05-11T14:04:58Z",
		"reason": "API Key has been revoked",
		"message": ""
	},

```

### Delete Apikey

See get service account, just replace `serviceaccounts` with `apikeys`


# Delete cloudconnections
Source: https://docs.streamnative.io/api-references/cloudapi/cloudconnection/cloudstreamnativeio_v1alpha1/delete-apiscloudstreamnativeiov1alpha1namespaces-cloudconnections

delete /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/cloudconnections/{name}
delete a CloudConnection



# List cloudconnectionss
Source: https://docs.streamnative.io/api-references/cloudapi/cloudconnection/cloudstreamnativeio_v1alpha1/get-apiscloudstreamnativeiov1alpha1namespaces-cloudconnections

get /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/cloudconnections
list or watch objects of kind CloudConnection



# Get cloudconnections
Source: https://docs.streamnative.io/api-references/cloudapi/cloudconnection/cloudstreamnativeio_v1alpha1/get-apiscloudstreamnativeiov1alpha1namespaces-cloudconnections-1

get /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/cloudconnections/{name}
read the specified CloudConnection



# Patch cloudconnections
Source: https://docs.streamnative.io/api-references/cloudapi/cloudconnection/cloudstreamnativeio_v1alpha1/patch-apiscloudstreamnativeiov1alpha1namespaces-cloudconnections

patch /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/cloudconnections/{name}
partially update the specified CloudConnection



# Update cloudconnections
Source: https://docs.streamnative.io/api-references/cloudapi/cloudconnection/cloudstreamnativeio_v1alpha1/post-apiscloudstreamnativeiov1alpha1namespaces-cloudconnections

post /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/cloudconnections/{name}
replace the specified CloudConnection



# Delete cloudenvironments
Source: https://docs.streamnative.io/api-references/cloudapi/cloudenvironment/cloudstreamnativeio_v1alpha1/delete-apiscloudstreamnativeiov1alpha1namespaces-cloudenvironments

delete /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/cloudenvironments/{name}
delete a CloudEnvironment



# List cloudenvironmentss
Source: https://docs.streamnative.io/api-references/cloudapi/cloudenvironment/cloudstreamnativeio_v1alpha1/get-apiscloudstreamnativeiov1alpha1namespaces-cloudenvironments

get /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/cloudenvironments
list or watch objects of kind CloudEnvironment



# Get cloudenvironments
Source: https://docs.streamnative.io/api-references/cloudapi/cloudenvironment/cloudstreamnativeio_v1alpha1/get-apiscloudstreamnativeiov1alpha1namespaces-cloudenvironments-1

get /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/cloudenvironments/{name}
read the specified CloudEnvironment



# Patch cloudenvironments
Source: https://docs.streamnative.io/api-references/cloudapi/cloudenvironment/cloudstreamnativeio_v1alpha1/patch-apiscloudstreamnativeiov1alpha1namespaces-cloudenvironments

patch /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/cloudenvironments/{name}
partially update the specified CloudEnvironment



# Update cloudenvironments
Source: https://docs.streamnative.io/api-references/cloudapi/cloudenvironment/cloudstreamnativeio_v1alpha1/post-apiscloudstreamnativeiov1alpha1namespaces-cloudenvironments

post /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/cloudenvironments/{name}
replace the specified CloudEnvironment



# Delete pulsarclusters
Source: https://docs.streamnative.io/api-references/cloudapi/pulsarcluster/cloudstreamnativeio_v1alpha1/delete-apiscloudstreamnativeiov1alpha1namespaces-pulsarclusters

delete /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/pulsarclusters/{name}
delete a PulsarCluster



# List pulsarclusterss
Source: https://docs.streamnative.io/api-references/cloudapi/pulsarcluster/cloudstreamnativeio_v1alpha1/get-apiscloudstreamnativeiov1alpha1namespaces-pulsarclusters

get /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/pulsarclusters
list or watch objects of kind PulsarCluster



# Get pulsarclusters
Source: https://docs.streamnative.io/api-references/cloudapi/pulsarcluster/cloudstreamnativeio_v1alpha1/get-apiscloudstreamnativeiov1alpha1namespaces-pulsarclusters-1

get /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/pulsarclusters/{name}
read the specified PulsarCluster



# Patch pulsarclusters
Source: https://docs.streamnative.io/api-references/cloudapi/pulsarcluster/cloudstreamnativeio_v1alpha1/patch-apiscloudstreamnativeiov1alpha1namespaces-pulsarclusters

patch /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/pulsarclusters/{name}
partially update the specified PulsarCluster



# Update pulsarclusters
Source: https://docs.streamnative.io/api-references/cloudapi/pulsarcluster/cloudstreamnativeio_v1alpha1/post-apiscloudstreamnativeiov1alpha1namespaces-pulsarclusters

post /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/pulsarclusters/{name}
replace the specified PulsarCluster



# Delete pulsargateways
Source: https://docs.streamnative.io/api-references/cloudapi/pulsargateway/cloudstreamnativeio_v1alpha1/delete-apiscloudstreamnativeiov1alpha1namespaces-pulsargateways

delete /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/pulsargateways/{name}
delete a PulsarGateway



# List pulsargatewayss
Source: https://docs.streamnative.io/api-references/cloudapi/pulsargateway/cloudstreamnativeio_v1alpha1/get-apiscloudstreamnativeiov1alpha1namespaces-pulsargateways

get /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/pulsargateways
list or watch objects of kind PulsarGateway



# Get pulsargateways
Source: https://docs.streamnative.io/api-references/cloudapi/pulsargateway/cloudstreamnativeio_v1alpha1/get-apiscloudstreamnativeiov1alpha1namespaces-pulsargateways-1

get /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/pulsargateways/{name}
read the specified PulsarGateway



# Patch pulsargateways
Source: https://docs.streamnative.io/api-references/cloudapi/pulsargateway/cloudstreamnativeio_v1alpha1/patch-apiscloudstreamnativeiov1alpha1namespaces-pulsargateways

patch /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/pulsargateways/{name}
partially update the specified PulsarGateway



# Update pulsargateways
Source: https://docs.streamnative.io/api-references/cloudapi/pulsargateway/cloudstreamnativeio_v1alpha1/post-apiscloudstreamnativeiov1alpha1namespaces-pulsargateways

post /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/pulsargateways/{name}
replace the specified PulsarGateway



# Delete pulsarinstances
Source: https://docs.streamnative.io/api-references/cloudapi/pulsarinstance/cloudstreamnativeio_v1alpha1/delete-apiscloudstreamnativeiov1alpha1namespaces-pulsarinstances

delete /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/pulsarinstances/{name}
delete a PulsarInstance



# List pulsarinstancess
Source: https://docs.streamnative.io/api-references/cloudapi/pulsarinstance/cloudstreamnativeio_v1alpha1/get-apiscloudstreamnativeiov1alpha1namespaces-pulsarinstances

get /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/pulsarinstances
list or watch objects of kind PulsarInstance



# Get pulsarinstances
Source: https://docs.streamnative.io/api-references/cloudapi/pulsarinstance/cloudstreamnativeio_v1alpha1/get-apiscloudstreamnativeiov1alpha1namespaces-pulsarinstances-1

get /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/pulsarinstances/{name}
read the specified PulsarInstance



# Update pulsarinstances
Source: https://docs.streamnative.io/api-references/cloudapi/pulsarinstance/cloudstreamnativeio_v1alpha1/post-apiscloudstreamnativeiov1alpha1namespaces-pulsarinstances

post /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/pulsarinstances/{name}
replace the specified PulsarInstance



# Delete secrets
Source: https://docs.streamnative.io/api-references/cloudapi/secrets/cloudstreamnativeio_v1alpha1/delete-apiscloudstreamnativeiov1alpha1namespaces-secrets

delete /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/secrets/{name}
delete a Secret



# List secretss
Source: https://docs.streamnative.io/api-references/cloudapi/secrets/cloudstreamnativeio_v1alpha1/get-apiscloudstreamnativeiov1alpha1namespaces-secrets

get /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/secrets
list or watch objects of kind Secret



# Get secrets
Source: https://docs.streamnative.io/api-references/cloudapi/secrets/cloudstreamnativeio_v1alpha1/get-apiscloudstreamnativeiov1alpha1namespaces-secrets-1

get /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/secrets/{name}
read the specified Secret



# Patch secrets
Source: https://docs.streamnative.io/api-references/cloudapi/secrets/cloudstreamnativeio_v1alpha1/patch-apiscloudstreamnativeiov1alpha1namespaces-secrets

patch /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/secrets/{name}
partially update the specified Secret



# Update secrets
Source: https://docs.streamnative.io/api-references/cloudapi/secrets/cloudstreamnativeio_v1alpha1/post-apiscloudstreamnativeiov1alpha1namespaces-secrets

post /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/secrets/{name}
replace the specified Secret



# Delete serviceaccountbindings
Source: https://docs.streamnative.io/api-references/cloudapi/serviceaccount/cloudstreamnativeio_v1alpha1/delete-apiscloudstreamnativeiov1alpha1namespaces-serviceaccountbindings

delete /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/serviceaccountbindings/{name}
delete a ServiceAccountBinding



# Delete serviceaccounts
Source: https://docs.streamnative.io/api-references/cloudapi/serviceaccount/cloudstreamnativeio_v1alpha1/delete-apiscloudstreamnativeiov1alpha1namespaces-serviceaccounts

delete /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/serviceaccounts/{name}
delete a ServiceAccount



# List serviceaccountbindingss
Source: https://docs.streamnative.io/api-references/cloudapi/serviceaccount/cloudstreamnativeio_v1alpha1/get-apiscloudstreamnativeiov1alpha1namespaces-serviceaccountbindings

get /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/serviceaccountbindings
list or watch objects of kind ServiceAccountBinding



# Get serviceaccountbindings
Source: https://docs.streamnative.io/api-references/cloudapi/serviceaccount/cloudstreamnativeio_v1alpha1/get-apiscloudstreamnativeiov1alpha1namespaces-serviceaccountbindings-1

get /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/serviceaccountbindings/{name}
read the specified ServiceAccountBinding



# List serviceaccountss
Source: https://docs.streamnative.io/api-references/cloudapi/serviceaccount/cloudstreamnativeio_v1alpha1/get-apiscloudstreamnativeiov1alpha1namespaces-serviceaccounts

get /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/serviceaccounts
list or watch objects of kind ServiceAccount



# Get serviceaccounts
Source: https://docs.streamnative.io/api-references/cloudapi/serviceaccount/cloudstreamnativeio_v1alpha1/get-apiscloudstreamnativeiov1alpha1namespaces-serviceaccounts-1

get /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/serviceaccounts/{name}
read the specified ServiceAccount



# Patch serviceaccountbindings
Source: https://docs.streamnative.io/api-references/cloudapi/serviceaccount/cloudstreamnativeio_v1alpha1/patch-apiscloudstreamnativeiov1alpha1namespaces-serviceaccountbindings

patch /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/serviceaccountbindings/{name}
partially update the specified ServiceAccountBinding



# Patch serviceaccounts
Source: https://docs.streamnative.io/api-references/cloudapi/serviceaccount/cloudstreamnativeio_v1alpha1/patch-apiscloudstreamnativeiov1alpha1namespaces-serviceaccounts

patch /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/serviceaccounts/{name}
partially update the specified ServiceAccount



# Update serviceaccountbindings
Source: https://docs.streamnative.io/api-references/cloudapi/serviceaccount/cloudstreamnativeio_v1alpha1/post-apiscloudstreamnativeiov1alpha1namespaces-serviceaccountbindings

post /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/serviceaccountbindings/{name}
replace the specified ServiceAccountBinding



# Update serviceaccounts
Source: https://docs.streamnative.io/api-references/cloudapi/serviceaccount/cloudstreamnativeio_v1alpha1/post-apiscloudstreamnativeiov1alpha1namespaces-serviceaccounts

post /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/serviceaccounts/{name}
replace the specified ServiceAccount



# Delete users
Source: https://docs.streamnative.io/api-references/cloudapi/user/cloudstreamnativeio_v1alpha1/delete-apiscloudstreamnativeiov1alpha1namespaces-users

delete /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/users/{name}
delete an User



# List userss
Source: https://docs.streamnative.io/api-references/cloudapi/user/cloudstreamnativeio_v1alpha1/get-apiscloudstreamnativeiov1alpha1namespaces-users

get /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/users
list or watch objects of kind User



# Get users
Source: https://docs.streamnative.io/api-references/cloudapi/user/cloudstreamnativeio_v1alpha1/get-apiscloudstreamnativeiov1alpha1namespaces-users-1

get /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/users/{name}
read the specified User



# Update users
Source: https://docs.streamnative.io/api-references/cloudapi/user/cloudstreamnativeio_v1alpha1/post-apiscloudstreamnativeiov1alpha1namespaces-users

post /apis/cloud.streamnative.io/v1alpha1/namespaces/{namespace}/users/{name}
replace the specified User



# Kafka Rest API Quickstart
Source: https://docs.streamnative.io/api-references/kafka-rest-api/kafka-rest-api



The StreamNative Kafka REST API provides a comprehensive HTTP-based interface for interacting with your Kafka clusters. Apache Kafka itself does not come with a native REST API. This feature allows you to manage critical resources and produce/consume messages without needing native Kafka clients or complex library setups.

StreamNative's Kafka REST API implementation provides:

* **HTTP-based Kafka Operations**: Manage topics, produce/consume messages, and administer your cluster using any language and standard tools like curl, without needing native Kafka client libraries.
* **Full Protocol Compatibility**: Faithfully supports the Kafka protocol, ensuring seamless integration and expected behavior for all standard operations.
* **Built-in Security**: Integrated with StreamNative's authentication and authorization systems
* **Multi-tenancy Support**: Native support for StreamNative's tenant/namespace isolation model

## Prerequisites

Before using the Kafka REST API, ensure you have:

* A StreamNative Cloud account with an active Kafka-enabled cluster
* Appropriate permissions to create service accounts and manage Kafka resources
* Basic familiarity with REST APIs and HTTP tools like `curl`

### Step 1: Create a service account

<Note title="Note">
  Currently, you can't edit a service account. If you need a service account to have Super Admin access, make sure to enable it when creating the service account. By default, service accounts do not have Super Admin enabled.
</Note>

To create a service account, follow these steps.

1. On the left navigation pane, click **Service Accounts**.

2. Click **Create Service Account**.

3. (Optional) Select **Super Admin** to grant the service account with Super admin access to a namespace or tenant.

4. Enter a name for the service account, and then click **Confirm**.

### Step 2: Create an API key for your service account

<Note title="Note">
  Before using an API key, verify that the service account is authorized to access the resources, such as tenants, namespaces, and topics.
</Note>

You can follow the instructions to [create an API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview#using-api-keys-to-connect-to-your-cluster) for the service account you choose to use.

### Step 3: Grant service account permissions

<Note title="Note">
  If you use a Super Admin service account, you can skip this step because a Super Admin service account already has the required permissions.
</Note>

You can grant permissions to the service account using RBAC. For a description of the available permissions, see the [predefined roles](/cloud/security/access/rbac/manage-rbac-roles#quick-reference). Granting permissions via the UI will be supported soon.

### Step 4: Get the HTTP Service URL of your StreamNative cluster

To get the service URL(s) of a StreamNative cluster, follow these steps.

<Tabs>
  <Tab title="StreamNative Console">
    1. Navigate to the **Cluster Dashboard** page by [switching to the cluster workspace](/cloud/get-started/cloud-console#switch-a-cluster).

    2. On the **Cluster Dashboard** page, click **Details** tab.

    3. You will see the available service URLs in the **Access Points** area.

    4. You can click **Copy** at the end of the row of the service URL that you want to use.
  </Tab>
</Tabs>

<Note>
  For the Kafka REST API, you need to use the **HTTP Service URL (TLS)** endpoint.
</Note>

### Step 5: Get topic list

The following example shows how to list topics using the Kafka REST API. For a complete list of all available API, see the full [Kafka REST API Reference](/api-references/kafka-rest-api).

```shell theme={null}
curl --location --request GET 'https://<your-http-service-url-tls>/rest-kafka/admin/v1/topics' \
--header 'Authorization: Bearer <your-token>'
```

<Note title="Security Best Practices">
  **Never hardcode authentication tokens in your applications.** Instead:

  * Store tokens in secure environment variables or secret management systems
  * Implement token rotation policies to regularly refresh credentials
  * Use service accounts with minimal required permissions following the principle of least privilege
  * Always use HTTPS (TLS) endpoints to encrypt data in transit
</Note>

Response
200 - A successful request returns a list of topic objects.

```json theme={null}
{
    "kind": "KafkaTopicList",
    "data": [
        {
            "kind": "KafkaTopic",
            "topic_name": "test-tenant.test-ns.topic-1",
            "is_internal": false,
            "partitions_count": 3
        },
        {
            "kind": "KafkaTopic",
            "topic_name": "topic-2",
            "is_internal": true,
            "partitions_count": 2
        },
        {
            "kind": "KafkaTopic",
            "topic_name": "topic-3",
            "is_internal": false,
            "partitions_count": 1
        }
    ]
}
```


# Return a list of consumer lags of the consumers belonging to the specified consumer group.
Source: https://docs.streamnative.io/api-references/kafka-rest-api/v1/consumer-groups/get-consumer-group-lags

get /admin/v1/consumer-groups/{consumer_group_id}/lags
Retrieves consumer lag information for all topic partitions consumed by the specified consumer group. Lag represents the difference between the latest offset and the consumer's current position.



# Return the consumer group specified by the consumer_group_id.
Source: https://docs.streamnative.io/api-references/kafka-rest-api/v1/consumer-groups/get-consumer-groups

get /admin/v1/consumer-groups/{consumer_group_id}
Gets detailed information about a specific consumer group identified by its ID. Returns consumer group metadata including state, members, and assigned partitions.



# Return a list of consumers that belong to the specified consumer group.
Source: https://docs.streamnative.io/api-references/kafka-rest-api/v1/consumer-groups/list-consumer-by-group-id

get /admin/v1/consumer-groups/{consumer_group_id}/consumers
Lists all active consumers that are members of the specified consumer group. Includes consumer metadata such as client ID, host, and partition assignments.



# Return the list of consumer groups.
Source: https://docs.streamnative.io/api-references/kafka-rest-api/v1/consumer-groups/list-consumer-groups

get /admin/v1/consumer-groups
Lists all consumer groups in the Kafka cluster. Returns a collection of consumer group objects with their basic metadata.



# Get messages from a topic
Source: https://docs.streamnative.io/api-references/kafka-rest-api/v1/messages/consume-messages

get /admin/v1/topics/{topicName}/messages
Get messages from a topic



# Produce records to the given topic, returning delivery reports for each record produced.
Source: https://docs.streamnative.io/api-references/kafka-rest-api/v1/messages/produce-messages

post /admin/v1/topics/{topicName}/records
Produces one or more records to the specified topic. Each record can include a key, value, headers, and optional partition assignment. Returns delivery reports confirming successful production.



# Create topic.
Source: https://docs.streamnative.io/api-references/kafka-rest-api/v1/topics/create-topics

post /admin/v1/topics
Creates a new topic in the Kafka cluster with the specified configuration. You can set partitions, replication factor, and other topic-level configurations.



# Delete the topic with the given topic_name.
Source: https://docs.streamnative.io/api-references/kafka-rest-api/v1/topics/delete-topics

delete /admin/v1/topics/{topicName}
Permanently deletes the specified topic and all its data from the Kafka cluster. This operation cannot be undone.



# Return the topic with the given topic_name.
Source: https://docs.streamnative.io/api-references/kafka-rest-api/v1/topics/get-topics

get /admin/v1/topics/{topicName}
Retrieves detailed information about a specific topic identified by its name. Returns topic metadata including configuration, partition details, and replication settings.



# Return the list of topics.
Source: https://docs.streamnative.io/api-references/kafka-rest-api/v1/topics/list-topics

get /admin/v1/topics
Lists all topics in the Kafka cluster or within a specific namespace. Returns topic metadata including partition count and internal/external topic classification.



# Unload the topic with the given topic_name. 
Source: https://docs.streamnative.io/api-references/kafka-rest-api/v1/topics/unload-topics

put /admin/v1/topics/{topicName}/unload
Unloads a topic from memory on the broker, releasing resources while preserving data. This operation is useful for reducing memory usage on topics that are not actively being used.



# Pulsar Message Rest API reference
Source: https://docs.streamnative.io/api-references/rest-messaging-api/rest-messaging-api



StreamNative Console supports a RESTful interface to Pulsar clusters. You can produce and consume messages without using the native Pulsar protocol or clients.

The Message Rest API supports both non-partitioned and partitioned topics as well as basic and Avro base struct schema.

Example use cases include:

* Send data to Pulsar from any frontend application built in any language
* Integrate Pulsar with existing automation tools
* Ingest Pulsar data into corporate dashboards and monitoring systems
* Provide instant access to data in motion for data scientist notebooks
* Ingest messages into a stream processing framework that may not support Pulsar

## Produce

**Description:**

Send a single message to the topic.

**Method:** `POST`

**Path:** `/admin/rest/topics/v1/<tenant>/<namespace>/<topic>/message`

**Request headers(`*` means required):**

* `Accept:application/json *` Accept the response body in JSON format
* `Content-Type: application/octet-stream *` Send the request body in binary format
* `X-Pulsar-Long-Schema-Version` Schema version in long type
* `X-Pulsar-Sequence-Id` Message sequence ID
* `X-Pulsar-Event-Time` Message event time
* `X-Pulsar-Property-<key>` Message property, key-value pair
* `X-Pulsar-Partition-Key` Message key

**Request body:**

`Binary data`

**Response headers:**

* `Content-Type: application/json *` Send the response body in JSON format

**Response body:**

`String`: message ID in string format

**Status code:**

* `201`: Send single message success
* `400`: Wrong request params
* `401`: Unauthorized
* `404`: Topic/Schema doesn't exist
* `500`: Internal server error

**Error**:

`JsonObject`:

reason(`string`): error information

## Consume

**Description:**

Consume a single message from the topic.

**Method:** `POST`

**Path:** `/admin/rest/topics/v1/<tenant>/<namespace>/<topic>/<subscriptionName>/message`

**Request headers(`*` means required):**

* `Accept:application/octet-stream *` Accept the response body in JSON format
* `Content-Type: application/json *` Send the request body in binary format

**Request body:**

* `JsonObject` :
  * `timeoutMillis (int)`: 3000

**Response headers( `*` means required):**

* `Content-Type: application/octet-stream *` Send the response body in JSON format
* `X-Pulsar-Base64-Schema-Version` Schema version in base64 encoded format
* `X-Pulsar-Long-Schema-Version` Schema version in long type
* `X-Pulsar-Message-Id *` Message ID in base64 encoded format
* `X-Pulsar-Message-String-Id *` Message ID in string format
* `X-Pulsar-Sequence-Id` Message sequence ID
* `X-Pulsar-Event-Time` Message event time
* `X-Pulsar-Property-<key>` Message property, key-value pair
* `X-Pulsar-Partition-Key` Message key

**Response body:**

`Binary data`

**Status code:**

* `200`: Get single message success
* `204`: Get null message by timeout
* `400`: Wrong request params
* `401`: Unauthorized
* `404`: Topic doesn't exist
* `500`: Internal server error

**Error**:

* `JsonObject`:
  * reason(`string`): error information

## Acknowledge

**Description:**

Acknowledge a single message from the topic.

**Method:** `PUT`

**Path:** `/admin/rest/topics/v1/<tenant>/<namespace>/<topic>/<subscriptionName>/message`

**Request headers(`*` means required):**

* `Accept: application/json *` Accept the response body in JSON format
* `Content-Type: application/json *` Send the request body in binary format

**Request body:**

`String`: message ID in base64 encoded format

**Response body:**

`no content`

**Status code:**

* `204`: Acknowledge success
* `400`: Wrong request params
* `401`: Unauthorized
* `404`: Topic doesn't exist
* `500`: Internal server error

**Error**:

`JsonObject`:

reason(`string`): error information

## Working with Schemas

**pulsar-rest** supports all schemas and is compatible with the Pulsar client in other languages. If you want to produce/consume a topic with schemas through pulsar-rest, see the following example.

Pulsar schema uses the Pulsar Admin API to create. see [manage schemas](https://pulsar.apache.org/docs/admin-api-schemas/) to learn the schema API or see [create schema for topics](/cloud/manage-data-streams/topic#create-schema-for-topics) to learn topic schema management on the StreamNative Console.

### Binary Data

Because the HTTP client needs to send binary data, use the [cURL](https://github.com/curl/curl) tool to encode the string `Hi Pulsar` into UTF-8 format bytes. If you need to send bytes in another format, specify the appropriate file. For more information see the [cURL documentation](https://everything.curl.dev/http/post/binary).

### Avro Base Struct Schema

Pulsar uses [Avro Specification](http://avro.apache.org/docs/current/spec.html) to declare the schema definition for `AvroBaseStructSchema`, which supports `AvroSchema`, `JsonSchema`, and `ProtobufSchema`.

### Multi-version schema

For basic schema types like String and JSON, **pulsar-rest** will automatically take the schema of the topic and put it into the metadata of the message. The client in other languages will serialize/deserialize the data based on this metadata.

### String Schema

**Create topic schema**

```bash theme={null}
curl -X POST https://<endpoint>:<port>/admin/v2/schemas/public/default/pulsar-rest-string/schema \
	--header "Authorization: Bearer <Access Token>" \
 	--header "Accept: application/json" \
 	--header "Content-Type: application/json" \
 	--data-raw '{"schema":"","type":"STRING","properties":{}}'
```

Expected output:

```bash theme={null}
{"version":{"version":0}}
```

**Create a subscription on the topic**

```bash theme={null}
curl -X PUT https://<endpoint>:<port>/admin/v2/persistent/public/default/pulsar-rest-string/subscription/rest-sub \
	--header 'Authorization: Bearer <Access Token>' \
 	--header 'Accept: application/json' \
 	--header 'Content-Type: application/json'
```

Expected output:

```bash theme={null}
# No content
```

**Produce messages with String schema**

```bash theme={null}
curl -X POST https://<endpoint>:<port>/admin/rest/topics/v1/persistent/public/default/pulsar-rest-string/message \
	-H "Authorization: Bearer <Access Token>" \
 	-H "Accept: application/json" \
 	-H "Content-Type: application/octet-stream" \
 	--data-binary 'Hi, Pulsar'
```

Expected output:

```bash theme={null}
#string format message id
10:0:-1:0
```

**Consume messages with String schema**

```bash theme={null}
curl -X POST https://<endpoint>:<port>/admin/rest/topics/v1/persistent/public/default/pulsar-rest-string/rest-sub/message \
	--header 'Authorization: Bearer <Access Token>' \
 	--header 'Accept: application/octet-stream' \
 	--data-raw '{"timeoutMillis":3000}' \
 	--header 'Content-Type: application/json' -v
```

Expected output:

```bash theme={null}
# Headers
X-Pulsar-Message-Id: CAoQACAAMAE=
X-Pulsar-Message-String-Id: 10:0:-1:0
X-Pulsar-Sequence-Id: 0

# Body
Hi, Pulsar
```

**Acknowledge messages**

```bash theme={null}
curl -X PUT https://<endpoint>:<port>/admin/rest/topics/v1/persistent/public/default/pulsar-rest-string/rest-sub/message \
	--header 'Authorization: Bearer <Access Token>' \
 	--header 'Accept: application/json' \
 	--header 'Content-Type: application/json' \
  --data-raw 'CAoQACAAMAE'
```

Expected output:

```bash theme={null}
# No content
```

**Negative acknowledge messages**

```bash theme={null}
curl -X PUT https://<endpoint>:<port>/admin/rest/topics/v2/persistent/public/default/pulsar-rest-string/rest-sub/message \
	--header 'Authorization: Bearer <Access Token>' \
 	--header 'Accept: application/json' \
 	--header 'Content-Type: application/json' \
    --data-raw '{"encodedMessageId":"CAoQACAAMAE=","negativeAck":"true"}'
```

Expected output:

```bash theme={null}
# No content
```

#### JSON Schema

**Create topic schema**

```bash theme={null}
curl -X POST https://<endpoint>:<port>/admin/v2/schemas/public/default/pulsar-rest-json/schema \
	--header "Authorization: Bearer <Access Token>" \
 	--header "Accept: application/json" \
 	--header "Content-Type: application/json" \
 	--data-raw '{"schema":"{\"type\":\"record\",\"name\":\"Student\",\"fields\":[{\"name\":\"age\",\"type\":\"int\"},{\"name\":\"name\",\"type\":[\"null\",\"string\"]}]}","type":"JSON","properties":{}}'
```

Expected output:

```bash theme={null}
{"version":{"version":0}}
```

**Create a subscription on the topic**

```bash theme={null}
curl -X PUT https://<endpoint>:<port>/admin/v2/persistent/public/default/pulsar-rest-json/subscription/rest-sub \
	--header 'Authorization: Bearer <Access Token>' \
 	--header 'Accept: application/json' \
 	--header 'Content-Type: application/json'
```

Expected output:

```bash theme={null}
# No content
```

**Produce messages with JSON schema**

```bash theme={null}
curl -X POST https://<endpoint>:<port>/admin/rest/topics/v1/persistent/public/default/pulsar-rest-json/message \
	--header 'Authorization: Bearer <Access Token>' \
  --header 'Accept: application/json' \
  --header 'Content-Type: application/octet-stream' \
 	--data-binary '{"name":"f8a42816-6183-4e3e-9f5d-51b36c17de5c","age":79}'
```

Expected output:

```bash theme={null}
#string format message id
10:0:-1:0
```

**Consume messages with JSON schema**

```bash theme={null}
curl -X POST https://<endpoint>:<port>/admin/rest/topics/v1/persistent/public/default/pulsar-rest-json/rest-sub/message \
	--header 'Authorization: Bearer <Access Token>' \
  --header 'Accept: application/octet-stream' \
  --data-raw '{"timeoutMillis":3000}' \
  --header 'Content-Type: application/json' -v
```

Expected output:

```bash theme={null}
# Headers
< X-Pulsar-Base64-Schema-Version: AAAAAAAAAAA=
< X-Pulsar-Long-Schema-Version: 0
< X-Pulsar-Message-Id: CIUCEAAgADAB
< X-Pulsar-Message-String-Id: 261:0:-1:0
< X-Pulsar-Sequence-Id: 0

# Body
{"name":"f8a42816-6183-4e3e-9f5d-51b36c17de5c","age":79}
```

**Acknowledge messages**

```bash theme={null}
curl -X PUT https://<endpoint>:<port>/admin/rest/topics/v1/persistent/public/default/pulsar-rest-json/rest-sub/message \
	--header "Authorization: Bearer <Access Token>" \
	--header "Accept: application/json" \
	--header "Content-Type: application/json" \
	--data-raw 'CIUCEAAgADAB'
```

Expected output:

```bash theme={null}
# No content
```

#### Avro Schema

**Create topic schema**

```bash theme={null}
curl -X POST https://<endpoint>:<port>/admin/v2/schemas/public/default/pulsar-rest-avro/schema \
	--header "Authorization: Bearer <Access Token>" \
 	--header "Accept: application/json" \
 	--header "Content-Type: application/json" \
 	--data-raw '{"schema":"{\"type\":\"record\",\"name\":\"Student\",\"fields\":[{\"name\":\"age\",\"type\":\"int\"},{\"name\":\"name\",\"type\":[\"null\",\"string\"]}]}","type":"AVRO","properties":{}}'
```

Expected output:

```bash theme={null}
{"version":{"version":0}}
```

**Create a subscription on the topic**

```bash theme={null}
curl -X PUT https://<endpoint>:<port>/admin/v2/persistent/public/default/pulsar-rest-avro/subscription/rest-sub \
	--header 'Authorization: Bearer <Access Token>' \
 	--header 'Accept: application/json' \
 	--header 'Content-Type: application/json'
```

Expected output:

```bash theme={null}
# No content
```

**Prepare Avro binary data**

Create `~/test-avro-student.avsc`

```json theme={null}
{
  "type": "record",
  "name": "Student",
  "fields": [
    {
      "name": "age",
      "type": "int"
    },
    {
      "name": "name",
      "type": ["null", "string"]
    }
  ]
}
```

Create `~/test-avro-student.json`

```json theme={null}
{ "age": 8, "name": { "string": "1ffc0cee-a3d9-42f2-b96d-efa09f54e782" } }
```

Renders a JSON-encoded Avro datum as binary.

```bash theme={null}
avro-tools jsontofrag --schema-file ~/test-avro-student.avsc   ~/test-avro-student.json > ~/test-avro-student.bin
```

<Note title="Note">
  If you don't have `avro-tools` installed, see the [Avro product page](https://github.com/apache/avro).
</Note>

**Produce messages with Avro schema**

When you want to send a message to a topic with a multi-version schema, put the version of the schema in the header.

`X-Pulsar-Long-Schema-Version` Schema version in long type

```bash theme={null}
cat ~/test-avro-student.bin | \
curl -X POST https://<endpoint>:<port>/admin/rest/topics/v1/persistent/public/default/pulsar-rest-avro/message \
	--header 'Authorization: Bearer <Access Token>' \
  --header 'Accept: application/json' \
  --header 'Content-Type: application/octet-stream' \
	--header 'X-Pulsar-Long-Schema-Version: 0' \
 	--data-binary @-
```

Expected output:

```bash theme={null}
#string format message id
10:0:-1:0
```

**Consume messages with Avro schema**

```bash theme={null}
curl -X POST https://<endpoint>:<port>/admin/rest/topics/v1/persistent/public/default/pulsar-rest-avro/rest-sub/message \
	--header 'Authorization: Bearer <Access Token>' \
  --header 'Accept: application/octet-stream' \
  --header 'Content-Type: application/json' \
	--dump-header ~/test-avro-student-recv-header.txt \
 	--data-raw '{"timeoutMillis":3000}' \
  --output ~/test-avro-student-recv.bin
```

**Verify the consumed message**

```bash theme={null}
avro-tools fragtojson --schema-file ~/test-avro-student.avsc ~/test-avro-student-recv.bin
```

Expected output:

```bash theme={null}
{
  "age" : 8,
  "name" : {
    "string" : "1ffc0cee-a3d9-42f2-b96d-efa09f54e782"
  }
}
```

**Acknowledge messages**

```bash theme={null}
curl -X PUT https://<endpoint>:<port>/admin/rest/topics/v1/persistent/public/default/pulsar-rest-avro/rest-sub/message \
	--header "Authorization: Bearer <Access Token>" \
	--header "Accept: application/json" \
	--header "Content-Type: application/json" \
	--data-raw 'CIUCEAAgADAB'
```

Expected output:

```bash theme={null}
# no content
```

#### Protobuf Schema

**Create topic schema**

```bash theme={null}
curl -X POST https://<endpoint>:<port>/admin/v2/schemas/public/default/pulsar-rest-proto/schema \
	--header "Authorization: Bearer <Access Token>" \
 	--header "Accept: application/json" \
 	--header "Content-Type: application/json" \
 	--data-raw '{"schema":"{\"type\":\"record\",\"name\":\"TestMessage\",\"fields\":[{\"name\":\"stringField\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"default\":\"\"},{\"name\":\"doubleField\",\"type\":\"double\",\"default\":0},{\"name\":\"intField\",\"type\":\"int\",\"default\":0}]}","type":"PROTOBUF","properties":{}}'
```

Expected output:

```bash theme={null}
{"version":{"version":0}}
```

**Create a subscription on the topic**

```bash theme={null}
curl -X PUT https://<endpoint>:<port>/admin/v2/persistent/public/default/pulsar-rest-proto/subscription/rest-sub \
	--header 'Authorization: Bearer <Access Token>' \
 	--header 'Accept: application/json' \
 	--header 'Content-Type: application/json'
```

Expected output:

```bash theme={null}
# No content
```

**Prepare protobuf data**

Create `~/test-proto-test.proto`

```protobuf theme={null}
syntax = "proto3";

message TestMessage {
    string stringField = 1;
    double doubleField = 2;
    int32 intField = 3;
}
```

Create `~/test-proto-message.txt`

```protobuf theme={null}
stringField:"string filed"
doubleField: 3.14
intField: 8
```

Read a text-format message of the given type from standard input and write it in binary to standard output.

```bash theme={null}
cat ~/test-proto-message.txt |  protoc --proto_path=<your user directory, ~> --encode='TestMessage' ~/test-proto-test.proto > ~/test-proto-message.bin
```

**Produce messages with Protobuf schema**

When you want to send a message to a topic with a multi-version schema, put the version of the schema in the header.

`X-Pulsar-Long-Schema-Version` Schema version in long type

```bash theme={null}
cat ~/test-proto-message.bin | \
curl -X POST https://<endpoint>:<port>/admin/rest/topics/v1/persistent/public/default/pulsar-rest-proto/message \
	--header 'Authorization: Bearer <Access Token>' \
  --header 'Accept: application/json' \
  --header 'Content-Type: application/octet-stream' \
	--header 'X-Pulsar-Long-Schema-Version: 0' \
 	--data-binary '@-'
```

Expected output:

```bash theme={null}
#string format message id
10:0:-1:0
```

**Consume messages with Protobuf schema**

```bash theme={null}
curl -X POST https://<endpoint>:<port>/admin/rest/topics/v1/persistent/public/default/pulsar-rest-proto/rest-sub/message \
	--header 'Authorization: Bearer <Access Token>' \
  --header 'Accept: application/octet-stream' \
  --header 'Content-Type: application/json' \
	--dump-header ~/test-proto-message-recv-header.txt \
 	--data-raw '{"timeoutMillis":3000}' \
  --output ~/test-proto-message-recv.bin
```

Verify the consumed data

```bash theme={null}
cat ~/test-proto-message-recv.bin |  protoc --proto_path=<your user directory> --decode='TestMessage' ~/test-proto-test.proto > ~/test-proto-message-recv.txt && cat ~/test-proto-message-recv.txt
```

Expected output:

```bash theme={null}
stringField: "string filed"
doubleField: 3.14
intField: 8
```

**Acknowledge messages**

```bash theme={null}
curl -X PUT https://<endpoint>:<port>/admin/rest/topics/v1/persistent/public/default/pulsar-rest-ptoto/rest-sub/message \
	--header "Authorization: Bearer <Access Token>" \
	--header "Accept: application/json" \
	--header "Content-Type: application/json" \
	--data-raw 'CIUCEAAgADAB'
```

Expected output:

```bash theme={null}
# No content
```


# Manage billing using StreamNative Cloud Console
Source: https://docs.streamnative.io/cloud/billing/billing



This section describes how you can view your invoices, change your payment method, and update your contact information through StreamNative Cloud Console.

<Note title="Note">
  * This section is only available for StreamNative Cloud services subscribed through StreamNative Cloud Console.
  * For AWS Marketplace information, see [get started with StreamNative Cloud on AWS Marketplace](/cloud/billing/billing-aws).
  * For Google Cloud Marketplace information, see [get started with StreamNative Cloud on Google Cloud Marketplace](/cloud/billing/billing-gcp).
</Note>

## Check invoices

To check your past monthly invoices, follow these steps.

1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Billing & Payment**. This takes you to the Stripe billing page.

   <img alt="screenshot of the settings menu with billing and payment highlighted" />

2. In the **Invoice History** section, select the invoice icon next to the date you want to review.

   <img alt="screenshot of invoice history section on the billing page" />

3. On the invoice page, click **Download invoice**.

## Change your payment method

To update your payment method, follow these steps.

1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Billing & Payment**.

2. In the **Payment Method** section, click **Add payment method**.

3. Enter your credit card information, make sure **Use as default payment method** is checked, and click **Add**. The payment method is added and is now the default.

4. (Optional) If you want to delete a previous payment method, click the X next to the payment method information and click **Confirm**. You can only delete a payment method if it is not the default method.

## Change the billing email address

Credit card payment receipts or invoices are emailed to the address that was initially provided during sign up for StreamNative Cloud. To update the email address where the billing is sent, follow these steps.

1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Billing & Payment**.

2. In the **Billing & Shipping Informations** section, click **Update information**.

3. Update the email on the Billing Information page. You do not need to add a physical address or phone number.

4. Click **Save**.

## Related topics

* Learn more about [StreamNative Cloud Billing Overview](/cloud/billing/billing-overview).
* [Stop charges for your Pulsar cluster](/cloud/billing/stop-charges).


# Get started with StreamNative Cloud on Alibaba Cloud Marketplace with Pay-As-You-Go
Source: https://docs.streamnative.io/cloud/billing/billing-alibaba



You can access the StreamNative Cloud product through Alibaba Cloud Marketplace. If you already have an Alibaba Cloud account, you can get started by subscribing to the StreamNative Cloud product in Alibaba Cloud Marketplace as detailed below. After subscribing and connecting your Alibaba Cloud account to your StreamNative Cloud account, you can view and pay your bills on the Billing Dashboard in your Alibaba Cloud account. This subscription uses a Pay-As-You-Go billing model, so your bills will vary depending on usage.

## Prerequisites

To use this service, you must complete the following:

* Have an Alibaba Cloud account with an account ID enabled for purchases. Contact your billing administrator if you have questions about your Alibaba Cloud account.
* Create a StreamNative Cloud account by following the sign-up of the [StreamNative Cloud Quick Start](https://console.streamnative.cloud/signup?from=site_docs).
* Log in to both [Alibaba Cloud Marketplace](https://marketplace.alibabacloud.com/) and [StreamNative Cloud Console](https://console.streamnative.cloud/signup?from=site_docs).

## Procedures

This section describes how to get up and running with StreamNative Cloud on Alibaba Cloud Marketplace using a self-service Pay-As-You-Go account.

### Subscribe to the StreamNative Cloud service

To get up and running with StreamNative Cloud on Alibaba Cloud Marketplace using a self-service Pay-As-You-Go account:

1. Go to [Alibaba Cloud Marketplace](https://marketplace.alibabacloud.com/products/56730001/sgcmgj00036040.html?spm=a3c0i.26795044.0.0.72632faaCZCqxJ\&innerSource=search).

2. Review the product information and click **Activate Now**.

   <img alt="Product overview of StreamNative Cloud service on Alibaba Cloud marketplace" />

3. Select the option toe agree to the community agreement and click OK

<img alt="agree-to-active-alibaba-subscription.png" />

4. Once the request for activation is successfully submitted, you will see a confirmation message. Click OK in the confirmation message to navigate to the subscription details.

<img alt="alibaba-activation-submitted.png" />

After getting redirected you must see the page which says 'My Software Subscription' as shown below

<img alt="view-alibaba-subscription.png" />

6. Click on Auto Login as shown in the picture above to automatically get redirected back to the StreamNative Cloud Console. This step also connects your Alibaba Cloud marketplace account to your StreamNative account.

<Note title="Note">
  Currently, you can't link your Alibaba Cloud subscription to an existing organization on StreamNative Console. To bind to an existing organization, [submit a ticket](https://support.streamnative.io/hc/en-us/requests/new) with this request to the support team.
</Note>

### Create your organization

Depending on your StreamNative Cloud account setup, do one of the followings:

* If you haven't not created an organization yet, enter a name for your first organization and follow the prompts to complete the setup of your organization.

<img alt="alibaba-org.png" />

* If you have already created at least one organization, on the Organization page, click Create organization, enter a name for your new organization, and click Create.

### Create your Pulsar instance and Pulsar cluster

<Note title="Note">
  For Bring Your Own Cloud (BYOC) users, contact your StreamNative sales to get your BYOC pool members provisioned before continuing to create your Pulsar instance and cluster.
</Note>

1. Navigate to the organization that you just created.

2. Click the name of your new organization, and then click **create instance** at the end of the row of the organization that you have just created.

3. On the **Instance** page, click **CREATE INSTANCE**.

4. Click **Deploy Dedicated** to start the instance creation process on StreamNative Cloud. Alternatively, you can click **Deploy Serverless** to create a Serverless cluster or click **Deploy BYOC** to create a BYOC cluster.

5. On the **Instance Configuration** page, enter a name for your instance, select the Alibaba Cloud infrastructure pool, and select the multi Availability Zone (AZ). The instance name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

6. Click **Cluster Location** to start the cluster creation process.

7. On the **Cluster Location** page, enter a name for your cluster, select the cluster location, and then click **Cluster Size**. The cluster name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

8. On the **Cluster Size** page, configure the cluster, and then click **Payment**.

   * On the **Basic** tab, select custom sizing options.

   * On the **Advanced** tab, in the **Features** area, enable the cluster features you want on your cluster.

It may take several minutes for the instance and cluster creation process to complete.

## Related topics

* Learn more about [StreamNative Cloud](/cloud/overview/cloud-overview).
* Learn more about [StreamNative Cloud Billing Overview](/cloud/billing/billing-overview).
* [Stop charges for your Pulsar cluster](/cloud/billing/stop-charges).


# Get started with StreamNative Cloud on AWS Marketplace with Pay-As-You-Go
Source: https://docs.streamnative.io/cloud/billing/billing-aws



You can access the StreamNative Cloud product through Amazon Web Services (AWS) Marketplace. If you already have an AWS account, you can get started by subscribing to the StreamNative Cloud product in AWS Marketplace as detailed below. After subscribing and connecting your AWS account to your StreamNative Cloud account, you can view and pay your bills on the Billing Dashboard in your AWS account. This subscription uses a Pay-As-You-Go billing model, so your bills will vary depending on usage.

## Prerequisites

To use this service, you must complete the following:

* Have an AWS account with an account ID enabled for purchases. Contact your billing administrator if you have questions about your AWS account.
* Create a StreamNative Cloud account by following the sign-up of the [StreamNative Cloud Quick Start](/cloud/get-started/quickstart-console#sign-up).
* Log in to both [AWS Marketplace](https://aws.amazon.com/marketplace) and [StreamNative Cloud Console](https://console.streamnative.cloud/?defaultMethod=signup).

## Procedures

This section describes how to get up and running with StreamNative Cloud on AWS Marketplace using a self-service Pay-As-You-Go account.

### Subscribe to the StreamNative Cloud service

To get up and running with StreamNative Cloud on AWS Marketplace using a self-service Pay-As-You-Go account:

1. Go to [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-rqfspbolevs3o?sr=0-2\&ref_=beagle\&applicationId=AWSMPContessa).

2. Review the product information and click **View purchase options**.

   <img alt="Product overview of StreamNative Cloud service on AWS marketplace" />

3. Confirm the pricing details and click **Subscribe**. A dialog box displays.

4. Click **Set up your account** to be automatically redirected back to the StreamNative Cloud Console. This step also connects your AWS marketplace account to your StreamNative account.

### Create your organization

<Note title="Note">
  Currently, you cannot link your AWS entitlement to an existing organization on StreamNative Console. To bind to an existing organization, [submit a ticket](https://support.streamnative.io/hc/en-us/requests/new) with this request to the support team.
</Note>

Depending on your StreamNative Cloud account setup, do one of the followings:

* If you have not created an organization yet, enter a name for your first organization and follow the prompts to complete the setup of your organization.

  <img alt="a screenshot of creating an organization for AWS marketplace" />

* If you have already created at least one organization, on the **Organization** page, click **Create organization**, enter a name for your new organization, and click **Create**.

### Create your Pulsar instance and Pulsar cluster

<Note title="Note">
  For BYOC users, contact your StreamNative sales to get your BYOC pool members provisioned before continuing to create your Pulsar instance and cluster.
</Note>

1. Navigate to the organization that you just created.

2. Click the name of your new organization, and then click **create instance** at the end of the row of the organization that you have just created.

3. On the **Instance** page, click **CREATE INSTANCE**.

4. Click **Deploy Dedicated** to start the instance creation process on StreamNative Cloud. Alternatively, you can click **Deploy Serverless** to create a Serverless cluster or click **Deploy BYOC** to create a BYOC cluster.

5. On the **Instance Configuration** page, enter a name for your instance, select the AWS infrastructure pool, and select the multi Availability Zone (AZ). The instance name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

6. Click **Cluster Location** to start the cluster creation process.

7. On the **Cluster Location** page, enter a name for your cluster, select the cluster location, and then click **Cluster Size**. The cluster name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

8. On the **Cluster Size** page, configure the cluster, and then click **Payment**.

   * On the **Basic** tab, select custom sizing options.

   * On the **Advanced** tab, in the **Features** area, enable the cluster features you want on your cluster.

It may take several minutes for the instance and cluster creation process to complete.

## Related topics

* Learn more about [StreamNative Cloud](/cloud/overview/cloud-overview).
* Learn more about [StreamNative Cloud Billing Overview](/cloud/billing/billing-overview).
* [Get started with StreamNative Cloud on AWS Marketplace with commitments](/cloud/billing/billing-aws-commitments).
* [Stop charges for your Pulsar cluster](/cloud/billing/stop-charges).


# Get started with StreamNative Cloud on AWS Marketplace with commitments
Source: https://docs.streamnative.io/cloud/billing/billing-aws-commitments



With a commitment, you sign up for a minimum spend amount and get a discount on your committed usage of StreamNative Cloud. If you spend beyond the commit, you actually pay as you go monthly for the additional usage.

This document describes how to get up and running with StreamNative Cloud on AWS Marketplace with usage-based billing commitment.

## Prerequisites

To use this service, you must complete the following:

* Have an AWS account with an account ID enabled for purchases. Contact your billing administrator if you have questions about your AWS account.
* Create a StreamNative Cloud account by following the sign-up of the [StreamNative Cloud Quick Start](/cloud/get-started/quickstart-console#sign-up).
* Log in to both [AWS Marketplace](https://aws.amazon.com/marketplace) and [StreamNative Cloud Console](https://console.streamnative.cloud/?defaultMethod=signup).

## Procedures

You will receive an email from StreamNative sales after you purchase an AWS Marketplace private offer with StreamNative. There is a link embedded in your email that opens an offer page in AWS Marketplace.

### Review and accept your private offer

1. Click the embedded link to your AWS Marketplace private offer in your email and you will be automatically redirected back to the AWS marketplace.
2. Review the offer details, pricing information, usage metric charges, policies, and terms of use, check the **I agree to the terms above** checkbox, and then click **ACCEPT**. A dialog box displays.
3. Click **Set up your account** to be automatically redirected back to the StreamNative Cloud Console. This step also connects your AWS marketplace account to your StreamNative account.

### Create your organization

<Note title="Note">
  Currently, you cannot link your AWS entitlement to an existing organization on StreamNative Console. To bind to an existing organization, [submit a ticket](https://support.streamnative.io/hc/en-us/requests/new) with this request to the support team.
</Note>

Depending on your StreamNative Cloud account setup, do one of the followings:

* If you have not created an organization yet, enter a name for your first organization and follow the prompts to complete the setup of your organization.

  <img alt="a screenshot of creating an organization for AWS marketplace" />

* If you have already created at least one organization, on the **Organization** page, click **Create organization**, enter a name for your new organization, and click **Create**.

### Create your Pulsar instance and Pulsar cluster

<Note title="Note">
  For BYOC users, contact your StreamNative sales to get your BYOC pool members provisioned before continuing to create your Pulsar instance and cluster.
</Note>

1. Navigate to the organization that you just created.

2. Click the name of your new organization, and then click **create instance** at the end of the row of the organization that you have just created.

3. On the **Instance** page, click **CREATE INSTANCE**.

4. Click **Deploy Dedicated** to start the instance creation process on StreamNative Cloud. Alternatively, you can click **Deploy Serverless** to create a Serverless cluster or click **Deploy BYOC** to create a BYOC cluster.

5. On the **Instance Configuration** page, enter a name for your instance, select the AWS infrastructure pool, and select the multi Availability Zone (AZ). The instance name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

6. Click **Cluster Location** to start the cluster creation process.

7. On the **Cluster Location** page, enter a name for your cluster, select the cluster location, and then click **Cluster Size**. The cluster name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

8. On the **Cluster Size** page, configure the cluster, and then click **Payment**.

   * On the **Basic** tab, select custom sizing options.

   * On the **Advanced** tab, in the **Features** area, enable the cluster features you want on your cluster.

It may take several minutes for the instance and cluster creation process to complete.

## Related topics

* Learn more about [StreamNative Cloud](/cloud/overview/cloud-overview).
* Learn more about [StreamNative Cloud Billing Overview](/cloud/billing/billing-overview).
* [Get started with StreamNative Cloud on AWS Marketplace with Pay-As-You-Go](/cloud/billing/billing-aws).
* [Stop charges for your Pulsar cluster](/cloud/billing/stop-charges).


# Get started with StreamNative Cloud on Azure Cloud Marketplace with Pay-As-You-Go
Source: https://docs.streamnative.io/cloud/billing/billing-azure



You can access the StreamNative Cloud product through Azure Cloud Marketplace. If you already have a Azure Cloud account, you can get started by subscribing to the StreamNative Cloud product in Azure Cloud Marketplace as detailed below. After subscribing and connecting your Azure Cloud account to your StreamNative Cloud account, you can view and pay your bills on the [**Billing**](https://learn.microsoft.com/en-us/marketplace/billing-invoicing) console in your Azure Portal account. This subscription uses a Pay-As-You-Go billing model, so your bills will vary depending on usage.

## Prerequisites

To use this service, you must complete the following:

* Have a Azure Portal account with payment / billing account setup to purchase any service in Azure marketplace. Contact your billing administrator if you have questions about setting up payment / billing account.
* Create a StreamNative Cloud account by following the sign-up of the [StreamNative Cloud Quick Start](/cloud/get-started/quickstart-console#sign-up).
* Log in to both [Microsoft Azure Marketplace](https://portal.azure.com/#view/Microsoft_Azure_Marketplace/MarketplaceOffersBlade/selectedMenuItemId/home) and [StreamNative Cloud Console](https://console.streamnative.cloud/?defaultMethod=signup).

## Procedures

This section describes how to get up and running with StreamNative Cloud on Azure Cloud Marketplace using a self-service Pay-As-You-Go account.

### Subscribe to the StreamNative Cloud service

1. Go to [Azure Cloud Marketplace](https://portal.azure.com/#view/Microsoft_Azure_Marketplace/GalleryItemDetailsBladeNopdl/id/streamnative.apache-pulsar-by-streamnative-azure/).

2. Review the product information and the price details, select Plan (Annual Commitment and Pay As You Go) and then click **SUBSCRIBE**.

   <img alt="View logs" />

3. Navigate to the SaaS subscription created in the previous step and click on 'Open SaaS Account on publisher's site'. That will navigate you to StreamNative Cloud Console.

   <img alt="View logs" />

### Create your organization

<Note title="Note">
  Currently, you cannot link your AWS entitlement to an existing organization on StreamNative Console. To bind to an existing organization, [submit a ticket](https://support.streamnative.io/hc/en-us/requests/new) with this request to the support team.
</Note>

Depending on your StreamNative Cloud account setup, do one of the followings:

* If you have not created an organization yet, enter a name for your first organization and follow the prompts to complete the setup of your organization.

  <img alt="a screenshot of creating an organization for AWS marketplace" />

* If you have already created at least one organization, on the **Organization** page, click **Create organization**, enter a name for your new organization, and click **Create**.

### Create your Pulsar instance and Pulsar cluster

<Note title="Note">
  For BYOC users, contact your StreamNative sales to get your BYOC pool members provisioned before continuing to create your Pulsar instance and cluster.
</Note>

1. Navigate to the organization that you just created.

2. Click the name of your new organization, and then click **create instance** at the end of the row of the organization that you have just created.

3. On the **Instance** page, click **CREATE INSTANCE**.

4. Click **Deploy Dedicated** to start the instance creation process on StreamNative Cloud. Alternatively, you can click **Deploy Serverless** to create a Serverless cluster or click **Deploy BYOC** to create a BYOC cluster.

5. On the **Instance Configuration** page, enter a name for your instance, select the AWS infrastructure pool, and select the multi Availability Zone (AZ). The instance name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

6. Click **Cluster Location** to start the cluster creation process.

7. On the **Cluster Location** page, enter a name for your cluster, select the cluster location, and then click **Cluster Size**. The cluster name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

8. On the **Cluster Size** page, configure the cluster, and then click **Payment**.

   * On the **Basic** tab, select custom sizing options.

   * On the **Advanced** tab, in the **Features** area, enable the cluster features you want on your cluster.

It may take several minutes for the instance and cluster creation process to complete.

## Related topics

* Learn more about [StreamNative Cloud](/cloud/overview/cloud-overview).
* Learn more about [StreamNative Cloud Billing Overview](/cloud/billing/billing-overview).
* [Get started with StreamNative Cloud on AWS Marketplace with commitments](/cloud/billing/billing-aws-commitments).
* [Stop charges for your Pulsar cluster](/cloud/billing/stop-charges).


# Get started with StreamNative Cloud on Azure Cloud Marketplace with commitments
Source: https://docs.streamnative.io/cloud/billing/billing-azure-commitments



With a commitment, you sign up for a minimum spend amount and get a discount on your committed usage of StreamNative Cloud. All usage, including overage, is [list price](https://www.suger.io/docs/gcp-marketplace/private-offer/#commit-discount-with-additional-usage-at-list-price). So, if you spend beyond the commit, you actually pay the list price for the additional usage.

This document describes how to get up and running with StreamNative Cloud on Azure Cloud Marketplace with usage-based billing commitment.

## Prerequisites

To use this service, you must complete the following:

* Have a Azure Portal account with payment / billing account setup to purchase any service in Azure marketplace. Contact your billing administrator if you have questions about setting up payment / billing account.
* Create a StreamNative Cloud account by following the sign-up of the [StreamNative Cloud Quick Start](/cloud/get-started/quickstart-console#sign-up).
* Log in to both [Microsoft Azure Marketplace](https://portal.azure.com/#view/Microsoft_Azure_Marketplace/MarketplaceOffersBlade/selectedMenuItemId/home) and [StreamNative Cloud Console](https://console.streamnative.cloud/?defaultMethod=signup).

## Procedures

You will receive an email from StreamNative sales after you purchase a Azure Cloud Marketplace private offer with StreamNative. There is a link embedded in your email that opens an offer page in Azure Cloud Marketplace.

### Review and accept your private offer

1. Click the embedded link in your email and you will be automatically redirected back to the Microsoft Azure Cloud Marketplace.
2. Within Azure Marketplace, click on Private Offer Management and view the private offer
3. Review the offer details, pricing information, usage metric charges, policies, and terms of use, check the **I agree to the terms above** checkbox, and then click **ACCEPT**. A dialog box displays.
4. Click on View Resources, and then on the SaaS subscription
5. Click on 'Open SaaS Account on publisher's site' to be automatically redirected back to the StreamNative Cloud Console. This step also connects your Azure Cloud Marketplace account to your StreamNative Cloud account.

### Create your organization

<Note title="Note">
  Currently, you cannot link your AWS entitlement to an existing organization on StreamNative Console. To bind to an existing organization, [submit a ticket](https://support.streamnative.io/hc/en-us/requests/new) with this request to the support team.
</Note>

Depending on your StreamNative Cloud account setup, do one of the followings:

* If you have not created an organization yet, enter a name for your first organization and follow the prompts to complete the setup of your organization.

  <img alt="a screenshot of creating an organization for AWS marketplace" />

* If you have already created at least one organization, on the **Organization** page, click **Create organization**, enter a name for your new organization, and click **Create**.

### Create your Pulsar instance and Pulsar cluster

<Note title="Note">
  For BYOC users, contact your StreamNative sales to get your BYOC pool members provisioned before continuing to create your Pulsar instance and cluster.
</Note>

1. Navigate to the organization that you just created.

2. Click the name of your new organization, and then click **create instance** at the end of the row of the organization that you have just created.

3. On the **Instance** page, click **CREATE INSTANCE**.

4. Click **Deploy Dedicated** to start the instance creation process on StreamNative Cloud. Alternatively, you can click **Deploy Serverless** to create a Serverless cluster or click **Deploy BYOC** to create a BYOC cluster.

5. On the **Instance Configuration** page, enter a name for your instance, select the AWS infrastructure pool, and select the multi Availability Zone (AZ). The instance name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

6. Click **Cluster Location** to start the cluster creation process.

7. On the **Cluster Location** page, enter a name for your cluster, select the cluster location, and then click **Cluster Size**. The cluster name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

8. On the **Cluster Size** page, configure the cluster, and then click **Payment**.

   * On the **Basic** tab, select custom sizing options.

   * On the **Advanced** tab, in the **Features** area, enable the cluster features you want on your cluster.

It may take several minutes for the instance and cluster creation process to complete.

## Related topics

* Learn more about [StreamNative Cloud](/cloud/overview/cloud-overview).
* Learn more about [StreamNative Cloud Billing Overview](/cloud/billing/billing-overview).
* [Get started with StreamNative Cloud on AWS Marketplace with commitments](/cloud/billing/billing-aws-commitments).
* [Stop charges for your Pulsar cluster](/cloud/billing/stop-charges).


# Billing FAQ
Source: https://docs.streamnative.io/cloud/billing/billing-faq



## Frequently Asked Questions

This page provides answers to frequently asked questions about billing on StreamNative Cloud. If you don't find the answer you're looking for, please [contact our support team](https://support.streamnative.io).

### What is a Throughput Unit (TU)?

A Throughput Unit (TU) is a capacity planning abstraction used for Dedicated Kafka and BYOC clusters. Each TU represents a standardized amount of throughput capacity (25 MBps ingress, 75 MBps egress, 2,500 entries per second). TUs allow you to configure cluster capacity without managing infrastructure details. How TUs are charged depends on the cluster type — see [RTU](/cloud/billing/billing-overview#rtu) for Dedicated Kafka and [ETU](/cloud/billing/billing-overview#etu) for BYOC clusters.

### What is the difference between RTU and ETU?

**RTU (Reserved Throughput Unit)** and **ETU (Elastic Throughput Unit)** are two charging models for Throughput Units:

* **RTU** applies to Dedicated Kafka clusters. You are charged for the number of TUs you reserve when configuring the cluster, regardless of actual usage. This is a fixed hourly charge.
* **ETU** applies to Serverless and BYOC/BYOC Pro clusters. You are charged based on actual throughput usage. For BYOC clusters, you configure TUs as the reserved capacity, but billing is based on actual consumption.

For more details, see the [billing model comparison](/cloud/billing/billing-overview#billing-model-comparison).

### Which billing model applies to my cluster?

The billing model depends on your cluster type:

| Cluster Type                       | Capacity Planning    | Charging Unit                         |
| ---------------------------------- | -------------------- | ------------------------------------- |
| Serverless                         | Auto-scaled          | ETU (Elastic Throughput Unit)         |
| Dedicated Kafka *(Public Preview)* | TU (Throughput Unit) | RTU (Reserved Throughput Unit)        |
| Dedicated Pulsar                   | CU + SU              | CU (Compute Unit) + SU (Storage Unit) |
| BYOC / BYOC Pro (Kafka)            | TU (Throughput Unit) | ETU (Elastic Throughput Unit)         |
| BYOC / BYOC Pro (Pulsar)           | CU + SU              | ETU (Elastic Throughput Unit)         |

For a detailed comparison, see the [billing model summary](/cloud/billing/billing-overview#billing-dimensions).

### How do I scale my Dedicated Kafka cluster?

You can adjust the number of Throughput Units (TUs) for your Dedicated Kafka cluster using the TU slider in the StreamNative Cloud Console. The self-service range is 1 to 20 TUs (integer values only). For configurations beyond 20 TUs, [contact StreamNative sales](https://www.streamnative.io/contact).

### What is the difference between pre-replication and post-replication write throughput and how does it impact pricing?

Pre-replication write throughput refers to the amount of data written by clients before it is replicated across multiple bookies. Post-replication write throughput, on the other hand, is the total amount of data written after replication has occurred.

For example, if you write 1 GB of data with a replication factor of 3, the pre-replication write throughput is 1 GB, while the post-replication write throughput is 3 GB.

In terms of pricing, StreamNative Cloud bills based on the pre-replication write throughput. This means you are charged for the actual data you write, not for the additional copies created for replication purposes. This pricing model ensures that you don't incur extra costs for maintaining data redundancy and fault tolerance in your Pulsar cluster.

### How often are billing metrics updated?

Billing metrics are typically updated hourly. However, there may be a slight delay in the reporting of usage data, so the most recent hour's data might not be immediately visible in your billing dashboard.

### Are there any additional costs for using the StreamNative Cloud Console?

No, there are no additional costs for using the StreamNative Cloud Console. The console is provided as a free tool to manage and monitor your Pulsar clusters, instances, and other resources.

### What happens if I exceed my free credits?

If you're approaching or have exceeded your free credit limit, StreamNative will notify you via email. In most cases, your services will continue to run, but you are required to add a payment method to ensure uninterrupted service. If you have questions about your free credits, please contact the StreamNative team at [https://streamnative.io/contact](https://streamnative.io/contact).

### What is dimensional consumption?

Dimensional consumption refers to the usage of resources across different dimensions in StreamNative Cloud, particularly for Serverless clusters. These dimensions include:

1. Ingress (Data In): The volume of data being written to your cluster, measured in bytes per second.
2. Egress (Data Out): The volume of data being read from your cluster, measured in bytes per second.
3. Data Entries: The number of entries (batches of messages) processed by the cluster per second, including both produce and consume operations.

For Serverless clusters, these dimensions are used to calculate your usage in terms of Elastic Throughput Units (ETUs). Your ETU consumption for a given hour is based on the highest usage across these dimensions, relative to the capacity provided by one ETU. This approach allows for flexible, scalable resource allocation and pricing, where you only pay for the capacity you actually use, up to your cluster's maximum capacity.

Understanding dimensional consumption is crucial for optimizing your usage and costs in StreamNative Cloud, especially for Serverless and BYOC clusters.

<Note title="Note">
  Dimensional consumption applies to ETU-based clusters (Serverless, BYOC). For Dedicated Kafka clusters using RTU-based billing, you are charged at a fixed rate for the reserved capacity regardless of actual dimensional consumption.
</Note>


# Get started with StreamNative Cloud on Google Cloud Marketplace with Pay-As-You-Go
Source: https://docs.streamnative.io/cloud/billing/billing-gcp



You can access the StreamNative Cloud product through Google Cloud Marketplace. If you already have a Google Cloud account, you can get started by subscribing to the StreamNative Cloud product in Google Cloud Marketplace as detailed below. After subscribing and connecting your Google Cloud account to your StreamNative Cloud account, you can view and pay your bills on the [**Billing**](https://cloud.google.com/billing/docs/how-to/reports) console in your Google Cloud account. This subscription uses a Pay-As-You-Go billing model, so your bills will vary depending on usage.

## Prerequisites

To use this service, you must complete the following:

* Have a Google Cloud account with an account ID enabled for purchases. Contact your billing administrator if you have questions about your Google Cloud account.
* Create a StreamNative Cloud account by following the sign-up of the [StreamNative Cloud Quick Start](/cloud/get-started/quickstart-console#sign-up).
* Log in to both [Google Cloud Marketplace](https://console.cloud.google.com/marketplace) and [StreamNative Cloud Console](https://console.streamnative.cloud/?defaultMethod=signup).

## Procedures

This section describes how to get up and running with StreamNative Cloud on Google Cloud Marketplace using a self-service Pay-As-You-Go account.

### Subscribe to the StreamNative Cloud service

1. Go to [Google Cloud Marketplace](https://console.cloud.google.com/marketplace/product/streamnative-public/apache-pulsar-managed-by-streamnative).

2. Review the product information and the price details, and then click **SUBSCRIBE**.

   <img alt="a screenshot of subscribing to Streamnative Cloud on GCP" />

3. Click **MANAGE ON PROVIDER** to be automatically redirected back to the StreamNative Cloud Console. This step also connects your Google Cloud Marketplace account to your StreamNative Cloud account.

   <img alt="a screenshot of managing on provider on GCP" />

### Create your organization

<Note title="Note">
  Currently, you cannot link your GCP entitlement to an existing organization on StreamNative Console. To bind to an existing organization, [submit a ticket](https://support.streamnative.io/hc/en-us/requests/new) with this request to the support team.
</Note>

Depending on your StreamNative Cloud account setup, do one of the followings:

* If you have not created an organization yet, enter a name for your first organization and follow the prompts to complete the setup of your organization.

  <img alt="a screenshot of creating an organization for GCP" />

* If you have already created at least one organization, on the **Organization** page, click **Create organization**, enter a name for your new organization, and click **Create**.

### Create your Pulsar instance and Pulsar cluster

<Note title="Note">
  For BYOC users, contact your StreamNative sales to get your BYOC pool members provisioned before continuing to create your Pulsar instance and cluster.
</Note>

1. Navigate to the organization that you just created.

2. Click the name of your new organization, and then click **create instance** at the end of the row of the organization that you have just created.

3. On the **Instance** page, click **CREATE INSTANCE**.

4. Click **Deploy Dedicated** to start the instance creation process on StreamNative Cloud. Alternatively, you can click **Deploy Serverless** to create a Serverless cluster or click **Deploy BYOC** to create a BYOC cluster.

5. On the **Instance Configuration** page, enter a name for your instance, select the Google Cloud infrastructure pool, and select the multi Availability Zone (AZ). The instance name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

6. Click **Cluster Location** to start the cluster creation process.

7. On the **Cluster Location** page, enter a name for your cluster, select the cluster location, and then click **Cluster Size**. The cluster name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

8. On the **Cluster Size** page, configure the cluster, and then click **Payment**.

   * On the **Basic** tab, select custom sizing options.

   * On the **Advanced** tab, in the **Features** area, enable the cluster features you want on your cluster.

It may take several minutes for the instance and cluster creation process to complete.

## Related topics

* Learn more about [StreamNative Cloud](/cloud/overview/cloud-overview).
* Learn more about [StreamNative Cloud Billing Overview](/cloud/billing/billing-overview).
* [Get started with StreamNative Cloud on Google Cloud Marketplace with commitments](/cloud/billing/billing-gcp-commitments).
* [Stop charges for your Pulsar cluster](/cloud/billing/stop-charges).


# Get started with StreamNative Cloud on Google Cloud Marketplace with commitments
Source: https://docs.streamnative.io/cloud/billing/billing-gcp-commitments



With a commitment, you sign up for a minimum spend amount and get a discount on your committed usage of StreamNative Cloud. All usage, including overage, is [list price](https://www.suger.io/docs/gcp-marketplace/private-offer/#commit-discount-with-additional-usage-at-list-price). So, if you spend beyond the commit, you actually pay the list price for the additional usage.

This document describes how to get up and running with StreamNative Cloud on Google Cloud Marketplace with usage-based billing commitment.

## Prerequisites

To use this service, you must complete the following:

* Have a Google Cloud account with an account ID enabled for purchases. Contact your billing administrator if you have questions about your Google Cloud account.
* Create a StreamNative Cloud account by following the sign-up of the [StreamNative Cloud Quick Start](/cloud/get-started/quickstart-console#sign-up).
* Log in to both [Google Cloud Marketplace](https://console.cloud.google.com/marketplace) and [StreamNative Cloud Console](https://console.streamnative.cloud/?defaultMethod=signup).

## Procedures

You will receive an email from StreamNative sales after you purchase a Google Cloud Marketplace private offer with StreamNative. There is a link embedded in your email that opens an offer page in Google Cloud Marketplace.

### Review and accept your private offer

1. Click the embedded link in your email and you will be automatically redirected back to the Google Cloud Marketplace.
2. Review the offer details, pricing information, usage metric charges, policies, and terms of use, check the **I agree to the terms above** checkbox, and then click **ACCEPT**. A dialog box displays.
3. Click **MANAGE ON PROVIDER** to be automatically redirected back to the StreamNative Cloud Console. This step also connects your Google Cloud Marketplace account to your StreamNative Cloud account.

   <img alt="a screenshot of managing on provider on GCP" />

### Create your organization

<Note title="Note">
  Currently, you cannot link your GCP entitlement to an existing organization on StreamNative Console. To bind to an existing organization, [submit a ticket](https://support.streamnative.io/hc/en-us/requests/new) with this request to the support team.
</Note>

Depending on your StreamNative Cloud account setup, do one of the followings:

* If you have not created an organization yet, enter a name for your first organization and follow the prompts to complete the setup of your organization.

  <img alt="a screenshot of creating an organization for GCP" />

* If you have already created at least one organization, on the **Organization** page, click **Create organization**, enter a name for your new organization, and click **Create**.

### Create your Pulsar instance and Pulsar cluster

<Note title="Note">
  For BYOC users, contact your StreamNative sales to get your BYOC pool members provisioned before continuing to create your Pulsar instance and cluster.
</Note>

1. Navigate to the organization that you just created.

2. Click the name of your new organization, and then click **create instance** at the end of the row of the organization that you have just created.

3. On the **Instance** page, click **CREATE INSTANCE**.

4. Click **Deploy Dedicated** to start the instance creation process on StreamNative Cloud. Alternatively, you can click **Deploy Serverless** to create a Serverless cluster or click **Deploy BYOC** to create a BYOC cluster.

5. On the **Instance Configuration** page, enter a name for your instance, select the Google Cloud infrastructure pool, and select the multi Availability Zone (AZ). The instance name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

6. Click **Cluster Location** to start the cluster creation process.

7. On the **Cluster Location** page, enter a name for your cluster, select the cluster location, and then click **Cluster Size**. The cluster name starts with a lowercase letter, contains any combination of lowercase letters (a-z), numbers (0-9), and hyphens (-), and must be 4-10 characters.

8. On the **Cluster Size** page, configure the cluster, and then click **Payment**.

   * On the **Basic** tab, select custom sizing options.

   * On the **Advanced** tab, in the **Features** area, enable the cluster features you want on your cluster.

It may take several minutes for the instance and cluster creation process to complete.

## Related topics

* Learn more about [StreamNative Cloud](/cloud/overview/cloud-overview).
* Learn more about [StreamNative Cloud Billing Overview](/cloud/billing/billing-overview).
* [Get started with StreamNative Cloud on Google Cloud Marketplace with Pay-As-You-Go](/cloud/billing/billing-gcp).
* [Stop charges for your Pulsar cluster](/cloud/billing/stop-charges).


# StreamNative Cloud Billing
Source: https://docs.streamnative.io/cloud/billing/billing-overview



StreamNative Cloud bills are based on the consumption of resources within your cloud organization. Billing dimensions vary by cluster type — Serverless, Dedicated Kafka, Dedicated Pulsar, and BYOC clusters each use different capacity planning and charging models.

Billing for each StreamNative Cloud component accrues at hourly intervals. Any usage of less than an hour is billed for the entire hour. All billing computations are conducted in Coordinated Universal Time (UTC).

<Note title="Note">
  Billing accrues hourly, with a monthly-in-arrears invoicing cycle. If you de-provision resources that have accrued billed usage during the current month, billing will no longer accrue for these resources, but the billed usage accrued so far in the invoicing cycle will appear on your next invoice.
</Note>

## Subscription plan

A subscription is an agreement between you and StreamNative to pay for service on a particular schedule. In the current release, when you create an instance and a cluster on the StreamNative Cloud Console, you are automatically enrolled in the default Pay-As-You-Go subscription plan. For customers who have legacy clusters, [submit a ticket](https://support.streamnative.io/hc/en-us/requests/new) to get assistance with moving your cluster to the updated subscription plan.

If you want to provision a cluster with `snctl` instead of on StreamNative Cloud Console, you'll need to first create a subscription with `snctl`. For more information about `snctl`, see the [StreamNative CLI (snctl)](/tools/cli/snctl/snctl-overview).

For more information about viewing your invoices, what resources are included on your invoice, how to update your payment information, and more, see the [billing documentation page](/cloud/billing/billing).

## Billing dimensions

StreamNative Cloud offers [multiple](/cloud/clusters/cluster-types) cluster types. The billing dimensions vary by cluster type. The following table summarizes which capacity planning and charging models apply to each cluster type.

| Cluster Type                       | Capacity Planning                           | Charging Unit                                                   | Charging Model                               |
| ---------------------------------- | ------------------------------------------- | --------------------------------------------------------------- | -------------------------------------------- |
| Serverless                         | Auto-scaled (ETU)                           | [ETU](#elastic-throughput-unit-etu) (Elastic Throughput Unit)   | Elastic — pay for actual usage               |
| Dedicated Kafka *(Public Preview)* | [TU](#throughput-unit-tu) (Throughput Unit) | [RTU](#reserved-throughput-unit-rtu) (Reserved Throughput Unit) | Reserved — pay for configured capacity       |
| Dedicated Pulsar                   | [CU + SU](#cu-and-su)                       | CU + SU                                                         | Resource-based — pay for allocated resources |
| BYOC / BYOC Pro (Kafka clusters)   | [TU](#throughput-unit-tu) (Throughput Unit) | [ETU](#elastic-throughput-unit-etu) (Elastic Throughput Unit)   | Elastic — pay for actual usage               |
| BYOC / BYOC Pro (Pulsar clusters)  | [CU + SU](#cu-and-su)                       | [ETU](#elastic-throughput-unit-etu) (Elastic Throughput Unit)   | Elastic — pay for actual usage               |

<Note title="Important">
  StreamNative storage and throughput are calculated in binary gigabytes (GB), where 1 GB is 2^30 bytes. This unit of measurement is also known as a gibibyte (GiB).
</Note>

### Serverless clusters

Serverless clusters adopt a throughput-based pricing model. The following table summarizes the billing dimensions for Serverless clusters.

| Dimension                     | Unit of measure             |
| ----------------------------- | --------------------------- |
| Elastic Throughput Unit (ETU) | Cost per ETU per hour       |
| Ingress (Data In)             | Cost per GB                 |
| Egress (Data Out)             | Cost per GB                 |
| Storage (Data Stored)         | Cost per GB stored per hour |

### Dedicated Kafka clusters (Public Preview)

<Note title="Public Preview">
  Dedicated Kafka clusters are currently in public preview. Pricing for billing dimensions other than RTU is pending and subject to change during the preview phase.
</Note>

Dedicated Kafka clusters adopt a throughput-based pricing model using Reserved Throughput Units (RTUs). You configure the cluster capacity in Throughput Units (TUs) and are charged for the number of TUs reserved, regardless of actual usage. Reserved Throughput Pricing applies to both the **Latency-Optimized** profile (disk-based storage) and the **Cost-Optimized** profile (diskless architecture backed by object storage), so you get predictable pricing on either profile while choosing the storage architecture that best fits your workload. The following table summarizes the billing dimensions for Dedicated Kafka clusters.

| Dimension                      | Unit of measure             |
| ------------------------------ | --------------------------- |
| Reserved Throughput Unit (RTU) | Cost per RTU per hour       |
| Ingress (Data In)              | Cost per GB                 |
| Egress (Data Out)              | Cost per GB                 |
| Storage - Latency Optimized    | Cost per GB stored per hour |
| Storage - Cost Optimized       | Cost per GB stored per hour |

### Dedicated Pulsar clusters

Dedicated Pulsar clusters adopt a resource-based pricing model. The following table summarizes the billing dimensions for Dedicated Pulsar clusters.

| Dimension             | Unit of measure             |
| --------------------- | --------------------------- |
| Compute Unit (CU)     | Cost per CU per hour        |
| Storage Unit (SU)     | Cost per SU per hour        |
| Ingress (Data in)     | Cost per GB                 |
| Egress (Data out)     | Cost per GB                 |
| Storage (Data stored) | Cost per GB stored per hour |

### BYOC & BYOC Pro clusters

BYOC & BYOC Pro clusters adopt a throughput-based pricing model and are charged based on actual throughput usage in Elastic Throughput Units (ETUs). How you configure the cluster's reserved capacity depends on the cluster type:

* **Kafka clusters**: You configure the reserved capacity in Throughput Units (TUs).
* **Pulsar clusters**: You configure the reserved capacity using Compute Units (CUs) and Storage Units (SUs).

Regardless of how capacity is configured, all BYOC clusters are charged by ETU based on actual usage. The following table summarizes the billing dimensions for BYOC & BYOC Pro clusters.

| Dimension                     | Unit of measure       |
| ----------------------------- | --------------------- |
| Elastic Throughput Unit (ETU) | Cost per ETU per hour |

Note: The Latency Optimized Clusters and Cost Optimized Clusters adopt the same throughput-based pricing model.

### Throughput Unit (TU)

<span />

A Throughput Unit (TU) is a capacity planning abstraction that represents a standardized amount of throughput capacity. TUs allow you to configure cluster capacity without managing infrastructure details such as CPU, memory, or broker counts.

Each TU represents the following capacity:

| Dimension         | Capacity per TU                |
| ----------------- | ------------------------------ |
| Ingress (Data In) | 25 megabytes per second (MBps) |
| Egress (Data Out) | 75 megabytes per second (MBps) |
| Data Entries      | 2,500 entries per second       |

TUs are used for capacity planning in **Dedicated Kafka** clusters and **BYOC / BYOC Pro Kafka** clusters. How TUs are charged depends on the cluster type:

* **Reserved Throughput Unit (RTU)**: For Dedicated Kafka clusters, you are charged for the number of TUs you reserve when configuring the cluster. This is a fixed charge regardless of actual usage. See [RTU](#rtu) for details.
* **Elastic Throughput Unit (ETU)**: For BYOC and BYOC Pro Kafka clusters, you configure TUs as the reserved capacity, but you are charged based on actual throughput usage. See [ETU](#etu) for details.

<Note title="Note">
  BYOC Pulsar clusters use CU/SU for capacity planning instead of TUs, but are still charged by ETU based on actual usage.
</Note>

### Elastic Throughput Unit (ETU)

<span />

Elastic Throughput Units (ETUs) are the basis for billing clusters that adopt a throughput-based pricing model, including Serverless clusters and BYOC (and BYOC Pro) clusters in StreamNative Cloud. ETUs provide a flexible, scalable approach to resource allocation and pricing.

#### How ETUs Work

Clusters adopting ETUs are elastic and have a minimum number of ETUs. They can automatically scale up to a maximum capacity based on your workload. The minimum ETUs define the baseline capacity of your cluster, which is the minimum amount you are billed for even when the cluster is idle. The maximum capacity is defined in terms of ETUs and governs the peak resources your StreamNative cluster can use. However, you are only billed for the actual capacity used in a given hour, up to this maximum.

To determine the number of ETUs used in a given hour, the billing system monitors the actual consumption across several dimensions:

1. Ingress (Data In): This dimension measures the volume of data being written to your cluster. It is calculated as the total amount of data (in bytes) ingested into the cluster per second.

2. Egress (Data Out): This dimension measures the volume of data being read from your cluster. It is calculated as the total amount of data (in bytes) retrieved from the cluster per second.

3. Data Entries: This dimension counts the number of entries processed by the cluster per second. An entry represents a collection of messages batched together by either Pulsar or Kafka client. It includes both produce and consume operations.

If your cluster has zero consumption across all these dimensions, you will be billed for the minimum ETUs. This occurs when your cluster has no partitions, no topics have been created (or all topics have been deleted), and there is no capacity usage from any ETU-eligible dimension.

#### Serverless ETU Capacity and Limits

Each Serverless ETU represents a certain capacity across different dimensions. Here's a breakdown of what one Serverless ETU provides:

| Dimension         | ETU Capacity                   |
| ----------------- | ------------------------------ |
| Ingress (Data In) | 5 megabytes per second (MBps)  |
| Egress (Data Out) | 15 megabytes per second (MBps) |
| Data Entries      | 500 entries per second         |

The maximum capacity for a Serverless cluster is currently set at 20 ETUs, which translates to the following limits:

| Dimension         | Maximum Capacity  |
| ----------------- | ----------------- |
| Ingress (Data In) | 100 MBps          |
| Egress (Data Out) | 300 MBps          |
| Data Entries      | 10,000 per second |

The minimum number of ETUs per Serverless cluster is 1.

#### BYOC ETU Capacity and Limits

The following outlines what is included in one ETU

| Dimension         | ETU Capacity                   |
| ----------------- | ------------------------------ |
| Ingress (Data In) | 25 megabytes per second (MBps) |
| Egress (Data Out) | 75 megabytes per second (MBps) |
| Data Entries      | 2500 entries per second        |

The maximum capacity for a BYOC (and BYOC Pro) clusters is unlimited. Your actual throughput is capped by the resources you use for running brokers.

The minimum ETUs for a BYOC (and BYOC Pro) clusters is 1 ETU.

#### Determining ETU Usage

To estimate your ETU usage, you can monitor your cluster's performance across the three dimensions mentioned above. Your ETU consumption for a given hour will be based on the highest usage across these dimensions, relative to the capacity provided by one ETU.

For example, if in one hour your Serverless cluster uses:

* 20 MBps ingress (4 ETUs worth)
* 45 MBps egress (3 ETUs worth)
* 1,500 data entries per second (3 ETUs worth)

Your usage for that hour would be billed as 4 ETUs, based on the highest consumption dimension (ingress in this case).

Another example is if your BYOC cluster uses:

* 100 MBps ingress (4 ETUs worth)
* 100 MBps egress (1 ETUs worth)
* 12,500 data entries per second (5 ETUs worth)

Your usage for that hour would be billed as 5 ETUs, based on the highest consumption dimension (data entries in this case).

By understanding and monitoring these metrics, you can effectively optimize your resource usage and control costs in StreamNative Cloud.

### Reserved Throughput Unit (RTU)

<span />

<Note title="Public Preview">
  Dedicated Kafka clusters and RTU-based billing are currently in public preview. Pricing details may change during the preview phase.
</Note>

Reserved Throughput Units (RTUs) are the basis for billing Dedicated Kafka clusters in StreamNative Cloud. Unlike Elastic Throughput Units (ETUs), RTUs represent reserved capacity — you are charged for the number of TUs you configure, regardless of actual usage.

#### How RTUs Work

When you create or resize a Dedicated Kafka cluster, you specify the number of Throughput Units (TUs) to reserve. Each reserved TU is billed as one RTU. Your cluster is provisioned with the corresponding capacity, and you are charged for the reserved amount at a fixed hourly rate.

RTUs abstract away infrastructure details. You do not need to configure brokers, CPUs, or memory — you simply specify the desired throughput capacity in TUs.

#### RTU Capacity

Each RTU provides the same capacity as one [Throughput Unit (TU)](#tu):

| Dimension         | Capacity per RTU               |
| ----------------- | ------------------------------ |
| Ingress (Data In) | 25 megabytes per second (MBps) |
| Egress (Data Out) | 75 megabytes per second (MBps) |
| Data Entries      | 2,500 entries per second       |

#### RTU Limits

* **Minimum**: 1 RTU per cluster
* **Self-service maximum**: 20 RTUs per cluster (configurable via the Cloud Console)
* **Beyond 20 RTUs**: [Contact StreamNative sales](https://www.streamnative.io/contact) for larger configurations

#### RTU Scaling

You can adjust the number of RTUs for your Dedicated Kafka cluster using the TU slider in the StreamNative Cloud Console. RTU values are integers (whole numbers only).

<Note title="Important">
  Changing the number of RTUs triggers a cluster reconfiguration. Allow sufficient time between scaling operations for the reconfiguration to complete.
</Note>

### CU and SU

<span />

Dedicated Pulsar clusters (formerly Hosted) are scaled using Compute Units (CUs) and Storage Units (SUs). These units provide a standardized way to allocate and manage resources for your Dedicated Pulsar clusters.

<Note title="Note">
  CU and SU apply to Dedicated Pulsar clusters only. For Dedicated Kafka clusters, see [Reserved Throughput Unit (RTU)](#rtu).
</Note>

#### Compute Units (CUs)

* **Definition**: One CU represents a unit of compute resources, specifically 2 CPUs and 8 GB RAM.
* **Usage**: CUs are used to scale all stateless components in your cluster, primarily the brokers.
* **Scaling**: You can adjust the number of CUs to increase or decrease the processing power of your cluster.

#### Storage Units (SUs)

* **Definition**: One SU represents a unit of storage resources, specifically 2 CPUs, 8 GB RAM, and 1 TB disk.
* **Usage**: SUs are used to scale all stateful components in your cluster, including BookKeeper and ZooKeeper.
* **Scaling**: Adjusting the number of SUs allows you to manage the storage capacity and performance of your cluster.

#### Billing and Resource Allocation

The number of CUs and SUs used by a StreamNative cluster determines the total resources allocated. Charges for CUs and SUs accrue each hour based on the allocated resources. You can expand or shrink your cluster by adding or removing CUs and SUs. When you modify the cluster capacity, you are billed for the new resource allocation starting from the next hour following the change.

#### Limits per CU and SU

CUs and SUs determine the capacity of a StreamNative cluster. For a StreamNative Cloud cluster, the expected performance for any given workload is dependent on a variety of dimensions, such as message size and number of partitions.

Use the following guidelines to determine the minimum number of CUs and SUs to use for a given workload, how to monitor a dimension, and suggestions to reduce your use of a particular dimension.

<Note title="Important">
  These dimensions provide guidelines for capacity planning. The ability to fully utilize these dimensions depends on the workload and utilization of other dimensions.
</Note>

**Recommended guideline for a broker CU**

| Dimension                   | Guideline per CU  | Details                                                                                                                                                                                                                                                                                 |
| --------------------------- | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Ingress                     | 50 MBps           | Number of bytes that can be produced to the cluster in one second. To reduce usage on this dimension, you can compress your messages. `lz4` is recommended for compression.                                                                                                             |
| Egress                      | 150 MBps          | Number of bytes that can be consumed from the cluster in one second. To reduce usage on this dimension, you can compress your messages and ensure each consumer is only consuming from the topics it requires. `lz4` is recommended for compression.                                    |
| Requests (aka Data Entries) | 10,000 per second | Number of data entries (each entry is a batch of messages batched at the client side) that can be produced to and consumed from the cluster in one second. To reduce usage on this dimension, you can adjust producer batching configurations and shut down otherwise inactive clients. |

**Recommended guideline for a bookie SU**

A **SU** can support a maximum post-replication write throughput of 125 MBps. However, the actual throughput varies based on different factors including data entry size, data entry rate, etc. To attain the peak post-replication write throughput of 125 MBps, clients must efficiently batch their requests.

### Billing model comparison

StreamNative Cloud uses three billing models depending on your cluster type. The following table compares the key differences.

| Aspect            | ETU (Elastic Throughput Unit)                                                 | RTU (Reserved Throughput Unit)     | CU/SU (Compute/Storage Unit)    |
| ----------------- | ----------------------------------------------------------------------------- | ---------------------------------- | ------------------------------- |
| Applies to        | Serverless, BYOC / BYOC Pro                                                   | Dedicated Kafka *(Public Preview)* | Dedicated Pulsar                |
| Capacity planning | Auto-scaled (Serverless), TU-based (BYOC Kafka), or CU/SU-based (BYOC Pulsar) | TU-based                           | CU + SU based                   |
| Charging          | Actual usage (elastic)                                                        | Reserved capacity (fixed)          | Allocated resources (fixed)     |
| Scaling           | Automatic                                                                     | Manual (1-20 self-service)         | Manual                          |
| Minimum billing   | 1 ETU                                                                         | 1 RTU                              | Varies by cluster configuration |

<Note title="Note">
  Organizations created prior to February 6th, 2026 still use CU/SU-based pricing for BYOC clusters. All BYOC clusters created on or after this date use ETU-based pricing.
</Note>

#### Dedicated Kafka clusters (RTU)

With Dedicated Kafka clusters, you configure the cluster capacity in Throughput Units (TUs) and are charged for the number of TUs reserved as RTUs. Billing is at a fixed hourly rate regardless of actual usage. You do not need to manage infrastructure details — capacity is expressed entirely in TUs.

#### Dedicated Pulsar clusters (CU/SU)

With Dedicated Pulsar clusters, you are billed based on the number of CUs and SUs allocated to your cluster. You determine the capacity at creation by specifying the number of brokers and bookies. If your cluster uses less capacity than what you configure, you still pay the same amount.

#### Serverless clusters (ETU)

Serverless clusters are elastic. You provision a cluster with a fixed maximum capacity, but billing is based on actual usage, subject to a minimum of 1 ETU, even when no workloads are actively running. Costs scale up only as consumption increases and will never exceed the configured maximum capacity.

#### BYOC and BYOC Pro clusters (ETU)

All BYOC clusters are charged by ETU based on actual throughput consumption. A minimum charge of 1 ETU applies even when no workloads are running. As consumption increases, billing scales with actual throughput usage. Your cluster has no predefined maximum capacity.

Capacity planning differs by cluster type: BYOC Kafka clusters use Throughput Units (TUs) to configure reserved capacity, while BYOC Pulsar clusters use Compute Units (CUs) and Storage Units (SUs).

### Ingress and Egress

StreamNative Cloud charges for data transfer in two directions:

1. **Ingress**: Data transferred into your Serverless and Dedicated clusters.
2. **Egress**: Data transferred out of your Serverless and Dedicated clusters.

These charges apply to all network traffic, including:

* Data produced to topics
* Data consumed from topics
* Replication traffic between clusters
* Admin operations (e.g., topic creation, subscription creation)

Billing for ingress and egress is based on the total volume of data transferred, measured in gigabytes (GB). The rates may differ for ingress and egress, so it's important to monitor both metrics separately.

To optimize your costs:

* Use compression for message payloads
* Implement efficient batching strategies
* Minimize unnecessary data transfers

<Note title="Important">
  For BYOC (Bring Your Own Cloud) clusters, StreamNative Cloud does not charge for data transfers. These resources are hosted and billed directly by your cloud provider, so you'll need to refer to your provider's pricing for these costs.
</Note>

### Storage

StreamNative Cloud charges for the total volume of data stored in your Serverless and Dedicated clusters. Here are key points to understand about storage billing:

1. **Pre-replication volume**: Billing is based on the pre-replication volume of data. This means you're charged for the actual data you store, not the replicated copies.

2. **Replication factor**: StreamNative Cloud uses a built-in replication factor of 3 for high availability. While this triples the actual storage used, you're only billed for the pre-replication volume.

3. **Billing calculation**: Storage is billed based on the average volume of data stored over each hour, measured in gigabytes (GB).

4. **Retention policy**: To optimize storage costs, you can configure retention policies for your topics. These policies can automatically delete data after a specified period or when certain size limits are reached.

5. **Compaction**: For topics that only need to retain the latest value for each key, you can use compaction to reduce storage usage while preserving the most recent updates.

<Note title="BYOC clusters">
  For Bring Your Own Cloud (BYOC) clusters, StreamNative Cloud does not charge for data storage. These resources are hosted and billed directly by your cloud provider.
</Note>

To monitor and manage your storage usage, use the StreamNative Cloud Console or API to track storage metrics and adjust your data retention strategies as needed.

### Functions and Connectors

StreamNative Cloud charges for functions and connectors based on the Function Processing Units (FPUs) they consume. This pricing model applies to all Pulsar cluster types: Serverless, Dedicated, and BYOC (including BYOC Pro).

An FPU represents a unit of compute resources, specifically 2 CPUs and 8 GB of RAM, dedicated to running functions or connectors. The charge for functions and connectors is based on the number of FPUs allocated and the duration of their usage.

Key points about function and connector billing:

1. Consistent across cluster types: The FPU-based billing model is uniform across all Pulsar cluster types, ensuring consistency in how you're charged for these components.

2. Resource allocation: When you deploy a function or connector, you specify the resources it requires. StreamNative Cloud then allocates the appropriate number of FPUs based on this specification.

3. Usage-based billing: You are billed for the FPUs allocated to your functions and connectors for as long as they are running. This means you only pay for the resources you use.

4. Scalability: If your functions or connectors auto-scale, the billing automatically adjusts to reflect the increased or decreased FPU usage.

5. Monitoring and optimization: You can monitor your function and connector usage in the StreamNative Cloud Console to optimize your resource allocation and manage costs effectively.

By using this FPU-based model, StreamNative Cloud provides a flexible and transparent way to charge for function and connector usage, regardless of the underlying Pulsar cluster type you're using.

### Consumption Units and Support Units

All StreamNative Cloud billing dimensions are calculated in StreamNative **Consumption Units**. Your overall charge for StreamNative Cloud usage will be equal to the total number of Consumption Units multiplied by the cost of that unit.

The support plan purchased from StreamNative is also based on the **Support Units**. Your overall support cost will be the total number of Support Units multiplied by the cost of that unit.

| Unit                          | Price | Description                                                                                                                                                                                                                                    |
| ----------------------------- | ----- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| StreamNative Consumption Unit | \$0.1 | A Consumption Unit captures your StreamNative Cloud usage based on underlying metrics such as Compute Units (CUs), Storage Units (SUs), Reserved Throughput Units (RTUs), Elastic Throughput Units (ETUs), data stored, data in, and data out. |
| StreamNative Support Unit     | \$0.1 | A Support Unit captures your StreamNative Support costs based on your support plan. The support plan is a separate and optional purchase item.                                                                                                 |

### Pricing dimensions

This section describes pricing dimensions for Serverless, Dedicated Kafka, Dedicated Pulsar, and BYOC clusters.

#### Pricing dimensions for Serverless clusters

The following table outlines the prices for usage dimensions for Serverless clusters.

| Dimension                               | Consumption Units                 | Price  |
| --------------------------------------- | --------------------------------- | ------ |
| Elastic Throughput Unit (ETU) (\$/hour) | 1 ETU/hour = 1 Consumption Unit   | \$0.10 |
| Ingress (Data In) (\$/GB)               | 1 GB = 1.3 Consumption Units      | \$0.13 |
| Egress (Data Out) (\$/GB)               | 1 GB = 0.4 Consumption Unit       | \$0.04 |
| Storage (Data Stored) (\$/GB-Month)     | 1 GB-Month = 0.9 Consumption Unit | \$0.09 |

#### Pricing dimensions for Dedicated Kafka clusters (Public Preview)

<Note title="Public Preview">
  Dedicated Kafka clusters are in public preview. Pricing is subject to change during the preview phase.
</Note>

The following table outlines the prices for usage dimensions for Dedicated Kafka clusters.

| Dimension                                 | Consumption Units                  | Price  |
| ----------------------------------------- | ---------------------------------- | ------ |
| Reserved Throughput Unit (RTU) (\$/hour)  | 1 RTU/hour = 7.5 Consumption Units | \$0.75 |
| Ingress (Data In) (\$/GB)                 | 1 GB = 1.3 Consumption Units       | \$0.13 |
| Egress (Data Out) (\$/GB)                 | 1 GB = 0.4 Consumption Unit        | \$0.04 |
| Storage - Latency Optimized (\$/GB-Month) | 1 GB-Month = 0.9 Consumption Unit  | \$0.09 |
| Storage - Cost Optimized (\$/GB-Month)    | 1 GB-Month = 0.4 Consumption Unit  | \$0.04 |

#### Pricing dimensions for Dedicated Pulsar clusters

The following table outlines the prices for usage dimensions for Dedicated Pulsar (formerly Hosted) clusters.

| Dimension                           | Consumption Units                 | Price  |
| ----------------------------------- | --------------------------------- | ------ |
| Compute Unit (CU) (\$/hour)         | 1 CU/hour = 2.4 Consumption Units | \$0.24 |
| Storage Unit (SU) (\$/hour)         | 1 SU/hour = 3 Consumption Units   | \$0.30 |
| Ingress (Data In) (\$/GB)           | 1 GB = 1.3 Consumption Units      | \$0.13 |
| Egress (Data Out) (\$/GB)           | 1 GB = 0.4 Consumption Unit       | \$0.04 |
| Storage (Data Stored) (\$/GB-Month) | 1 GB-Month = 0.9 Consumption Unit | \$0.09 |

#### Pricing dimensions for BYOC clusters

The following table outlines the prices for usage dimensions for BYOC clusters.

| Dimension                               | Consumption Units               | Price  |
| --------------------------------------- | ------------------------------- | ------ |
| Elastic Throughput Unit (ETU) (\$/hour) | 1 ETU/hour = 5 Consumption Unit | \$0.50 |

You can [contact StreamNative sales](https://www.streamnative.io/contact) to get a quote for BYOC Pro clusters.

#### Pricing dimensions for Functions and Connectors

The following table outlines the prices for usage dimensions for Functions and Connectors.

| Dimension                 | Serverless | Dedicated | BYOC   |
| ------------------------- | ---------- | --------- | ------ |
| Functions (FPU) (\$/hour) | \$0.18     | \$0.18    | \$0.10 |

## Pricing models

You can either pay as you go or make an annual commitment. Discounts based on usage are available with annual commitments.

### Annual commitments

StreamNative Cloud offers the ability to make a commitment to a minimum amount of spend over a specified time period. This commitment gives you access to discounts and allows you to use this commitment across the entire StreamNative Cloud stack, including any [StreamNative cluster type](/cloud/clusters/cluster-types), connectors, functions, and support.

If you use more than your committed amount, you can continue using StreamNative Cloud without interruption. You will be charged at the on-demand price for usage beyond the committed amount until the end of your commitment term. Commitments are minimums, and there is no negative impact to exceeding your committed usage. If you exceed this minimum, overage charges will be billed to the payment method set for your organization.

[Contact StreamNative](https://www.streamnative.io/contact) to learn more about annual commitments, or review these topics.

* [Get Started with StreamNative Cloud on the AWS Marketplace with Commitments](/cloud/billing/billing-aws-commitments)
* [Get Started with StreamNative Cloud on the Google Cloud Marketplace with Commitments](/cloud/billing/billing-gcp-commitments)

### Pay-As-You-Go

With the Pay-As-You-Go pricing model, you can sign up and pay monthly in arrears. If you sign up for StreamNative Cloud directly through [StreamNative](https://console.streamnative.cloud/), your organization will be on the Pay-As-You-Go billing model by default.

The Pay As You Go billing model is also available on the Cloud Marketplace channels. For more information on the cloud provider Marketplace integrations, see:

* [Get Started with StreamNative Cloud on the AWS Marketplace with Pay-As-You-Go](/cloud/billing/billing-aws)
* [Get Started with StreamNative Cloud on the Google Cloud Marketplace with Pay-As-You-Go](/cloud/billing/billing-gcp)

## Pay-As-You-Go billing schedule

This section describes the billing schedule for the Pay-As-You-Go pricing model.

### Monthly, billed by StreamNative

When you sign up for StreamNative Cloud service by [adding your credit card details](/cloud/billing/billing#change-your-payment-method) in the StreamNative Cloud Console, you are billed monthly.

At each billing cycle, on the first day of each month, all usage for the previous month is aggregated, invoiced, and charged in arrears on the credit card used to sign up for the service.

All usage is expressed and charged in US dollars only.

### Monthly, billed through Marketplace

<Note title="Note">
  * Typically, marketplaces invoice you in arrears on the first day of each month. However, there are exceptions, such as in the case of the [Google Cloud billing cycle](https://cloud.google.com/billing/docs/how-to/billing-cycle).
  * AWS marketplace only accepts integer usage. If your usage is smaller than the billing unit, the user is still charged at the billing unit price.
</Note>

You can sign up for StreamNative Cloud service through Cloud Marketplace channels. In this case, all usage is reported hourly to the marketplace.

At the marketplace's billing cycle, all usage is aggregated and charged as part of your cloud provider bill.

StreamNative Cloud service usage is a single invoice line with the total amount charged.

## Pricing examples

This section provides pricing examples for different cluster types, functions, and connectors.

<Note title="Note">
  The examples below estimate service costs based on a normalized monthly time frame. It assumes there are 730 hours in a month ((365 days \* 24 hours) / 12 months in a year). The actual hours in a given billing period will vary slightly.
</Note>

### Pricing example for a Dedicated Kafka cluster (Public Preview)

<Note title="Public Preview">
  Dedicated Kafka clusters are in public preview. Pricing is subject to change during the preview phase.
</Note>

Suppose you have a Dedicated Kafka cluster on StreamNative Cloud, running an entire month, with 5 RTUs reserved and the following usage:

* 100 GB data in

* 100 GB data out

* 500 GB data retained (Latency Optimized)

* RTU: The total Consumption Units are `5 RTUs x 7.5 Consumption Units / RTU-hour x 730 hours = 27,375 Consumption Units`

* Data in: The total Consumption Units are `100 GB x 1.3 Consumption Units / GB = 130 Consumption Units`

* Data out: The total Consumption Units are `100 GB x 0.4 Consumption Unit / GB = 40 Consumption Units`

* Storage (Latency Optimized): The total Consumption Units are `500 GB-Month x 0.9 Consumption Unit / GB-Month = 450 Consumption Units`

Therefore, the total cost of this cluster will be `27,375 + 130 + 40 + 450 = 27,995` Consumption Units x `$0.10` per Consumption Unit = `$2,799.50` per month.

### Pricing example for a Dedicated Pulsar cluster

<Note title="Note">
  On StreamNative Cloud, a minimum Dedicated Pulsar cluster consists of 2 brokers and 3 bookies. Each broker consumes 0.5 compute units per month and each bookie consumes 0.5 storage units per month. Therefore, the minimum cost for a Dedicated Pulsar cluster is about `$505` per month.
</Note>

Suppose you have a Dedicated Pulsar cluster on StreamNative Cloud, running an entire month, with the following details:

* 3 brokers, each with 2 CUs

* 3 bookies, each with 2 SUs

* 200 GB data in

* 200 GB data out

* 200 GB data retained

* CU: The total Consumption Units are `6 CUs x 2.4 Consumption Units / CU-hour x 730 hours = 10,512 Consumption Units`

* SU: The total Consumption Units are `6 SUs x 3 Consumption Units / SU-hour x 730 hours = 13140 Consumption Units`

* Data in: The total Consumption Units are `200 GB x 1.3 Consumption Units / GB = 260 Consumption Units`

* Data out: The total Consumption Units are `200 GB x 0.4 Consumption Unit / GB = 80 Consumption Units`

* Data stored: The total Consumption Units are `200 GB-Month x 0.9 Consumption Unit / GB-Month = 180 Consumption Units`

Therefore, the total cost of this cluster will be `24,172` Consumption Units x `$0.10` per Consumption Unit = `$2417.2`.

### Pricing example for a Serverless cluster

<Note title="Note">
  Serverless clusters are billed based on actual usage of resources, without the need to provision specific compute or storage units upfront.
</Note>

Suppose you have a Serverless cluster on StreamNative Cloud, running for an entire month, with the following usage:

* Average write throughput (Ingress): 1 MBps
* Average read throughput (Egress): 3 MBps
* Average entry size (batch size at the client side): 64 KB
* Retention: 7 days

So:

* the total Data-Stored for 7-days retention is `1 MBps x 60 x 60 x 24 x 7 = 604,800 MB = 604,800 / 1024 = 590.625 GB`.
* the total Data-In in a month is `1 MBps x 60 x 60 x 730 = 2,628,000 MB = 2,628,000 / 1024 = 2,566.21 GB`.
* the total Data-Out in a month is `3 MBps x 60 x 60 x 730 = 7,884,000 MB = 7,884,000 / 1024 = 7,699.22 GB`.

Let's calculate the total ETUs used by this cluster.

* ETUs by ingress: `1 MBps / 5 MBps/ETU = 0.2 ETUs`
* ETUs by egress: `3 MBps / 15 MBps/ETU = 0.2 ETUs`
* ETUs by data entries: `(1 MBps + 3 MBps) / 64 KB/entry / 500 entries/ETU = 0.128 ETUs`

So the total ETUs used by this cluster is `max(0.2, 0.2, 0.128) = 0.2 ETUs`.

Let's calculate the total consumption units used by this cluster.

* ETU: The total Consumption Units are `0.2 ETUs x 1 Consumption Unit / ETU-hour x 730 hours = 146 Consumption Units`
* Data in: The total Consumption Units are `2,566.21 GB x 1.3 Consumption Units / GB = 3,336.073 Consumption Units`
* Data out: The total Consumption Units are `7,699.22 GB x 0.4 Consumption Unit / GB = 3,079.688 Consumption Units`
* Data stored: The total Consumption Units are `590.625 GB x 0.9 Consumption Unit / GB = 531.5625 Consumption Units`

Therefore, the total cost of this cluster will be `146 + 3,336.073 + 3,079.688 + 531.5625 = 7,093.3235` Consumption Units x `$0.10` per Consumption Unit = `$709.33`.

### Pricing example for a BYOC Cluster

<Note title="Note">
  BYOC Latency-Optimized Clusters and Cost-Optimized Clusters are billed based on actual throughput.
</Note>

Suppose you have a BYOC cluster on StreamNative Cloud, running for an entire month, with the following usage:

* Average write throughput (Ingress): 50 MBps
* Average read throughput (Egress): 150 MBps
* Average entry size (batch size at the client side): 64 KB
* Retention: 7 days

Let's calculate the total ETUs used by this cluster.

* ETUs by ingress: `50 MBps / 25 MBps/ETU = 2 ETUs`
* ETUs by egress: `150 MBps / 75 MBps/ETU = 2 ETUs`
* ETUs by data entries: `(50 MBps + 150 MBps) / 64 KB/entry / 2500 entries/ETU = 1.28 ETUs`

So the total ETUs used by this cluster is `max(2, 2, 1.28) = 2 ETUs`.

Let's calculate the total consumption units used by this cluster.

* ETU: The total Consumption Units are `2 ETUs x 5 Consumption Unit / ETU-hour x 730 hours = 7300 Consumption Units`

Therefore, the total cost of this cluster will be `7300` Consumption Units x `$0.10` per Consumption Unit = `$730`.

### Pricing Example for Pulsar Functions and Connectors

Pulsar functions and connectors are billed based on the [Function Processing Unit (FPU)](#usage-dimensions). You can convert the CPU and memory usage of your functions and connectors to FPUs based on the following formula:

`max((CPU / 2),(Mem (in GB) / 8))`

For example, if you specify 1 CPU and 1 GB memory for a function, the total FPU for this function is `max(1 / 2,1 / 8) = 0.5 CU`.

If you have a Dedicated cluster on StreamNative Cloud, running an entire month, with

* 32 functions, each with 0.5 CPU and 1 GB memory
* 20 connectors, each with 1 CPU and 1 GB memory

The total FPUs used by these functions and connectors are `32 x max((0.5 / 2), (1 / 8) ) + 20 x max((1 / 2),(1 / 8)) = 18 FPUs`.

The total Consumption Units used by these functions and connectors are `18 FPUs x 1.8 Consumption Unit / CU-hour x 730 hours = 23652 Consumption Units`.

Therefore, the total cost of these functions and connectors will be `23652` Consumption Units x `$0.10` per Consumption Unit = `$2365.20`.


# View Credits and Discounts
Source: https://docs.streamnative.io/cloud/billing/discounts



StreamNative may apply credits or discounts to your organization's account, such as welcome credits for new enterprise customers or promotional credits. These credits reduce your outstanding balance in Stripe and are reflected in your billing dashboard.

## View applied credits

To view credits applied to your organization, follow these steps.

1. On the StreamNative Cloud Console home page, select **Settings** under **Admin**.

   <img alt="a screenshot of navigating to the settings page" />

2. On the **Settings** page, click **Credits & Discounts**.

   <img alt="a screenshot of navigating to the discounts credits & discounts page" />

3. The **Credits & Discounts** table lists each credit applied to your organization with the following details.

   <img alt="a screenshot of credits & discounts page" />

   | Column            | Description                                          |
   | ----------------- | ---------------------------------------------------- |
   | **Date Applied**  | The date the credit was applied                      |
   | **Discount Name** | The credit name (for example, `welcome-credit-2026`) |
   | **Applied By**    | The StreamNative operator who applied the credit     |
   | **Amount**        | The credit value in USD (for example, `$500.00`)     |

## How credits affect your bill

Credits are applied as a **negative balance** against your Stripe account. StreamNative deducts available credits from your outstanding charges at the end of each billing cycle before charging your payment method.

For example, if your monthly invoice is $800 and you have a $500 credit, you are charged \$300 for that billing period. Credits reduce your balance until they are fully consumed.

## Frequently asked questions

### How do I request a credit for my organization?

Contact your StreamNative account representative or [submit a support ticket](https://support.streamnative.io/hc/en-us/requests/new) to request a credit.

### Can a credit be applied more than once?

No. Each named credit can only be applied once per organization.

### Are credits refundable?

Credits are non-refundable. If a credit was applied in error, contact StreamNative support. Note that the underlying Stripe transaction must be reversed manually by the StreamNative team.

### Where can I see credits on my invoice?

Credits appear as a negative line item on your Stripe invoice. You can download your invoice from the Stripe billing portal by navigating to the **Billing & Payment** page in the StreamNative Cloud Console.

### Can credits be applied to AWS, GCP, Azure, or any other marketplace offer?

Credits and discounts are currently available for Stripe customers. For other billing channels, [contact our support team](https://support.streamnative.io/hc/en-us/requests/new) to discuss your options.


# Stop charges for your cluster
Source: https://docs.streamnative.io/cloud/billing/stop-charges



If you no longer need a cluster and don’t want it to be charged for any longer, you can delete your clusters and instances using the StreamNative Cloud Console or the StreamNative Cloud CLI tool (snctl).

<Note title="Note">
  * Before deleting a cluster, ensure you have proper backup and disaster recovery procedures in place to minimize the risk of data loss or service disruption.
  * Billing for usage is by hour and any charges for usage before you deleted your cluster and instance will still appear on your next bill.
</Note>

## Step 1: Log in to the StreamNative Cloud Console

Navigate to the [StreamNative Cloud Console login page](https://console.streamnative.cloud/?defaultMethod=signup). Follow the prompts to log in to the StreamNative Cloud Console.

## Step 2: Delete your Pulsar cluster

<Warning title="Warning">
  After you delete your Pulsar cluster, all data is lost.
</Warning>

<Tabs>
  <Tab title="Delete a successfully-deployed Pulsar cluster">
    If your Pulsar cluster has been deployed successfully, follow these steps:

    1. On the left navigation pane, in the **Admin** area, click **Pulsar Clusters**.
    2. Select the **Details** tab.
    3. In the **Warning** section, click **Delete Cluster**. A dialog box displays asking, *Are you sure you want to delete this?*
    4. Enter the cluster name and then click **Confirm**.
  </Tab>

  <Tab title="Delete a Pulsar cluster that is being deployed">
    If your Pulsar cluster is being deployed, follow these steps:

    1. On the left navigation pane, click **Dashboard**.
    2. On the left navigation pane, in the **Admin** area, click **Pulsar Clusters**.
    3. In the **Warning** section, click **Delete Cluster**. A dialog box displays asking, *Are you sure you want to delete this?*
    4. Enter the cluster name and then click **Confirm**.
  </Tab>
</Tabs>

## Step 3: Delete your instance

1. On the left navigation pane, click **Dashboard**.
2. On the **Instances** card, click the **Setting** icon to list all the instances available for the organization.
3. Click the ellipsis at the end of the row of the instance that you want to delete, and then click **Delete**.
4. Enter the instance name and then click **Confirm**.

## Related topics

For details about how to delete your Pulsar cluster and instance using the StreamNative Cloud CLI tool (snctl), see the following topics:

* [Delete a Pulsar cluster](/tools/cli/snctl/snctl-tutorials#delete-a-cluster).
* [Delete a Pulsar instance](/tools/cli/snctl/snctl-tutorials#delete-an-instance).


# View your usage using StreamNative Cloud Console
Source: https://docs.streamnative.io/cloud/billing/view-usage-console



StreamNative Cloud provides real-time and historical data usage for all instances in your organization through the **Organization Usage** page on StreamNative Cloud Console, whether you subscribe to the StreamNative Cloud service through StreamNative website or Marketplace channels.

## View usage details

To view your real-time and historical data usage, follow these steps.

1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Organization Usage**.

   <img alt="gif of viewing your organization usage" />

2. Select the target instance and set the billing period.

   By default, all instances are selected and a 7-day billing period starting from the current date is set. You can view up to the past 60 days of usage based on the [usage dimensions](/cloud/billing/billing-overview#usage-dimensions).

   The metrics displayed vary based on your cluster profile type:

   ### Latency Optimized Clusters:

   * **CUs | Compute Units**: View how many CUs are committed by your Pulsar brokers or proxies. Clusters with Latency Optimized Profile use fixed resource allocation.
   * **SUs | Storage Units**: View how many SUs are committed for BookKeeper and ZooKeeper storage in your organization.
   * **Functions | Function Processing Units**: View how many FPUs are committed by your functions and connectors (only visible for function-enabled clusters).
   * **Throughput**: View the total amount (in gigabytes) of data produced and consumed (only visible for Dedicated clusters).
   * **Storage size**: View the total amount (in gigabytes) of data stored in your cluster (only visible for Dedicated clusters).

   ### Cost Optimized Clusters:

   * **ETUs | Elastic Throughput Units**: View how many ETUs are consumed by your clusters. ETUs provide dynamic, usage-based resource allocation that automatically scales with your actual throughput needs. Unlike clusters with Latency Optimized Profile, the clusters with Cost Optimized Profile use ETUs instead of separate CUs and SUs. Clusters with Cost Optimized Profile have a minimum allocation of 1 ETU.
   * **Functions | Function Processing Units**: View how many FPUs are committed by your functions and connectors (only visible for function-enabled clusters).

   ### For Serverless clusters:

   * **ETUs | Elastic Throughput Units**: View how many ETUs are consumed by your clusters. Serverless clusters use consumption-based pricing with automatic scaling. Currently, Serverless clusters start from 0 ETUs, though a minimum of 1 ETU may be introduced in the future.
   * **Functions | Function Processing Units**: View how many FPUs are committed by your functions and connectors (only visible for function-enabled clusters).
   * **Throughput**: View the total amount (in gigabytes) of data produced and consumed.
   * **Storage size**: View the total amount (in gigabytes) of data stored in your cluster.

   <Note>
     Serverless clusters and clusters with Cost Optimized Profile use a simplified architecture without BookKeeper or ZooKeeper, so you won't see separate Storage Units (SUs). All compute and storage resources are managed through ETUs.
   </Note>

3. (Optional) Use the slider under each chart to view hourly-reported usage.

## Export your usage data to a CSV file

You can export a summary of your organization usage to a CSV file (`.csv)`. By exporting your CSV file, you can easily find the usage and cost information for your organization or understand more about your costs.

1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select **Organization Usage**.

2. Select the target instances and the billing period, and then click the **Download** icon.

## Related topics

* [StreamNative Cloud Billing Overview](/cloud/billing/billing-overview)
* [Manage billing using StreamNative Cloud Console](/cloud/billing/billing)


# Advanced Observability Integration
Source: https://docs.streamnative.io/cloud/log-and-monitor/advanced-observability



<Note title="Note">
  The advanced observability features, including the Local Metrics Endpoint and Remote Write Integration, are only available for [**BYOC Pro**](/cloud/clusters/cluster-types#byoc-pro-clusters) clusters. Use the [Metrics API](/cloud/log-and-monitor/cloud-metrics-api) for other clusters, including [Serverless](/cloud/clusters/cluster-types#serverless-clusters), [Dedicated](/cloud/clusters/cluster-types#dedicated-clusters), and [BYOC](/cloud/clusters/cluster-types#byoc-clusters).

  If you want to access these advanced observability features, you can [contact StreamNative sales](https://www.streamnative.io/contact) to get a quote for BYOC Pro clusters.
</Note>

In addition to the [Metrics API](/cloud/log-and-monitor/cloud-metrics-api), StreamNative Cloud offers advanced observability features to help you gain deeper insights into your Pulsar clusters and applications. This document covers two key aspects of advanced observability:

1. Local Metrics Endpoint: A local Prometheus endpoint that is deployed with your BYOC Pro cluster. It provides access to all available metrics from your Pulsar clusters including both resource-related metrics and system-level metrics.

2. Remote Write Integration: Capabilities for seamlessly forwarding metrics to external monitoring and analytics platforms.

## Local Metrics Endpoint

The [Metrics API](/cloud/log-and-monitor/cloud-metrics-api) can be used for monitoring and troubleshooting your business applications. You can get the metrics from your Pulsar clusters, including tenants/namespaces/topics, connectors, functions, and so on. These metrics focus on the Pulsar resources inside your cluster. Metrics API excludes system-level metrics, such as the broker, bookie, and Pulsar Functions worker metrics. StreamNative manages these system-level metrics for you to ensure the stability and reliability of your Pulsar clusters.

If you need to access these system-level metrics, you can use the Local Metrics Endpoint. The Local Metrics Endpoint provides access to all available metrics from your Pulsar clusters, including the system-level metrics.

<Note title="Note">
  The Local Metrics Endpoint is only available for **BYOC Pro** clusters.
</Note>

### Enable the Local Metrics Endpoint

The Local Metrics Endpoint can be enabled on a per-cluster basis. Submit a **Change Request** in our [support portal](https://support.streamnative.cloud/) to enable the Local Metrics Endpoint for your cluster.

After the request is approved and the Local Metrics Endpoint is enabled, you will be provided with a Local Metrics Endpoint URL. Use this URL to access the metrics data, similar to the existing [Metrics API](/cloud/log-and-monitor/cloud-metrics-api).

## Metrics Remote Write Integration

For some reasons your observability stack can't pull from the [Metrics API](/cloud/log-and-monitor/cloud-metrics-api), StreamNative Cloud supports pushing metrics data with Remote Write to Prometheus-compatible systems and Datadog platform.

The Metrics Remote Write is enabled on the [Cloud Environment](/cloud/clusters/byoc/create-cloud-environment):

<Note title="Note">
  StreamNative Cloud does not yet support enabling the Metrics Remote Write on existing Cloud Environment through StreamNative Cloud Console.
</Note>

<img alt="Metrics Remote Write" />

### Remote Write to Prometheus

<img alt="Prometheus Remote Write" />

To enable the Prometheus Remote Write:

1. Enable the Metrics Remote Write switch.
2. Select the Prometheus icon.
3. Input the Prometheus endpoint address.
4. Select the Prometheus authentication type, for now supports Basic and Bearer token.

### Remote Write to Datadog

<img alt="Datadog Remote Write" />

To enable the Datadog Remote Write:

1. Enable the Metrics Remote Write switch.
2. Select the Datadog icon.
3. Input the [Datadog Sites](https://docs.datadoghq.com/getting_started/site/).
4. Input the Datadog [API Key](https://docs.datadoghq.com/account_management/api-app-keys/#api-keys).


# Cluster Metrics
Source: https://docs.streamnative.io/cloud/log-and-monitor/cloud-metrics-api



Metrics is a valuable tool for getting visibility into your Cloud deployment. StreamNative Cloud provides a broad range of metrics that you can use to help fine-tune performance and troubleshoot issues.

# Metrics endpoint

StreamNative Cloud provides an endpoint that exposes real-time metrics in [Prometheus metrics format](https://prometheus.io/docs/concepts/data_model/). The following table displays the currently available metrics endpoints.

<Note title="Important">
  Currently, the Cloud Metrics API only exposes resource-related metrics for Pulsar, including Tenants, Namespaces, Topics, Functions, Connectors, and others. System-level metrics are not exposed through this API. These system-level metrics are actively monitored and managed by the StreamNative Cloud team. However, for advanced observability use cases, you might need access to these system-level metrics. To meet this requirement, you can use the [Local Metrics Endpoint](/cloud/log-and-monitor/advanced-observability#local-metrics-endpoint). Please note that the Local Metrics Endpoint is only available for [**BYOC Pro** clusters](/cloud/clusters/cluster-types#byoc-pro-clusters).
</Note>

| Endpoint                                                                  | Description                                                  |
| ------------------------------------------------------------------------- | ------------------------------------------------------------ |
| `https://metrics.streamnative.cloud/v1/cloud/metrics/export`              | [Export Pulsar resource metrics](#pulsar-resource-metrics)   |
| `https://metrics.streamnative.cloud/v1/cloud/metrics/kafka/export`        | [Export Kafka resource metrics](#kafka-resource-metrics)     |
| `https://metrics.streamnative.cloud/v1/cloud/metrics/source/export`       | [Export Source connector metrics](#source-connector-metrics) |
| `https://metrics.streamnative.cloud/v1/cloud/metrics/sink/export`         | [Export Sink connector metrics](#sink-connector-metrics)     |
| `https://metrics.streamnative.cloud/v1/cloud/metrics/function/export`     | [Export Function metrics](#function-metrics)                 |
| `https://metrics.streamnative.cloud/v1/cloud/metrics/kafkaconnect/export` | [Export Kafka Connect metrics](#kafka-connect-metrics)       |
| `https://metrics.streamnative.cloud/v1/cloud/metrics/health/export`       | [Export Cluster health metrics](#health-metrics)             |

## Metrics authorization

To access and scrape metrics from the Cloud endpoints, you must use a Super Admin service account or a normal service account with `metrics-viewer` role.

### Super Admin service account

To create a super admin service account, please check the [create a service account](/cloud/security/authentication/service-accounts/service-accounts#create-a-service-account).

### metrics-viewer role

To bind a service account with `metrics-viewer`, your can configure it through `snctl` or `terraform`.

<Tabs>
  <Tab title="snctl">
    * create a normal service account

    ```
    snctl create serviceaccount metrics-account
    ```

    * create role binding with metrics-viewer

    ```
    snctl create rolebinding metrics-viewer --serviceaccount metrics-account --clusterrole metrics-viewer
    ```

    * In case you want to remove the permission to list metrics you can delete the rolebinding

    ```
    snctl delete rolebinding metrics-viewer
    ```
  </Tab>

  <Tab title="terraform">
    * Add a `streamnative_role_binding` resource in your terraform manifest file

    ```
    terraform {
      required_providers {
        streamnative = {
          source = "streamnative/streamnative"
        }
      }
    }

    provider "streamnative" {
        # Please replace path use your own key file path
        key_file_path = "/path/to/your/service/account/key.json"
    }

    resource "streamnative_service_account" "metrics-account" {
      organization = "xxxx"
      name = "metrics-account"
      admin = false
    }

    resource "streamnative_role_binding" "metrics-viewer" {
      organization = "xxxx"
      name         = "metrics-viewer"
      cluster_role_name = "metrics-viewer"
      service_account_names = ["metrics-account"]
    }
    ```

    * Run the terraform command to apply

    ```
    terrafrom apply --auto-approve
    ```
  </Tab>

  <Tab title="Console">
    This is not supported yet but will be available soon.
  </Tab>
</Tabs>

## Pulsar resource metrics

| Name                                    | Type      | Description                                                                                                                                       |
| --------------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| pulsar\_topics\_count                   | Gauge     | The number of Pulsar topics of the namespace owned by this broker.                                                                                |
| pulsar\_subscriptions\_count            | Gauge     | The number of Pulsar subscriptions of the topic served by this broker.                                                                            |
| pulsar\_producers\_count                | Gauge     | The number of active producers of the topic connected to this broker.                                                                             |
| pulsar\_consumers\_count                | Gauge     | The number of active consumers of the topic connected to this broker.                                                                             |
| pulsar\_rate\_in                        | Gauge     | The total message rate of the namespace coming into this broker (message/second).                                                                 |
| pulsar\_rate\_out                       | Gauge     | The total message rate of the namespace going out from this broker (message/second).                                                              |
| pulsar\_throughput\_in                  | Gauge     | The total throughput of the topic coming into this broker (byte per second).                                                                      |
| pulsar\_throughput\_out                 | Gauge     | The total throughput of the topic going out from this broker (byte per second).                                                                   |
| pulsar\_storage\_size                   | Gauge     | The total storage size of the topics in this topic owned by this broker (bytes).                                                                  |
| pulsar\_storage\_backlog\_size          | Gauge     | The total backlog size of the topics of this topic owned by this broker (in bytes).                                                               |
| pulsar\_storage\_offloaded\_size        | Gauge     | The total amount of the data in this topic offloaded to the tiered storage (bytes).                                                               |
| pulsar\_storage\_write\_rate            | Gauge     | The total message batches (entries) written to the storage for this topic (message batch per second).                                             |
| pulsar\_storage\_read\_rate             | Gauge     | The total message batches (entries) read from the storage for this topic (message batch per second).                                              |
| pulsar\_subscription\_delayed           | Gauge     | The total message batches (entries) are delayed for dispatching.                                                                                  |
| pulsar\_broker\_publish\_latency        | Summary   | The total latency of pulsar broker publish.                                                                                                       |
| pulsar\_broker\_storage\_read\_rate     | Gauge     | The total message batches (entries) read from the storage for this broker (message batch per second).                                             |
| pulsar\_broker\_storage\_write\_rate    | Gauge     | The total message batches (entries) written to the storage for this broker (message batch per second).                                            |
| pulsar\_entry\_size\_le\_\*             | Histogram | The entry rate of a namespace that the entry size is smaller with a given thresholds(128 bytes,512 bytes,1 KB,2 KB,4 KB,16 KB,100 KB,1 MB,>1 MB). |
| pulsar\_in\_bytes\_total                | Counter   | The total number of messages in bytes received for this topic.                                                                                    |
| pulsar\_msg\_backlog                    | Gauge     | The total number of message backlogs in this broker (entries).                                                                                    |
| pulsar\_storage\_write\_latency\_le\_\* | Histogram | The entry rate of a namespace that the storage write latency is smaller with a given threshold(0.5ms,1ms,5ms,10ms,20ms,50ms,100ms,200ms,1s,>1s).  |
| pulsar\_subscription\_back\_log         | Gauge     | The number of entries (messages/batched-messages) in unacknowledged state for a subscription.                                                     |

## Kafka resource metrics

| Name                                | Type    | Description                                                                                                                                                                                                                                                                                             |
| ----------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| kop\_server\_MESSAGE\_IN            | Counter | The producer message in stats. <br /> Available labels: *topic*, *partition*. <br /> <ul><li>*topic*: the topic name to produce.</li><li>*partition*: the partition id for the topic to produce</li></ul>                                                                                               |
| kop\_server\_MESSAGE\_OUT           | Counter | The consumer message out stats. <br /> Available labels: *topic*, *partition*, *group*. <br /> <ul><li>*topic*: the topic name to consume.</li><li>*partition*: the partition id for the topic to consume</li><li>*group*: the group id for consumer to consumer message from topic-partition</li></ul> |
| kop\_server\_BYTES\_IN              | Counter | The producer bytes in stats. <br /> Available labels: *topic*, *partition*. <br /> <ul><li>*topic*: the topic name to produce.</li><li>*partition*: the partition id for the topic to produce</li></ul>                                                                                                 |
| kop\_server\_BYTES\_OUT             | Counter | The consumer bytes out stats. <br /> Available labels: *topic*, *partition*, *group*. <br /> <ul><li>*topic*: the topic name to consume.</li><li>*partition*: the partition id for the topic to consume</li><li>*group*: the group id for consumer to consumer message from topic-partition</li></ul>   |
| kop\_server\_ACTIVE\_CHANNEL\_COUNT | Gauge   | The number of active connections                                                                                                                                                                                                                                                                        |
| kop\_server\_LAG                    | Gauge   | The consumer lag stats. <br /> Available labels: *topic*, *partition*, *group*. <br /> <ul><li>*topic*: the topic name to consume.</li><li>*partition*: the partition id for the topic to consume</li><li>*group*: the group id for consumer to consumer message from topic-partition</li></ul>         |

## Source connector metrics

| Name                                            | Type    | Description                                                                |
| ----------------------------------------------- | ------- | -------------------------------------------------------------------------- |
| pulsar\_source\_written\_total                  | Counter | The total number of records written to a Pulsar topic                      |
| pulsar\_source\_written\_1min\_total            | Counter | The total number of records written to a Pulsar topic in the last 1 minute |
| pulsar\_source\_received\_total                 | Counter | The total number of records received from source                           |
| pulsar\_source\_received\_1min\_total           | Counter | The total number of records received from source in the last 1 minute      |
| pulsar\_source\_last\_invocation                | Gauge   | The timestamp of the last invocation of the source                         |
| pulsar\_source\_source\_exception               | Gauge   | The exception from a source                                                |
| pulsar\_source\_source\_exceptions\_total       | Counter | The total number of source exceptions                                      |
| pulsar\_source\_source\_exceptions\_1min\_total | Counter | The total number of source exceptions in the last 1 minute                 |
| pulsar\_source\_system\_exception               | Gauge   | The exception from system code                                             |
| pulsar\_source\_system\_exceptions\_total       | Counter | The total number of system exceptions                                      |
| pulsar\_source\_system\_exceptions\_1min\_total | Counter | The total number of system exceptions in the last 1 minute                 |
| pulsar\_source\_user\_metric\_\*                | Summary | The user-defined metrics                                                   |
| process\_cpu\_seconds\_total                    | Counter | Total user and system CPU time spent in seconds.                           |
| jvm\_memory\_bytes\_committed                   | Gauge   | Committed (bytes) of a given JVM memory area.                              |
| jvm\_memory\_bytes\_max                         | Gauge   | Max (bytes) of a given JVM memory area.                                    |
| jvm\_memory\_direct\_bytes\_used                | Gauge   | Used bytes of a given JVM memory area.                                     |
| jvm\_memory\_bytes\_init                        | Gauge   | Initial bytes of a given JVM memory area.                                  |
| jvm\_gc\_collection\_seconds\_sum               | Summary | Time spent in a given JVM garbage collector in seconds.                    |

## Sink connector metrics

| Name                                          | Type    | Description                                                                |
| --------------------------------------------- | ------- | -------------------------------------------------------------------------- |
| pulsar\_sink\_written\_total                  | Counter | The total number of records written to a Pulsar topic                      |
| pulsar\_sink\_written\_1min\_total            | Counter | The total number of records written to a Pulsar topic in the last 1 minute |
| pulsar\_sink\_received\_total                 | Counter | The total number of records received from sink                             |
| pulsar\_sink\_received\_1min\_total           | Counter | The total number of records received from sink in the last 1 minute        |
| pulsar\_sink\_last\_invocation                | Gauge   | The timestamp of the last invocation of the sink                           |
| pulsar\_sink\_sink\_exception                 | Gauge   | The exception from a sink                                                  |
| pulsar\_sink\_sink\_exceptions\_total         | Counter | The total number of sink exceptions                                        |
| pulsar\_sink\_sink\_exceptions\_1min\_total   | Counter | The total number of sink exceptions in the last 1 minute                   |
| pulsar\_sink\_system\_exception               | Gauge   | The exception from system code                                             |
| pulsar\_sink\_system\_exceptions\_total       | Counter | The total number of system exceptions                                      |
| pulsar\_sink\_system\_exceptions\_1min\_total | Counter | The total number of system exceptions in the last 1 minute                 |
| pulsar\_sink\_user\_metric\_\*                | Summary | The user-defined metrics                                                   |
| process\_cpu\_seconds\_total                  | Counter | Total user and system CPU time spent in seconds.                           |
| jvm\_memory\_bytes\_committed                 | Gauge   | Committed (bytes) of a given JVM memory area.                              |
| jvm\_memory\_bytes\_max                       | Gauge   | Max (bytes) of a given JVM memory area.                                    |
| jvm\_memory\_direct\_bytes\_used              | Gauge   | Used bytes of a given JVM memory area.                                     |
| jvm\_memory\_bytes\_init                      | Gauge   | Initial bytes of a given JVM memory area.                                  |
| jvm\_gc\_collection\_seconds\_sum             | Summary | Time spent in a given JVM garbage collector in seconds.                    |

## Function metrics

| Name                                                   | Type    | Description                                                                   |
| ------------------------------------------------------ | ------- | ----------------------------------------------------------------------------- |
| pulsar\_function\_processed\_successfully\_total       | Counter | The total number of messages processed successfully                           |
| pulsar\_function\_processed\_successfully\_1min\_total | Counter | The total number of messages processed successfully in the last 1 minute      |
| pulsar\_function\_system\_exceptions\_total            | Counter | The total number of system exceptions                                         |
| pulsar\_function\_system\_exceptions\_1min\_total      | Counter | The total number of system exceptions in the last 1 minute                    |
| pulsar\_function\_user\_exceptions\_total              | Counter | The total number of user exceptions                                           |
| pulsar\_function\_user\_exceptions\_1min\_total        | Counter | The total number of user exceptions in the last 1 minute                      |
| pulsar\_function\_process\_latency\_ms                 | Summary | The process latency in milliseconds                                           |
| pulsar\_function\_process\_latency\_ms\_1min           | Summary | The process latency in milliseconds in the last 1 minute                      |
| pulsar\_function\_last\_invocation                     | Gauge   | The timestamp of the last invocation of the function                          |
| pulsar\_function\_received\_total                      | Counter | The total number of messages received from source                             |
| pulsar\_function\_received\_1min\_total                | Counter | The total number of messages received from source in the last 1 minute        |
| pulsar\_function\_user\_metric\_\*                     | Summary | The user-defined metrics                                                      |
| process\_cpu\_seconds\_total                           | Counter | Total user and system CPU time spent in seconds.                              |
| jvm\_memory\_bytes\_committed                          | Gauge   | Committed (bytes) of a given JVM memory area. (Java Functions only)           |
| jvm\_memory\_bytes\_max                                | Gauge   | Max (bytes) of a given JVM memory area. (Java Functions only)                 |
| jvm\_memory\_direct\_bytes\_used                       | Gauge   | Used bytes of a given JVM memory area. (Java Functions only)                  |
| jvm\_memory\_bytes\_init                               | Gauge   | Initial bytes of a given JVM memory area. (Java Functions only)               |
| jvm\_gc\_collection\_seconds\_sum                      | Summary | Time spent in a given JVM garbage collector in seconds. (Java Functions only) |

## Kafka Connect metrics

| Name                                                                 | Type    | Description                                                                                                                                                                        |
| -------------------------------------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| kafka\_connect\_connector\_task\_batch\_size\_avg                    | Gauge   | The average size of the batches processed by the connector                                                                                                                         |
| kafka\_connect\_connector\_task\_batch\_size\_max                    | Gauge   | The maximum size of the batches processed by the connector                                                                                                                         |
| kafka\_connect\_connector\_task\_offset\_commit\_avg\_time\_ms       | Gauge   | The average time in milliseconds taken by this task to commit offsets                                                                                                              |
| kafka\_connect\_connector\_task\_offset\_commit\_failure\_percentage | Gauge   | The average percentage of this task's offset commit attempts that failed                                                                                                           |
| kafka\_connect\_connector\_task\_offset\_commit\_max\_time\_ms       | Gauge   | The maximum time in milliseconds taken by this task to commit offsets                                                                                                              |
| kafka\_connect\_connector\_task\_offset\_commit\_success\_percentage | Gauge   | The average percentage of this task's offset commit attempts that succeeded                                                                                                        |
| kafka\_connect\_connector\_task\_pause\_ratio                        | Gauge   | The fraction of time this task has spent in the pause state                                                                                                                        |
| kafka\_connect\_connector\_task\_running\_ratio                      | Gauge   | The fraction of time this task has spent in the running state                                                                                                                      |
| kafka\_connect\_source\_task\_source\_record\_poll                   | Gauge   | The total number of records produced/polled (before transformation) by this task belonging to the named source connector in this worker                                            |
| kafka\_connect\_source\_task\_source\_record\_poll\_rate             | Gauge   | The average per-second number of records produced/polled (before transformation) by this task belonging to the named source connector in this worker                               |
| kafka\_connect\_source\_task\_source\_record\_write                  | Gauge   | The number of records output from the transformations and written to Kafka for this task belonging to the named source connector in this worker, since the task was last restarted |
| kafka\_connect\_source\_task\_source\_record\_write\_rate            | Gauge   | The average per-second number of records output from the transformations and written to Kafka for this task belonging to the named source connector in this worker                 |
| kafka\_connect\_source\_task\_poll\_batch\_avg\_time\_ms             | Gauge   | The average time in milliseconds taken by this task to poll for a batch of source records                                                                                          |
| kafka\_connect\_source\_task\_poll\_batch\_max\_time\_ms             | Gauge   | The maximum time in milliseconds taken by this task to poll for a batch of source records                                                                                          |
| kafka\_connect\_source\_task\_source\_record\_active\_count          | Gauge   | The number of records that have been produced by this task but not yet completely written to Kafka                                                                                 |
| kafka\_connect\_source\_task\_source\_record\_active\_count\_avg     | Gauge   | The average number of records that have been produced by this task but not yet completely written to Kafka                                                                         |
| kafka\_connect\_source\_task\_source\_record\_active\_count\_max     | Gauge   | The maximum number of records that have been produced by this task but not yet completely written to Kafka                                                                         |
| kafka\_connect\_sink\_task\_offset\_commit\_completion               | Gauge   | The total number of offset commit completions that were completed successfully                                                                                                     |
| kafka\_connect\_sink\_task\_offset\_commit\_completion\_rate         | Gauge   | The average per-second number of offset commit completions that were completed successfully                                                                                        |
| kafka\_connect\_sink\_task\_offset\_commit\_seq\_no                  | Gauge   | The current sequence number for offset commits                                                                                                                                     |
| kafka\_connect\_sink\_task\_offset\_commit\_skip                     | Gauge   | The total number of offset commit completions that were received too late and skipped/ignored                                                                                      |
| kafka\_connect\_sink\_task\_offset\_commit\_skip\_rate               | Gauge   | The average per-second number of offset commit completions that were received too late and skipped/ignored                                                                         |
| kafka\_connect\_sink\_task\_partition\_count                         | Gauge   | The number of topic partitions assigned to this task belonging to the named sink connector in this worker                                                                          |
| kafka\_connect\_sink\_task\_put\_batch\_avg\_time\_ms                | Gauge   | The average time taken by this task to put a batch of sinks records                                                                                                                |
| kafka\_connect\_sink\_task\_put\_batch\_max\_time\_ms                | Gauge   | The maximum time taken by this task to put a batch of sinks records                                                                                                                |
| kafka\_connect\_sink\_task\_sink\_record\_active\_count              | Gauge   | The number of records that have been read from Kafka but not yet completely committed/flushed/acknowledged by the sink task                                                        |
| kafka\_connect\_sink\_task\_sink\_record\_active\_count\_avg         | Gauge   | The average number of records that have been read from Kafka but not yet completely committed/flushed/acknowledged by the sink task                                                |
| kafka\_connect\_sink\_task\_sink\_record\_active\_count\_max         | Gauge   | The maximum number of records that have been read from Kafka but not yet completely committed/flushed/acknowledged by the sink task                                                |
| kafka\_connect\_sink\_task\_sink\_record\_read                       | Gauge   | The total number of records read from Kafka by this task belonging to the named sink connector in this worker, since the task was last restarted                                   |
| kafka\_connect\_sink\_task\_sink\_record\_read\_rate                 | Gauge   | The average per-second number of records read from Kafka for this task belonging to the named sink connector in this worker. This is before transformations are applied            |
| kafka\_connect\_sink\_task\_sink\_record\_send                       | Gauge   | The total number of records output from the transformations and sent/put to this task belonging to the named sink connector in this worker, since the task was last restarted      |
| kafka\_connect\_sink\_task\_sink\_record\_send\_rate                 | Gauge   | The average per-second number of records output from the transformations and sent/put to this task belonging to the named sink connector in this worker                            |
| kafka\_connect\_task\_error\_deadletterqueue\_produce\_failures      | Gauge   | The number of failed writes to the dead letter queue                                                                                                                               |
| kafka\_connect\_task\_error\_deadletterqueue\_produce\_requests      | Gauge   | The number of attempted writes to the dead letter queue                                                                                                                            |
| kafka\_connect\_task\_error\_last\_error\_timestamp                  | Gauge   | The epoch timestamp when this task last encountered an error                                                                                                                       |
| kafka\_connect\_task\_error\_total\_errors\_logged                   | Gauge   | The total number of errors that were logged                                                                                                                                        |
| kafka\_connect\_task\_error\_total\_record\_errors                   | Gauge   | The total number of record processing errors in this task                                                                                                                          |
| kafka\_connect\_task\_error\_total\_record\_failures                 | Gauge   | The total number of record processing failures in this task                                                                                                                        |
| kafka\_connect\_task\_error\_total\_records\_skipped                 | Gauge   | The total number of records skipped due to errors                                                                                                                                  |
| kafka\_connect\_task\_error\_total\_retries                          | Gauge   | The total number of operations retried                                                                                                                                             |
| kafka\_connect\_worker\_connector\_destroyed\_task\_count            | Gauge   | The number of destroyed tasks of the connector on the worker                                                                                                                       |
| kafka\_connect\_worker\_connector\_failed\_task\_count               | Gauge   | The number of failed tasks of the connector on the worker                                                                                                                          |
| kafka\_connect\_worker\_connector\_paused\_task\_count               | Gauge   | The number of paused tasks of the connector on the worker                                                                                                                          |
| kafka\_connect\_worker\_connector\_restarting\_task\_count           | Gauge   | The number of restarting tasks of the connector on the worker                                                                                                                      |
| kafka\_connect\_worker\_connector\_running\_task\_count              | Gauge   | The number of running tasks of the connector on the worker                                                                                                                         |
| kafka\_connect\_worker\_connector\_total\_task\_count                | Gauge   | The number of tasks of the connector on the worker                                                                                                                                 |
| kafka\_connect\_worker\_connector\_unassigned\_task\_count           | Gauge   | The number of unassigned tasks of the connector on the worker                                                                                                                      |
| process\_cpu\_seconds\_total                                         | Counter | Total user and system CPU time spent in seconds                                                                                                                                    |
| jvm\_memory\_committed\_bytes                                        | Gauge   | Committed (bytes) of a given JVM memory area                                                                                                                                       |
| jvm\_memory\_max\_bytes                                              | Gauge   | Max (bytes) of a given JVM memory area                                                                                                                                             |
| jvm\_memory\_init\_bytes                                             | Gauge   | Initial bytes of a given JVM memory area                                                                                                                                           |
| jvm\_memory\_used\_bytes                                             | Gauge   | Used bytes of a given JVM memory area                                                                                                                                              |
| jvm\_gc\_collection\_seconds\_sum                                    | Summary | Time spent in a given JVM garbage collector in seconds                                                                                                                             |

## Health metrics

| Name                                          | Type    | Description                                                                                               |
| --------------------------------------------- | ------- | --------------------------------------------------------------------------------------------------------- |
| pulsar\_detector\_e2e\_latency\_ms            | Summary | The latency distribution from message sending to message consumption                                      |
| pulsar\_detector\_publish\_latency\_ms        | Summary | The latency distribution of message sending                                                               |
| pulsar\_detector\_pulsar\_sla\_messaging\_up  | Gauge   | The gauge for indicating the messaging service up or down                                                 |
| pulsar\_detector\_pulsar\_sla\_webservice\_up | gauge   | The gauge for indicating the webservice up or down                                                        |
| pulsar\_detector\_geo\_latency\_ms            | Summary | The latency distribution Latency distribution from message sending to message consumption across clusters |

# Metrics API integration

<Note title="Note">
  The examples below demonstrate how to configure your observability tool to scrape the metrics endpoint. While StreamNative Cloud provides the metrics endpoint, it is your responsibility to set up and manage your own observability stack.
</Note>

## Prometheus integration

To collect Pulsar metrics into Prometheus, add the following to your Prometheus configuration file. The bearer tokens have a limited life cycle, therefore it is recommended to use the OAuth2 authentication method.

```yaml theme={null}
global:
  scrape_interval: 120s
  scrape_timeout: 60s
scrape_configs:
  - job_name: streamnative
    metrics_path: /v1/cloud/metrics/export
    scheme: https
    oauth2:
      client_id: '${client_id}'
      client_secret: '${client_secret}'
      token_url: https://auth.streamnative.cloud/oauth/token
      endpoint_params:
        grant_type: 'client_credentials'
        audience: '${audience}'
    static_configs:
      - targets: [metrics.streamnative.cloud]
```

You can find the values of `client_id` and `client_secret` in the `Key` file of a Super Admin Service Account. For more information, see [work with service accounts](/cloud/security/authentication/service-accounts/service-accounts).

The `audience` parameter is the [Uniform Resource Name (URN)](/cloud/references/glossary#urn), which is a combination of the `urn:sn:pulsar`, the organization name, and the Pulsar instance name at StreamNative:

```bash theme={null}
"urn:sn:pulsar:${org_name}:${instance_name}"
```

<Note>
  The Prometheus response can be large, if your cluster has a lot of topics. Make sure to set the `scrape_timeout` parameter large enough to cover the duration of the curl request above.
  Your `scrape_interval` parameter should also be larger than your `scrape_timeout` parameter.
</Note>

## OpenTelemetry collector integration

The [OpenTelemetry collector](https://opentelemetry.io/docs/collector/getting-started/), as described on its official page, is a vendor-agnostic agent process designed for gathering and sending telemetry data from various sources. StreamNative Cloud, which outputs its metrics in the Prometheus format, is compatible with the OpenTelemetry collector. To collect metrics from StreamNative Cloud, configure your OpenTelemetry collector to utilize the [Prometheus Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/prometheusreceiver), which is fully compatible with Prometheus's scape\_config settings.

To configure your collector, refer to the guidance provided in the [Prometheus Integration section](#prometheus-integration). There, you will find instructions to create a `scape_config` for collecting metrics from StreamNative Cloud. This config should be placed in your collector's configuration file under the following section:

```yaml theme={null}
receivers:
  prometheus:
    config:
```

An example of such configuration is as follows:

```yaml theme={null}
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: streamnative
          metrics_path: /v1/cloud/metrics/export
          scheme: https
          oauth2:
            client_id: '${client_id}'
            client_secret: '${client_secret}'
            token_url: https://auth.streamnative.cloud/oauth/token
            endpoint_params:
              grant_type: 'client_credentials'
              audience: '${audience}'
          static_configs:
            - targets: [metrics.streamnative.cloud]
```

The OpenTelemetry collector's versatility allows it to support a range of exporters, facilitating the routing of metrics from StreamNative Cloud to various observability platforms. A comprehensive list of supported exporters by the OpenTelemetry collector is available [here](https://opentelemetry.io/docs/collector/configuration/#exporters).

## NewRelic integration

You can use a Prometheus instance to forward metrics to NewRelic. To do this, add a `remote_write` entry to the `prometheus.yml` configuration file as described [in the Prometheus Integration section](#prometheus-integration):

```yml theme={null}
remote_write:
  - url: https://metric-api.newrelic.com/prometheus/v1/write?prometheus_server=streamnative
    authorization:
      credentials: '${newrelic_ingest_key}'
```

<Note title="Note">
  The NewRelic ingestion point could also be `metric-api.eu.newrelic.com` depending on your account configuration.
</Note>

Then by running a Prometheus instance, the Pulsar metrics are scraped from the StreamNative endpoint and forwarded to NewRelic:

```bash theme={null}
prometheus --config.file=prometheus.yml
```

If you want to keep data from going into this Prometheus instance, you can setup a short retention time with the `storage.tsdb.retention.time` parameter:

```bash theme={null}
prometheus --config.file=prometheus.yml --storage.tsdb.retention.time=15m
```

## Grafana Cloud integration

You can use a Prometheus instance to forward metrics to Grafana Cloud. To do this, add a `remote_write` entry to the `prometheus.yml` configuration file as described [in the Prometheus Integration section](#prometheus-integration):

```yml theme={null}
remote_write:
  - url: ${grafana_cloud_endpoint}/api/prom/push
    basic_auth:
      username: '${grafana_cloud_username}'
      password: '${grafana_cloud_api_key}'
```

You can find the `grafana_cloud_endpoint` and `grafana_cloud_username` values by selecting Prometheus at `https://grafana.com/orgs/${grafana_org}`. You can find `grafana_cloud_api_key` at `https://grafana.com/orgs/${grafana_org}/api-keys`.

Then by running a Prometheus instance, the Pulsar metrics are scraped from the StreamNative endpoint and forwarded to Grafana Cloud:

```bash theme={null}
prometheus --config.file=prometheus.yml
```

If you want to keep data from going into this Prometheus instance, you can setup a short retention time with the `storage.tsdb.retention.time` parameter:

```bash theme={null}
prometheus --config.file=prometheus.yml --storage.tsdb.retention.time=15m
```

## Datadog integration

### Integrate with Datadog Agent

<Note title="Note">
  The integration with StreamNative Cloud requires the [PR 16812](https://github.com/DataDog/integrations-core/pull/16812) which released in the Datadog Agent [7.52.0](https://github.com/DataDog/datadog-agent/releases/tag/7.52.0).
</Note>

Using Datadog Agent, you can connect Datadog to the StreamNative Cloud Metrics endpoint to start collecting metrics. Datadog Agent supports most platform to host and this documentation will mainly to demonstrate with Docker and Kubernetes.

<Tabs>
  <Tab title="Docker">
    Create a file `conf.yaml`, with the spec of your Datadog Agent deployment configuration.

    ```yaml theme={null}
    init_config:
      service: docker

    instances:
      - openmetrics_endpoint: https://metrics.streamnative.cloud/v1/cloud/metrics/export
        request_size: 900
        min_collection_interval: 180
        metrics:
          - pulsar_topics_count:
              type: gauge
              name: pulsar_topics_count
          - pulsar_subscriptions_count:
              type: gauge
              name: pulsar_subscriptions_count
          - pulsar_producers_count:
              type: gauge
              name: pulsar_producers_count
          - pulsar_consumers_count:
              type: gauge
              name: pulsar_consumers_count
          - pulsar_rate_in:
              type: gauge
              name: pulsar_rate_in
          - pulsar_rate_out:
              type: gauge
              name: pulsar_rate_out
          - pulsar_throughput_in:
              type: gauge
              name: pulsar_throughput_in
          - pulsar_throughput_out:
              type: gauge
              name: pulsar_throughput_out
          - pulsar_storage_size:
              type: gauge
              name: pulsar_storage_size
          - pulsar_storage_backlog_size:
              type: gauge
              name: pulsar_storage_backlog_size
          - pulsar_storage_offloaded_size:
              type: gauge
              name: pulsar_storage_offloaded_size
          - pulsar_storage_read_rate:
              type: gauge
              name: pulsar_storage_read_rate
          - pulsar_subscription_delayed:
              type: gauge
              name: pulsar_subscription_delayed
          - pulsar_storage_write_latency_le_0_5:
              type: histogram
              name: pulsar_storage_write_latency_le_0_5
          - pulsar_storage_write_latency_le_1:
              type: histogram
              name: pulsar_storage_write_latency_le_1
          - pulsar_storage_write_latency_le_5:
              type: histogram
              name: pulsar_storage_write_latency_le_5
          - pulsar_storage_write_latency_le_10:
              type: histogram
              name: pulsar_storage_write_latency_le_10
          - pulsar_storage_write_latency_le_20:
              type: histogram
              name: pulsar_storage_write_latency_le_20
          - pulsar_storage_write_latency_le_50:
              type: histogram
              name: pulsar_storage_write_latency_le_50
          - pulsar_storage_write_latency_le_100:
              type: histogram
              name: pulsar_storage_write_latency_le_100
          - pulsar_storage_write_latency_le_200:
              type: histogram
              name: pulsar_storage_write_latency_le_200
          - pulsar_storage_write_latency_le_1000:
              type: histogram
              name: pulsar_storage_write_latency_le_1000
          - pulsar_storage_write_latency_le_overflow:
              type: histogram
              name: pulsar_storage_write_latency_le_overflow
          - pulsar_entry_size_le_128:
              type: histogram
              name: pulsar_entry_size_le_128
          - pulsar_entry_size_le_512:
              type: histogram
              name: pulsar_entry_size_le_512
          - pulsar_entry_size_le_1_kb:
              type: histogram
              name: pulsar_entry_size_le_1_kb
          - pulsar_entry_size_le_4_kb:
              type: histogram
              name: pulsar_entry_size_le_4_kb
          - pulsar_entry_size_le_16_kb:
              type: histogram
              name: pulsar_entry_size_le_16_kb
        auth_token:
          reader:
            type: oauth
            url: https://auth.streamnative.cloud/oauth/token
            client_id: { your-admin-service-account-client-id }
            client_secret: { your-admin-service-account-client-secret }
            options:
              audience: urn:sn:pulsar:{your-organization}:{your-instance}
          writer:
            type: header
            name: Authorization
            value: Bearer <TOKEN>
            placeholder: <TOKEN>
    ```

    * \[1] `client_id`: Required. You need to prepare a [service account](/cloud/security/authentication/service-accounts/service-accounts) with Super Admin pemision and the `client_id` can be obtained from an [OAuth2 credential file](/cloud/security/authentication/service-accounts/service-accounts#get-a-key-file).
    * \[2] `client_secret`: Required. You need to prepare a [service account](/cloud/security/authentication/service-accounts/service-accounts) with Super Admin pemision andt the `client_id` can be obtained from an [OAuth2 credential file](/cloud/security/authentication/service-accounts/service-accounts#get-a-key-file).
    * \[3] `audience`: Required. Audience is the [Uniform Resource Name (URN)](/cloud/references/glossary#urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name. `{organization}` is the name of your [organization](/cloud/references/glossary#organization) and the `{instance}` is the name of your [instance](/cloud/references/glossary#instance).

    Run the docker commands to create a Datadog Agent container:

    ```bash theme={null}
    docker run -d --name dd-agent \
    -e DD_API_KEY={ your-Datadog-API-Key } \
    -e DD_SITE={ your-Datadog-Site-region } \
    -e DD_APM_NON_LOCAL_TRAFFIC=true \
    -v {your-config-yaml-file-path}:/etc/datadog-agent/conf.d/openmetrics.d/conf.yaml:ro \
    -v /var/run/docker.sock:/var/run/docker.sock:ro \
    -v /proc/:/host/proc/:ro \
    -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
    -v /var/lib/docker/containers:/var/lib/docker/containers:ro \
    datadog/agent:7.52.0
    ```

    * \[1] `DD_API_KEY`: Your Datadog API key.
    * \[2] `DD_SITE`: Destination site for your metrics, traces, and logs. Set your Datadog site to: `datadoghq.com`. Defaults to `datadoghq.com`.
    * \[3] `your-config-yaml-file-path`: The `conf.yaml` configuration file created in the first step.

    More detailed usage please refer the [Docker Agent for Docker](https://docs.datadoghq.com/containers/docker/?tab=standard).
  </Tab>

  <Tab title="Kubernetes">
    This documentation will use the [Datadog Operator](https://docs.datadoghq.com/containers/datadog_operator/) to demonstrate.

    Install the Datadog Operator

    ```bash theme={null}
    helm repo add datadog https://helm.datadoghq.com
    helm install datadog-operator datadog/datadog-operator
    ```

    Create a Kubernetes secret with your API and app keys

    ```bash theme={null}
    kubectl create secret generic datadog-secret --from-literal api-key=<DATADOG_API_KEY> --from-literal app-key=<DATADOG_APP_KEY>

    ```

    * \[1] `DATADOG_API_KEY`: Your Datadog API key.
    * \[2] `DATADOG_APP_KEY`: Your Datadog Application key.

    Create a file `datadog-agent.yaml`, with the spec of your Datadog Agent deployment configuration.

    ```yaml theme={null}
    apiVersion: datadoghq.com/v2alpha1
    kind: DatadogAgent
    metadata:
      namespace: datadog
      name: datadog-agent
    spec:
      global:
        kubelet:
          tlsVerify: false
        site: datadoghq.com
        credentials:
          apiSecret:
            secretName: datadog-secret
            keyName: api-key
          appSecret:
            secretName: datadog-secret
            keyName: app-key
      override:
        nodeAgent:
          image:
            name: gcr.io/datadoghq/agent:7.52.0
          extraConfd:
            configDataMap:
              openmetrics.yaml: |-
                init_config:
                  service: datadog_operator
                instances:
                  - openmetrics_endpoint: https://metrics.streamnative.cloud/v1/cloud/metrics/export
                    request_size: 900
                    min_collection_interval: 180
                    metrics:
                      - pulsar_topics_count:
                          type: gauge
                          name: pulsar_topics_count
                      - pulsar_subscriptions_count:
                          type: gauge
                          name: pulsar_subscriptions_count
                      - pulsar_producers_count:
                          type: gauge
                          name: pulsar_producers_count
                      - pulsar_consumers_count:
                          type: gauge
                          name: pulsar_consumers_count
                      - pulsar_rate_in:
                          type: gauge
                          name: pulsar_rate_in
                      - pulsar_rate_out:
                          type: gauge
                          name: pulsar_rate_out
                      - pulsar_throughput_in:
                          type: gauge
                          name: pulsar_throughput_in
                      - pulsar_throughput_out:
                          type: gauge
                          name: pulsar_throughput_out
                      - pulsar_storage_size:
                          type: gauge
                          name: pulsar_storage_size
                      - pulsar_storage_backlog_size:
                          type: gauge
                          name: pulsar_storage_backlog_size
                      - pulsar_storage_offloaded_size:
                          type: gauge
                          name: pulsar_storage_offloaded_size
                      - pulsar_storage_read_rate:
                          type: gauge
                          name: pulsar_storage_read_rate
                      - pulsar_subscription_delayed:
                          type: gauge
                          name: pulsar_subscription_delayed
                      - pulsar_storage_write_latency_le_0_5:
                          type: histogram
                          name: pulsar_storage_write_latency_le_0_5
                      - pulsar_storage_write_latency_le_1:
                          type: histogram
                          name: pulsar_storage_write_latency_le_1
                      - pulsar_storage_write_latency_le_5:
                          type: histogram
                          name: pulsar_storage_write_latency_le_5
                      - pulsar_storage_write_latency_le_10:
                          type: histogram
                          name: pulsar_storage_write_latency_le_10
                      - pulsar_storage_write_latency_le_20:
                          type: histogram
                          name: pulsar_storage_write_latency_le_20
                      - pulsar_storage_write_latency_le_50:
                          type: histogram
                          name: pulsar_storage_write_latency_le_50
                      - pulsar_storage_write_latency_le_100:
                          type: histogram
                          name: pulsar_storage_write_latency_le_100
                      - pulsar_storage_write_latency_le_200:
                          type: histogram
                          name: pulsar_storage_write_latency_le_200
                      - pulsar_storage_write_latency_le_1000:
                          type: histogram
                          name: pulsar_storage_write_latency_le_1000
                      - pulsar_storage_write_latency_le_overflow:
                          type: histogram
                          name: pulsar_storage_write_latency_le_overflow
                      - pulsar_entry_size_le_128:
                          type: histogram
                          name: pulsar_entry_size_le_128
                      - pulsar_entry_size_le_512:
                          type: histogram
                          name: pulsar_entry_size_le_512
                      - pulsar_entry_size_le_1_kb:
                          type: histogram
                          name: pulsar_entry_size_le_1_kb
                      - pulsar_entry_size_le_4_kb:
                          type: histogram
                          name: pulsar_entry_size_le_4_kb
                      - pulsar_entry_size_le_16_kb:
                          type: histogram
                          name: pulsar_entry_size_le_16_kb
                    auth_token:
                      reader:
                        type: oauth
                        url: https://auth.streamnative.cloud/oauth/token
                        client_id: { your-admin-service-account-client-id }
                        client_secret: { your-admin-service-account-client-secret }
                        options:
                          audience: urn:sn:pulsar:{your-organization}:{your-instance}
                      writer:
                        type: header
                        name: Authorization
                        value: Bearer <TOKEN>
                        placeholder: <TOKEN>
    ```

    * \[1] `client_id`: Required. You need to prepare a [service account](/cloud/security/authentication/service-accounts/service-accounts) with Super Admin pemision and the `client_id` can be obtained from an [OAuth2 credential file](/cloud/security/authentication/service-accounts/service-accounts#get-a-key-file).
    * \[2] `client_secret`: Required. You need to prepare a [service account](/cloud/security/authentication/service-accounts/service-accounts) with Super Admin pemision andt the `client_id` can be obtained from an [OAuth2 credential file](/cloud/security/authentication/service-accounts/service-accounts#get-a-key-file).
    * \[3] `audience`: Required. Audience is the [Uniform Resource Name (URN)](/cloud/references/glossary#urn), which is a combination of the `urn:sn:pulsar`, your organization name, and your Pulsar instance name. `{organization}` is the name of your [organization](/cloud/references/glossary#organization) and the `{instance}` is the name of your [instance](/cloud/references/glossary#instance).

    Deploy the Datadog Agent with the above configuration file

    ```bash theme={null}
    kubectl apply -f /path/to/your/datadog-agent.yaml
    ```

    More detailed usage please refer the [Install the Datadog Agent on Kubernetes](https://docs.datadoghq.com/containers/kubernetes/installation/).
  </Tab>
</Tabs>

### Bridge with OpenTelemetry

You can use [OpenTelemetry Collector](#opentelemetry-collector-integration) to collect the metrics from StreamNative Cloud and export them to Datadog.

To export metrics to Datadog, you can use the [Datadog Exporter](https://docs.datadoghq.com/opentelemetry/otel_collector_datadog_exporter/) and add it to your [OpenTelemetry Collector configuration](https://opentelemetry.io/docs/collector/configuration/). Use the example file which provides a basic configuration that is ready to use after you set your Datadog API key as the `${DD_API_KEY}` variable:

```yaml theme={null}
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: streamnative
          metrics_path: /v1/cloud/metrics/export
          scheme: https
          oauth2:
            client_id: '${client_id}'
            client_secret: '${client_secret}'
            token_url: https://auth.streamnative.cloud/oauth/token
            endpoint_params:
              grant_type: 'client_credentials'
              audience: '${audience}'
          static_configs:
            - targets: [metrics.streamnative.cloud]

processors:
  batch:
    send_batch_max_size: '10MiB'
    send_batch_size: 4096
    timeout: 120s

exporters:
  datadog:
    api:
      site: ${DD_SITE}
      key: ${DD_API_KEY}

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [datadog]
```

Where `${DD_SITE}` is your site, .

The above configuration enables the receiving of metrics from StreamNative Cloud, sets up a batch processor, which is mandatory for any non-development environment, and exports to Datadog. You can refer to [this full documented example configuration file](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/datadogexporter/examples/collector.yaml) for all possible configuration options for Datadog Exporter.


# Grafana Dashboards and Alerting rules
Source: https://docs.streamnative.io/cloud/log-and-monitor/grafana-dashboards



## Grafana Dashboards

StreamNative Cloud users can use the pre-built grafana dashboards to monitor the Pulsar cluster resources.

### Setup the Prometheus instance

Follow the [Prometheus integration](/cloud/log-and-monitor/cloud-metrics-api#prometheus-integration) to setup a prometheus as the Grafana datasource.

### Setup the Grafana instance

Follow the [Set up Grafana](https://grafana.com/docs/grafana/latest/setup-grafana/) to set the Grafana up and running. Import the [pre-built grafana dashboards](https://github.com/streamnative/streamnative-cloud-dashboard/tree/main/dashboards):

<img alt="dashboard-import" />

## Alerting rules

StreamNative Cloud users can refer the [alerting rules](https://raw.githubusercontent.com/streamnative/streamnative-cloud-dashboard/refs/heads/main/alerts/rule.yaml) to configure for the [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/).


# Log Console
Source: https://docs.streamnative.io/cloud/log-and-monitor/log-conosole



The logs console enables developers to view, filter, search, and sort logs from Kafka Connectors, Pulsar Connectors, and Functions, streamlining log management and troubleshooting.

## View logs

Navigate to Logs tab to view the logs.

<img alt="View logs" />

## Filter logs

Apply filters to efficiently access and review relevant logs.

### Filter logs by levels

Select the desired log level from the dropdown menu. Multiple log levels can be selected simultaneously. For normal operation, the INFO log level provides quick insights. In cases where a connector or function encounters issues, filtering logs by DEBUG or ERROR levels helps to identify and resolve problems efficiently.

<img alt="View logs" />

### Filter logs by time stamp

Refine log visibility by specifying a start and end timestamp to filter logs within a defined time range.

<img alt="View logs" />

## Sort logs

Organize logs in ascending or descending order by clicking the corresponding arrow icons.

<img alt="View logs" />

## Search logs

Search logs by entering keywords in the search field.

<img alt="View logs" />


# Manage Notifications
Source: https://docs.streamnative.io/cloud/log-and-monitor/manage-notifications



StreamNative Cloud Notifications let you proactively monitor the health and status of your critical resources. You can create alert rules from a set of built-in templates, each based on operational best practices. When a rule's condition is met, StreamNative Cloud automatically sends a detailed email notification to the designated recipients.

## How notifications work

The notification system is built on the following concepts:

* **Template-based rules**: StreamNative provides a set of built-in alert templates. Each template defines the alert condition, severity, and configurable parameters. You create one or more rules from a template and specify the exact scope and recipients for each.
* **Fine-grained scope**: When creating a rule, you choose the scope at which the alert applies—organization-wide, or narrowed down to a specific instance, cluster, tenant, namespace, or function. This lets you route different alerts to different teams.
* **Configurable recipients**: Each rule can have its own list of recipient email addresses. If no recipients are specified, the alert falls back to your organization's **Technical Contact Email**. You can configure the Technical Contact in your [Organization Profile](https://docs.streamnative.io/cloud/security/access/resource-hierarchy/organizations#organization-profile).
* **Automatic email lifecycle**: When a rule is triggered, an email is sent immediately. A follow-up email is sent once the issue is resolved.

Below is an example of the notification email you receive when an alert is triggered.

<img alt="Example notification email" />

## Available alert templates

The following built-in templates are available. You can create multiple rules from the same template with different scopes or recipients.

| Template                      | Service                | Condition                                                                                      | Severity  |
| :---------------------------- | :--------------------- | :--------------------------------------------------------------------------------------------- | :-------- |
| **Function OOMKilled**        | Functions / Connectors | A function or connector container has been `OOMKilled` 2 or more times in the last 15 minutes. | `error`   |
| **Function CrashLoopBackOff** | Functions / Connectors | A function or connector pod has been in `CrashLoopBackOff` status for 5 minutes.               | `error`   |
| **API Key Expiration**        | API Key                | An API key is about to expire within 7 days. You can filter by service account.                | `warning` |
| **BookKeeper Disk Usage**     | BookKeeper             | Cluster disk is projected to be full within 2 days.                                            | `warning` |

<Note>
  Alert conditions and severity levels are defined by StreamNative based on operational best practices and cannot be modified.
</Note>

## Navigate to Notifications

1. Click the **user icon** in the top-right corner of the StreamNative Cloud Console.
2. Select **Notifications** from the menu.

<img alt="Navigate to Notifications page" />

The Notifications page lists all alert rules you have created, along with their severity, scope, receivers, and current status.

<img alt="Example notification email" />

## Create an alert rule

1. On the **Notifications** page, click **Create Rule**.

2. In the **Create Alert Rule** dialog, fill in the following fields:

   * **Rule Name**: A unique, lowercase identifier for this rule (for example, `oom-killed-prod-cluster`). Must contain only lowercase letters, numbers, and hyphens.
   * **Alert Template**: Select the condition type from the dropdown. A short description of the template appears below the selector.

<img alt="Example notification email" />

3. **(Optional) Set the scope**: By default, a rule applies to your entire organization. To narrow the scope, click a scope pill (for example, **Instance**, **Cluster**, **Function**) and fill in the target value(s). Leaving the scope at **Organization** means the rule monitors all matching resources in your organization.

4. **(Optional) Set recipients**: Enter one or more email addresses (comma-separated) in the **Receivers** field.

   <Note>
     For the **BookKeeper Disk Usage** template, the receiver is fixed to your organization's Technical Contact Email and cannot be overridden here. To change that email, update your [Organization Profile](https://docs.streamnative.io/cloud/security/access/resource-hierarchy/organizations#organization-profile).
   </Note>

5. Click **Save**. The new rule appears in the list with status **Reconciling** while it is being set up, and transitions to **Ready** once active.

## View rule details

Click any row in the Notifications list to open the rule detail panel. The detail view shows:

* **Basic Info**: Rule name, alert template, description, scope, and severity.
* **Delivery**: The configured recipient email addresses (or a note indicating the organization default is used).

## Delete an alert rule

1. Hover over the rule row in the list to reveal the **Delete** icon on the right.
2. Click the **Delete** icon.
3. In the confirmation dialog, type the rule name to confirm deletion, then click **Delete**.

Deleting a rule stops all future notifications for that rule. Historical alert data is not affected.

## Rule status reference

| Status          | Description                                                                                |
| :-------------- | :----------------------------------------------------------------------------------------- |
| **Ready**       | The rule is active and monitoring.                                                         |
| **Reconciling** | The rule was just created or updated and is being applied.                                 |
| **Not Ready**   | The rule encountered an error during setup. Contact StreamNative support if this persists. |


# AWS Networking Overview on StreamNative Cloud
Source: https://docs.streamnative.io/cloud/networking/networking-on-aws/aws-networking-overview



StreamNative Cloud supports the public and private networking solutions on AWS.

## Public networking solutions

StreamNative Cloud offers data streaming services that can be shared across organizations over the secure public endpoints. StreamNative Cloud services include the public connectivity for all cluster types.

All connections to public endpoints on StreamNative Cloud are encrypted using TLS 1.2 and require authentication using OAuth2 or API keys, regardless of network configuration.

StreamNative Cloud clusters with secure public endpoints are protected by a proxy layer that prevents types of DoS, DDoS, syn flooding, and other network-level attacks.

## Private networking solutions

StreamNative Cloud supports data streaming services that are shared privately with organizations on private networks and offers additional customization and controls for security and privacy.

StreamNative Cloud currently only supports private networking for BYOC and BYOC Pro clusters with these networking solutions:

| Supported Networking Solutions                                                                  | Cluster Type                   |
| ----------------------------------------------------------------------------------------------- | ------------------------------ |
| [AWS PrivateLink](/cloud/networking/networking-on-aws/aws-privatelink/aws-privatelink-overview) | BYOC Cluster, BYOC Pro Cluster |
| AWS VPC peering                                                                                 | BYOC Pro Cluster               |
| AWS Transit Gateway                                                                             | BYOC Pro Cluster               |

Private networking solutions are not supported for **Serverless** and **Dedicated** clusters.


# Use AWS Inbound PrivateLink with StreamNative BYOC Clusters
Source: https://docs.streamnative.io/cloud/networking/networking-on-aws/aws-privatelink/aws-inbound-privatelink-byoc



[AWS PrivateLink](https://aws.amazon.com/privatelink/) enables secure, one-way connection access from your application VPC to a StreamNative Managed VPC in StreamNative Cloud, providing added protection against data exfiltration. This networking option is popular for its unique combination of security and simplicity.

The following diagram summarizes the AWS PrivateLink architecture between your application VPC and a StreamNative Managed VPC within your BYOC AWS account.

<img alt="AWS Inbound PrivateLink with StreamNative BYOC Clusters" />

To set up to use AWS Inbound PrivateLink with your BYOC Cluster, follow the instructions below.

1. Review the [requirements and considerations](#requirements-and-considerations) below.

2. Ensure you [provision a BYOC Cloud Environment with proper networking configuration](#provision-byoc-cloud-environment) at the time of provisioning.

3. [Get the VPC Endpoint Service Name](#get-vpc-endpoint-service-name) of your BYOC Cluster.

4. [Provision PrivateLink endpoints in your AWS account that runs your application](#provision-private-link-endpoints-in-aws).

## Requirements and considerations

Review the following requirements and considerations before you set up an Inbound PrivateLink in AWS with your BYOC Clusters:

* The AWS Inbound PrivateLink described in this document is only available for use with BYOC & BYOC Pro clusters.

* If you are using [OAuth2 authentication](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview), your VPC must allow outbound internet connections to [Auth0](https://auth0.com/), StreamNative's OAuth2 service provider. [API Keys](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview) authentication doesn't require this restriction.

* The default gateway type for BYOC [Cloud Environment](/cloud/clusters/byoc/create-cloud-environment) can be either public or private. But you can't switch the gateway type after the BYOC Cloud Environment is created. If you need to switch the gateway type, you have to re-provision a new BYOC Cloud Environment with the desired gateway type.

## Provision BYOC Cloud Environment

StreamNative Clusters are exposed to external networks through a gateway service. Each BYOC [Cloud Environment](/cloud/clusters/byoc/create-cloud-environment) is provisioned with a default gateway type, either public or private. You can't switch the gateway type after the Cloud Environment is created. If you need to switch the gateway type, you have to re-provision a new Cloud Environment with the desired gateway type.

So in order to set up AWS Inbound PrivateLink for your BYOC Cluster, you must provision a BYOC Cloud Environment with the right settings at the time of provisioning:

1. Set the default gateway type to **private**.
2. Add the AWS Account ID where your application VPC is located to the **allowed IDs list** of the private service.

<Tabs>
  <Tab title="Console">
    When [creating a BYOC Cloud Environment](/cloud/clusters/byoc/create-cloud-environment#create-a-cloud-environment-on-ui) on the StreamNative Cloud Console, make sure to select **private** as the **Default Gateway** type and input the AWS Account ID where your application VPC is located in the **Allowed IDs** field. See the screenshot below for reference.

    <img alt="Configure Cloud Environment with Private Gateway" />
  </Tab>

  <Tab title="snctl">
    When [creating a BYOC Cloud Environment](/cloud/clusters/byoc/create-cloud-environment#create-a-cloud-environment-with-snctl) with `snctl`, you need to set `spec.defaultGateway.access` to `private` and add the AWS Account ID where your application VPC is located to the `spec.defaultGateway.privateService.allowedIds` field when you prepare the YAML manifest file for the Cloud Environment. The example YAML manifest file is as follows:

    ```yaml theme={null}
    apiVersion: cloud.streamnative.io/v1alpha1
    kind: CloudEnvironment
    metadata:
      name: <your-cloud-environment-name>
      namespace: <your-namespace>
    spec:
      cloudConnectionName: <your-aws-connection-name>
      defaultGateway:
        # Set the default gateway type to private
        access: private
        privateService:
          # Add the AWS Account ID where your application VPC is located
          allowedIds:
            - <your-aws-account-id>
      network:
        cidr: <your-cidr>
      region: <your-region>
    # ...
    ```
  </Tab>

  <Tab title="Terraform">
    When [creating a BYOC Cloud Environment](/cloud/clusters/byoc/create-cloud-environment#create-a-cloud-environment-with-terraform) with Terraform, you need to set `default_gateway.access` to `private` and add the AWS Account ID where your application VPC is located to the `default_gateway.private_service.allowed_ids` field when you prepare the Terraform configuration file. The example Terraform configuration file is as follows:

    ```hcl theme={null}
    resource "streamnative_cloud_environment" "your_environment" {
        organization = <your-organization-id>
        region = <your-region>
        cloud_connection_name = <your-byoc-cloud-connection-name>
        environment_type = "production"

        network {
          cidr = <your-cidr>
        }
        default_gateway {
          access = "private"
          private_service {
            allowed_ids = [<your-aws-account-id>]
          }
        }
    }
    ```
  </Tab>
</Tabs>

## Get the VPC Endpoint Service Name

Before you can provision PrivateLink endpoints in your AWS account, you need to get the VPC Endpoint Service Name of your BYOC Cluster. You can get the **VPC Endpoint Service Name** from the StreamNative Cloud Console or `snctl`.

<Tabs>
  <Tab title="Console">
    1. Navigate to the **Cloud Environments** page in the StreamNative Cloud Console.

    2. Find the BYOC Cloud Environment that you want to set up PrivateLink for.

    3. You will find your **VPC Endpoint Service Name** under the column **Default gateway** and click the copy icon to copy the value.
  </Tab>

  <Tab title="snctl">
    You can get the **VPC Endpoint Service Name** of your BYOC Cluster by running the following `snctl` command:

    ```bash theme={null}
    snctl get cloudenvironment -O <your-organization-id> <your-cloud-environment-name> --output jsonpath='{.status.defaultGateway.privateServiceIds}'
    ```

    You will get the output like the following:

    ```
    [{"id":"<vpc-endpoint-service-name>"}]
    ```

    Copy the value of the `id` field, which is the **VPC Endpoint Service Name** of your BYOC Cluster.
  </Tab>
</Tabs>

## Provision PrivateLink endpoints in AWS

After your BYOC Cloud Environment is ready, you can create the StreamNative [Instance](/cloud/clusters/manage-instances/instance) and [Cluster](/cloud/clusters/manage-clusters/cluster). The cluster will expose its services through the private gateway. To access these services, you'll need to provision a VPC private endpoint in your application VPC within your AWS account. This endpoint will establish the AWS PrivateLink connection to your StreamNative BYOC cluster.

For the current process to create VPC private endpoints, refer to [Create a VPC endpoint](https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html#create-interface-endpoint-aws).

<Note title="Note">
  StreamNative recommends using a [Terraform module](https://github.com/streamnative/terraform-managed-cloud/tree/main/modules/aws/private-link) for setting up Private Link endpoints. This configuration automates the manual steps described below.
</Note>

<Tabs>
  <Tab title="AWS Console">
    [AWS VPC dashboard](https://console.aws.amazon.com/vpc/home)

    1. Open the [AWS VPC Console](https://console.aws.amazon.com/vpc/home) and browse to the VPC you want to use for the PrivateLink connection.

    2. Verify subnet availability in your AWS VPC, and confirm the selected subnets match the availability zones of StreamNative BYOC Clusters that you created in the previous steps.

    <Note title="Important">
      The zones for the StreamNative BYOC VPC and cluster must match the zones of the VPC you want to make the AWS PrivateLink connections from. Have the matching subnets in your VPC for those zones so that IP addresses can be allocated fromthem.
    </Note>

    3. Verify that **Enable DNS resolution** and **Enable DNS hostnames** are enabled.

       If the settings are not enabled, click **Actions > VPC settings**, and enable **Enable DNS resolution** and **Enable DNS hostnames** settings.

    4. Create or edit a security group you want to use for the new VPC endpoint.

       * Add four inbound rules for each of ports `443`, `6651`, `9093`, and `8883` from your desired source (your VPC CIDR). The **Protocol** should be **TCP** for all four rules.

    5. Create a VPC endpoint.

       1. In the navigation menu under **Virtual Private Cloud**, click **Endpoints**.

       2. Click **Create endpoint**, and specify the following settings for the endpoint:

          * **Service category**: Select **Endpoint services that use NLBs and GWLBs**.

          * **Service settings**: Enter the **Service name** for your BYOC Cluster **VPC Endpoint service name**, that you noted in the step 1.

            Click **Verify service**. If you get an error, ensure that your account is allowed to create PrivateLink connections.

          * **VPC**: Select the VPC in which to create your endpoint.

          * **Subnets**: Select the subnet for the availability zones for your BYOC Cluster.

            Ensure that the desired subnet is selected for each zone. By default, a BYOC cluster is a regional cluster, which means it spans all availability zones in the region. Make sure to add all availability zones of that region to the subnets. Failure to add all zones of your BYOC cluster can cause connectivity issues to brokers in the omitted zones, resulting in an unusable cluster.

          * **Security groups**: Select the security group that you previously created or edited.

          * **Enable Private DNS name**: Make sure you check the box for **Enable Private DNS name**. This step is required to ensure that the private DNS name for the service resolves to the endpoint's private IP address. The private DNS name is automatically associated with the endpoint in your VPC.

       3. Click **Create endpoint**.

          Your VPC endpoint is created and displayed. Since your application AWS account is already whitelisted when creating your BYOC environment, the PrivateLink connection is automatically established.
  </Tab>

  <Tab title="Terraform">
    You can use the [Terraform module](https://github.com/streamnative/terraform-managed-cloud/tree/main/modules/aws/private-link) to set up Private Link endpoints.

    Before you start, make sure you already have the **VPC Endpoint Service Name** of your BYOC Cluster from the previous step.

    Below is an example Terraform configuration file:

    ```hcl theme={null}
    module "aws_private_link" {
      source = "github.com/streamnative/terraform-managed-cloud//modules/aws/private-link?ref=main"

      region       = "<the region of VPC Endpoint Service>"
      vpc_id       = "<the VPC ID where the PrivateLink endpoint will be created>"
      subnet_ids   = ["<the subnet IDs in which to create the network interface for this endpoint>"]
      service_name = "<the VPC Endpoint Service Name of your BYOC Cluster>"
    }
    ```
  </Tab>
</Tabs>

Till this point, your cluster is now ready to connect from your application VPC via PrivateLink. If you encounter any problem, you can reach out to [StreamNative Support](https://support.streamnative.io/) for help.


# AWS PrivateLink Overview in StreamNative Cloud
Source: https://docs.streamnative.io/cloud/networking/networking-on-aws/aws-privatelink/aws-privatelink-overview



[AWS PrivateLink](https://aws.amazon.com/privatelink/) provides a one-way secure connection between **your VPC** (VPC running your applications) and the **StreamNative Managed VPC** (VPC running StreamNative Cloud clusters) in StreamNative Cloud, with added protection against data exfiltration. This networking option is popular for its unique combination of security and simplicity.

## Inbound PrivateLink

Inbound PrivateLink is used for your applications running in your VPC to securely connect to StreamNative Cloud clusters in a **StreamNative Managed VPC** via a private network.

Currently, StreamNative Cloud supports inbound PrivateLink for **BYOC Clusters** and **BYOC Pro Clusters**.

See [AWS Inbound PrivateLink for BYOC Clusters](/cloud/networking/networking-on-aws/aws-privatelink/aws-inbound-privatelink-byoc) for details.

## Outbound PrivateLink

Outbound PrivateLink is typically required by functions and connectors to access external data sources, sinks, and other services that are not in the same VPC as the StreamNative Managed VPC.

However, StreamNative Cloud doesn't support **outbound PrivateLink** from StreamNative Managed VPC to your Application VPC at this time. This feature will be supported in the future.


# Use Public Connectivity for StreamNative Cloud Clusters on AWS
Source: https://docs.streamnative.io/cloud/networking/networking-on-aws/aws-public-networking



StreamNative Cloud offers data streaming services, such as Pulsar, Kafka, Functions, Connectors, Schema Registry, and Audit Logs, that can be shared across organizations over the internet on AWS. StreamNative Cloud services include internet connectivity for the all cluster types.

## Ingress IP addresses

Because the cloud infrastructure used by StreamNative Cloud does not guarantee static IP addresses for ingress public endpoints, such as for Pulsar brokers, Pulsar Admin API, and Metrics API, DNS is used to provide a consistent address. The underlying IP addresses might be stable for a period of time, but are subject to change at any time, and they can assume any public IP the cloud provider uses in the region where the cluster is located, so they should not be relied upon for any use.

StreamNative Cloud does not provide static public ingress IP addresses. Instead, DNS resolution is used to provide consistent endpoints for each Pulsar cluster. The DNS names follow a predictable pattern based on your cluster configuration.

For example, if your cluster domain is `pc-9293889f.<pool-name>.<cloud>.snio.cloud` (where `pc-9293889f` is your unique cluster ID), and you are using the default broker prefix, all broker endpoints will follow a standardized format as shown in the examples below.

If the cluster is a Class-Engine cluster, the broker endpoints will be of the following format:

```
pb0-pc-9293889f.<pool-name>.<cloud>.snio.cloud
pb1-pc-9293889f.<pool-name>.<cloud>.snio.cloud
pb2-pc-9293889f.<pool-name>.<cloud>.snio.cloud
...
```

If the cluster is a Ursa-Engine cluster, the broker endpoints include the availability zone name in the DNS name for zone-aware routing. For example:

```
pb0-pc-9293889f.<zone-name>.<pool-name>.<cloud>.snio.cloud
pb1-pc-9293889f.<zone-name>.<pool-name>.<cloud>.snio.cloud
pb2-pc-9293889f.<zone-name>.<pool-name>.<cloud>.snio.cloud
...
```

The following blogs describe how the common outbound proxies handle IP address changes:

* [DNS for Service Discovery in HAProxy - HAProxy Technologies](https://www.haproxy.com/blog/dns-service-discovery-haproxy/).
* [DNS for Service Discovery with NGINX and NGINX Plus](https://www.nginx.com/blog/dns-service-discovery-nginx-plus/).

## Egress IP addresses

StreamNative Cloud doesn't provide egress public IP addresses that you can use for communicating between StreamNative clusters (within public networking) in StreamNative Cloud and external data sources and sinks. This feature will be supported in the future.


# Test Connectivity to StreamNative Cloud
Source: https://docs.streamnative.io/cloud/networking/networking-testing



Brokers hosted on StreamNative Cloud do not respond to `ping` requests. Instead, you can use the methods presented in this guide to test the connectivity to your StreamNative Cloud cluster and its endpoints before whitelisting them.

Run through the following steps to validate StreamNative Cloud connectivity is working as expected.

## Test connectivity to StreamNative Cloud

You can test connectivity to any StreamNative Cloud cluster using `openssl`, `Netcat`, or `Telnet`.

For clusters with public endpoints, you can run connectivity tests from any computer with internet access.

For clusters in private network environments (such as PrivateLink, Private Service Connect, VPC peering, VNet peering, and AWS Transit Gateway), run tests from within your VPC or VNet that is connected to the StreamNative Cloud cluster.

* For PrivateLink, ensure enabling Private DNS name when setting up the Private Endpoint in your VPC. See [Inbound PrivateLink for BYOC Clusters](id:inbound-private-link-for-byoc-clusters) for more details.

All the services of a StreamNative Cloud cluster share the same DNS name, but use different ports:

* Use port `443` to test the connection to the Pulsar HTTP service, Websocket service, REST messaging service, and the Kafka schema registry service.
* Use port `6651` to test the connection to the Pulsar Broker service.
* Use port `9093` to test the connection to the Kafka Broker service.
* Use port `8883` to test the connection to the MQTT service.

To test the connection to the Kafka Broker and MQTT service, ensure that both Kafka and MQTT protocols are enabled on your cluster. These protocols are enabled by default on new clusters. However, for older clusters, you may need to enable them manually.

To only test TCP connectivity, use `Telnet` or `Netcat`:

* **Netcat**

  ```bash theme={null}
  nc -zv <cluster-endpoint> 443
  nc -zv <cluster-endpoint> 6651
  nc -zv <cluster-endpoint> 9093
  nc -zv <cluster-endpoint> 8883
  ```

* **Telnet**

  ```bash theme={null}
  telnet <cluster-endpoint> 443
  telnet <cluster-endpoint> 6651
  telnet <cluster-endpoint> 9093
  telnet <cluster-endpoint> 8883
  ```

In addition to TCP connectivity, to also test the TLS handshake and the certificate, use `openssl`. With `openssl`, you can an SNI header:

* **OpenSSL**

  ```bash theme={null}
  openssl s_client -servername <cluster-endpoint> -connect <cluster-endpoint>:443
  openssl s_client -servername <cluster-endpoint> -connect <cluster-endpoint>:6651
  openssl s_client -servername <cluster-endpoint> -connect <cluster-endpoint>:9093
  openssl s_client -servername <cluster-endpoint> -connect <cluster-endpoint>:8883
  ```

  For more details, see the [OpenSSL documentation](https://www.openssl.org/docs/man3.0/man1/openssl-s_client.html) for the `-connect` option.

It is recommended that you use `openssl` to test TCP and TLS because with the TCP testing only, it is difficult to make the distinction among the various cases when a connection fails, such as:

* Timeout because of routing problems.
* Established connection, but you as the client not initiating the TLS handshake.
* Envoy disconnecting your connection because you do not send the SNI header.

  For Kafka clients, see the [TLS SNI extension requirements](/cloud/build/kafka-clients/compatibility/kafka-compatibility#tls-sni-extension-requirements) for more details.

## Test connectivity using Pulsar or Kafka tools

After connectivity is successfully established, you can use the Pulsar or Kafka clients and/or tools to test producing/consuming messages. Examples are `pulsar-client`, `kcat`, `kafkacat`, `kafka-console-consumer`, `kafka-console-producer`, native command line tools, or Java and other clients. The following are a few of the test workflows you can use as references:

* For using the Pulsar clients to produce and consume messages, see [Pulsar Client Guides](/clients/pulsar-clients/pulsar-clients-overview)
* For using the Kafka clients to produce and consume messages, see [Kafka Client Guides](/clients/kafka-clients/kafka-clients-overview)
* For using the Pulsar CLI tools to produce and consume messages, see [Use Pulsar Tools with StreamNative Cloud](/tools/cli/other-tools/use-pulsar-tools-with-streamnative-cloud)
* For using the Kafka CLI tools to produce and consume messages, see [Use Kafka Tools with StreamNative Cloud](/tools/cli/other-tools/use-kafka-tools-with-streamnative-cloud)

## Troubleshoot connectivity issues

If connectivity to the cluster endpoint cannot be established, first check your firewall and other security configurations and restrictions that could prevent the connection to the StreamNative Cloud cluster endpoint.

Here are some common issues and their troubleshooting steps:

### Failed connection to brokers in private link cluster

**Issue**: You got an error message similar to the following from a Kafka client over a private link, such as produce or consume messages:

```
SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 0ms in state SSL_HANDSHAKE)
```

**Possible causes**:

* The private endpoint in use does not correspond to the correct availability zone.
* Zone affinity between your application VPC and StreamNative managed VPC is not respected. See [Inbound PrivateLink for BYOC Clusters](/cloud/networking/networking-on-aws/aws-privatelink/aws-inbound-privatelink-byoc) for more details.
* The private DNS name is not enabled when setting up the private endpoint in your VPC.


# Generate a HAR file for Troubleshooting
Source: https://docs.streamnative.io/cloud/references/generate-har-file



To help with debugging issues on StreamNative Cloud, you can generate an HTTP Archive (HAR) file by using the developer tools in your browser.

The HAR file provides information about the network requests that are generated in your browser while you interact with StreamNative Cloud Console.

<Note title="Important">
  A HAR file includes data such as the content of your cookies and the pages that you downloaded while making the recording. Anyone with access to the HAR file can view the data submitted while recording, which may include personal information or other sensitive data. Ensure that you secure your HAR files accordingly.
</Note>

The following sections show how to generate a HAR file in popular browsers.

## Google Chrome and Microsoft Edge

1. Open Chrome or Edge and log in to StreamNative Cloud Console at [https://console.streamnative.cloud](https://console.streamnative.cloud).

2. Navigate to the page where you are experiencing an issue.

3. In the browser toolbar, click `⋮` or `⋯` and navigate to **More Tools > Developer Tools**.

4. In the Developer Tools panel, click **Network**. You must keep the Network panel open while you reproduce the issue.

5. Ensure that the record button is red, and the network log is being recorded. If the button is gray, click it to start recording.

6. Click **Preserve log**.

7. Click the **Clear network log** button to remove any existing logs from the Network tab.

8. In the StreamNative Cloud Console, navigate to the page where the issue is occurring. Reproduce the issue while the network requests are recorded.

9. After you have reproduced the issue, click the **Export HAR** button to save the file to your computer.

## Mozilla Firefox

1. Open Firefox and log in to StreamNative Cloud Console at [https://console.streamnative.cloud](https://console.streamnative.cloud).

2. Click the **Application menu** and navigate to **More tools > Web developer tools**.

3. Click **Network**.

4. In the StreamNative Cloud Console, navigate to the page where the issue is occurring. Reproduce the issue while the network requests are recorded.

5. After you have reproduced the issue, click **Network Settings** and select **Save All as HAR** to save the file to your computer.

## Apple Safari

1. Open Safari and log in to StreamNative Cloud Console at [https://console.streamnative.cloud](https://console.streamnative.cloud).

2. Click the Develop menu. If you don’t see the Develop menu, follow the instructions in this article from the Safari User Guide: [Use the developer tools in the Develop menu in Safari on Mac](https://support.apple.com/en-ie/guide/safari/sfri20948/mac).

3. Select **Show Web Inspector**.

4. Click the **Network** tab. You must keep it open while you reproduce the issue.

5. In the StreamNative Cloud Console, navigate to the page where the issue is occurring. Reproduce the issue while the network requests are recorded.

6. After you have reproduced the issue, click **Export** to save the file to your computer.


# Io activemq source
Source: https://docs.streamnative.io/connect/connectors/activemq-source/current/io-activemq-source

ActiveMQ Connector integrates Apache Pulsar with Apache ActiveMQ.

The ActiveMQ source connector receives messages from ActiveMQ clusters and writes messages to Pulsar topics.

# Installation

```
git clone https://github.com/streamnative/pulsar-io-activemq.git
cd pulsar-io-activemq/
mvn clean install -DskipTests
cp target/pulsar-io-activemq-0.0.1.nar $PULSAR_HOME/pulsar-io-activemq-0.0.1.nar
```

# Configuration

The configuration of the ActiveMQ source connector has the following properties.

## ActiveMQ source connector configuration

| Name        | Type   | Required | Sensitive | Default            | Description                                                              |
| ----------- | ------ | -------- | --------- | ------------------ | ------------------------------------------------------------------------ |
| `protocol`  | String | true     | false     | "tcp"              | The ActiveMQ protocol.                                                   |
| `host`      | String | true     | false     | " " (empty string) | The ActiveMQ host.                                                       |
| `port`      | int    | true     | false     | 5672               | The ActiveMQ port.                                                       |
| `username`  | String | false    | true      | " " (empty string) | The username used to authenticate to ActiveMQ.                           |
| `password`  | String | false    | true      | " " (empty string) | The password used to authenticate to ActiveMQ.                           |
| `queueName` | String | false    | false     | " " (empty string) | The ActiveMQ queue name that messages should be read from or written to. |
| `topicName` | String | false    | false     | " " (empty string) | The ActiveMQ topic name that messages should be read from or written to. |

## Configure ActiveMQ source connector

Before using the ActiveMQ source connector, you need to create a configuration file through one of the following methods.

* JSON

  ```json theme={null}
  {
      "tenant": "public",
      "namespace": "default",
      "name": "activemq-source",
      "topicName": "user-op-queue-topic",
      "archive": "connectors/pulsar-io-activemq-2.5.1.nar",
      "parallelism": 1,
      "configs": {
          "protocol": "tcp",
          "host": "localhost",
          "port": "61616",
          "username": "admin",
          "password": "admin",
          "queueName": "user-op-queue"
      }
  }
  ```

* YAML

  ```yaml theme={null}
  tenant: "public"
  namespace: "default"
  name: "activemq-source"
  topicName: "user-op-queue-topic"
  archive: "connectors/pulsar-io-activemq-2.5.1.nar"
  parallelism: 1

  configs:
      protocol: "tcp"
      host: "localhost"
      port: "61616"
      username: "admin"
      password: "admin"
      queueName: "user-op-queue"
  ```

1. Prepare ActiveMQ service.

   ```
   docker pull rmohr/activemq
   docker run -p 61616:61616 -p 8161:8161 rmohr/activemq
   ```

2. Put the `pulsar-io-activemq-2.5.1.nar` in the pulsar connectors catalog.

   ```
   cp pulsar-io-activemq-2.5.1.nar $PULSAR_HOME/connectors/pulsar-io-activemq-2.5.1.nar
   ```

3. Start Pulsar in standalone mode.

   ```
   $PULSAR_HOME/bin/pulsar standalone
   ```

4. Run ActiveMQ source locally.

   ```
   $PULSAR_HOME/bin/pulsar-admin source localrun --source-config-file activemq-source-config.yaml
   ```

5. Consume Pulsar messages.

   ```
   bin/pulsar-client consume -s "sub-products" public/default/user-op-queue-topic -n 0
   ```

6. Send ActiveMQ messages.

   Use the test method `sendMessage` of the `class org.apache.pulsar.ecosystem.io.activemq.ActiveMQDemo`
   to send ActiveMQ messages.

   ```
   @Test
   private void sendMessage() throws JMSException {

       ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory("tcp://localhost:61616");

       @Cleanup
       Connection connection = connectionFactory.createConnection();
       connection.start();

       @Cleanup
       Session session = connection.createSession(false, Session.CLIENT_ACKNOWLEDGE);

       Destination destination = session.createQueue("user-op-queue");

       @Cleanup
       MessageProducer producer = session.createProducer(destination);
       producer.setDeliveryMode(DeliveryMode.NON_PERSISTENT);

       for (int i = 0; i < 10; i++) {
           String msgContent = "Hello ActiveMQ - " + i;
           ActiveMQTextMessage message = new ActiveMQTextMessage();
           message.setText(msgContent);
           producer.send(message);
       }
   }
   ```


# Amqp 1 0 source
Source: https://docs.streamnative.io/connect/connectors/amqp-1-0-source/current/amqp-1-0-source

support sink/source for AMQP version 1.0.0

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

# AMQP 1.0 source connector

The AMQP 1.0 source connector receives messages from [AMQP 1.0](https://www.amqp.org/) and writes messages to Pulsar topics.

<img alt="" />

## Quick start

### 1. Start AMQP 1.0 service

Start a service that supports the AMQP 1.0 protocol, such as [Solace](https://docs.solace.com/index.html).

```bash theme={null}
docker run -d -p 8080:8080 -p:8008:8008 -p:1883:1883 -p:8000:8000 -p:5672:5672 -p:9000:9000 -p:2222:2222 --shm-size=2g --env username_admin_globalaccesslevel=admin --env username_admin_password=admin --name=solace solace/solace-pubsub-standard
```

### 2. Create a connector

The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector,
you need to replace `--source-type amqp1_0` with `--archive /path/to/pulsar-io-amqp1_0.nar`. You can find the button to download the `nar` package at the beginning of the document.

<Note title="For StreamNative Cloud User">
  If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first.
</Note>

```bash theme={null}
pulsarctl sources create \
  --source-type amqp1_0 \
  --name amqp1_0-source \
  --tenant public \
  --namespace default \
  --destination-topic-name "Your topic name" \
  --parallelism 1 \
  --source-config \
  '{
    "connection": {
      "failover": {
        "useFailover": true
      },
      "uris": [
        {
          "protocol": "amqp",
          "host": "localhost",
          "port": 5672,
          "urlOptions": [
            "transport.tcpKeepAlive=true"
          ]
        }
      ]
    },
    "username": "guest",
    "password": "guest",
    "queue": "user-op-queue-pulsar"
  }'
```

The `--source-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.
If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference.

<Note title="Note">
  You can also choose to use a variety of other tools to create a connector:

  * [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [RestAPI](https://pulsar.apache.org/source-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document.
</Note>

### 2. Send messages to the AMQP 1.0 service

<Note title="Note">
  * The following sample code uses the **Apache qpid** library.
</Note>

```java theme={null}
    public static void main(String[] args) {
        ConnectionFactory connectionFactory = new JmsConnectionFactory("amqp://localhost:5672");
        Connection connection = connectionFactory.createConnection();
        connection.start();
        JMSProducer producer = connectionFactory.createContext().createProducer();
        producer.setDeliveryMode(DeliveryMode.NON_PERSISTENT);
        Destination destination = new JmsQueue("user-op-queue");
        for (int i = 0; i < 10; i++) {
            producer.send(destination, "Hello AMQP 1.0 - " + i);
        }
        connection.close();
    }
```

### 3. Consume data from Pulsar

<Note title="Note">
  * If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information.
</Note>

```java theme={null}
bin/pulsar-client \
--url "Your Pulsar serviceUrl" \
consume "The topic that you specified when you created the connector" -s "test-sub" -n 10 -p Earliest
```

## Configuration Properties

Before using the AMQP 1.0 sink connector, you need to configure it.

You can create a configuration file (JSON or YAML) to set the following properties.

| Name                | Type       | Required                                     | Sensitive | Default             | Description                                                                                                                                   |
| ------------------- | ---------- | -------------------------------------------- | --------- | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `protocol`          | String     | required if connection is not used           | false     | "amqp"              | \[deprecated: use connection instead] The AMQP protocol.                                                                                      |
| `host`              | String     | required if connection is not used           | false     | " " (empty string)  | \[deprecated: use connection instead] The AMQP service host.                                                                                  |
| `port`              | int        | required if connection is not used           | false     | 5672                | \[deprecated: use connection instead] The AMQP service port.                                                                                  |
| `connection`        | Connection | required if protocol, host, port is not used | false     | " "  (empty string) | The connection details.                                                                                                                       |
| `username`          | String     | false                                        | true      | " " (empty string)  | The username used to authenticate to ActiveMQ.                                                                                                |
| `password`          | String     | false                                        | true      | " " (empty string)  | The password used to authenticate to ActiveMQ.                                                                                                |
| `queue`             | String     | false                                        | false     | " " (empty string)  | The queue name that messages should be read from or written to.                                                                               |
| `topic`             | String     | false                                        | false     | " " (empty string)  | The topic name that messages should be read from or written to.                                                                               |
| `activeMessageType` | String     | false                                        | false     | 0                   | The ActiveMQ message simple class name.                                                                                                       |
| `onlyTextMessage`   | boolean    | false                                        | false     | false               | If it is set to `true`, the AMQP message type must be set to `TextMessage`. Pulsar consumers can consume the messages with schema ByteBuffer. |

A `Connection` object can be specified as follows:

| Name       | Type                  | Required | Default            | Description                                                                                                                                                       |
| ---------- | --------------------- | -------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `failover` | Failover              | false    | " " (empty string) | The configuration for a failover connection.                                                                                                                      |
| `uris`     | list of ConnectionUri | true     | " " (empty string) | A list of ConnectionUri objects. When useFailover is set to true 1 or more should be provided. Currently only 1 uri is supported when useFailover is set to false |

A `Failover` object can be specified as follows:

| Name                           | Type           | Required                                         | Default            | Description                                                                                                                                                                                                                                                                                                                                                         |
| ------------------------------ | -------------- | ------------------------------------------------ | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `useFailover`                  | boolean        | true                                             | false              | If it is set to true, the connection will be created from the uris provided under uris, using qpid's failover connection factory.                                                                                                                                                                                                                                   |
| `jmsClientId`                  | String         | required if failoverConfigurationOptions is used | " " (empty string) | Identifying name for the jms Client                                                                                                                                                                                                                                                                                                                                 |
| `failoverConfigurationOptions` | List of String | required if jmsClientId is used                  | " " (empty string) | A list of options (e.g. `<key=value>`). The options wil be joined using an '&', prefixed with a the jmsClientId and added to the end of the failoverUri. see also: [https://qpid.apache.org/releases/qpid-jms-2.2.0/docs/index.html#failover-configuration-options](https://qpid.apache.org/releases/qpid-jms-2.2.0/docs/index.html#failover-configuration-options) |

A `ConnectionUri` object can be specified as follows:

| Name         | Type           | Required | Default            | Description                                                                                                                                 |
| ------------ | -------------- | -------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `protocol`   | String         | true     | " " (empty string) | The AMQP protocol.                                                                                                                          |
| `host`       | String         | true     | " " (empty string) | The AMQP service host.                                                                                                                      |
| `port`       | int            | true     | 0                  | The AMQP service port.                                                                                                                      |
| `urlOptions` | List of String | false    | " " (empty string) | A list of url-options (e.g. `<key=value>`). The url options wil be joined using an '&', prefixed with a '?' and added to the end of the uri |


# StreamNative Connector Hub
Source: https://docs.streamnative.io/connect/overview

Discover and deploy connectors to integrate StreamNative with hundreds of data sources and sinks.

## What are Connectors?

StreamNative connectors enable you to easily integrate with external systems, allowing you to move data into and out of your Pulsar clusters. Connectors are available in two main categories:

* **Source Connectors**: Pull data from external systems into Pulsar topics
* **Sink Connectors**: Push data from Pulsar topics to external systems

## Getting Started

<CardGroup>
  <Card title="Pulsar IO Overview" icon="info" href="/cloud/connect/pulsar-io/connector-overview">
    Learn the fundamentals of Pulsar IO connectors and how they work.
  </Card>

  <Card title="Deploy Connectors" icon="rocket" href="/cloud/connect/pulsar-io/deploy-connectors/deploy-connector-index">
    Step-by-step guide to deploy your first connector.
  </Card>

  <Card title="Manage Connectors" icon="settings" href="/cloud/connect/pulsar-io/connector-manage">
    Learn how to manage, update, and troubleshoot your connectors.
  </Card>

  <Card title="Monitor Connectors" icon="chart-line" href="/cloud/connect/pulsar-io/connector-monitoring">
    Set up monitoring and troubleshooting for your connectors.
  </Card>
</CardGroup>

## Kafka Connect

<CardGroup>
  <Card title="Kafka Connect Overview" icon="link" href="/cloud/connect/kafka-connect/kafka-connect-overview">
    Use Kafka Connect framework with StreamNative Cloud.
  </Card>

  <Card title="Deploy Kafka Connectors" icon="rocket" href="/cloud/connect/kafka-connect/deploy-kafka-connectors/deploy-kafka-connect-index">
    Deploy and manage Kafka Connect connectors.
  </Card>
</CardGroup>

## Available Connectors

Browse our comprehensive library of pre-built connectors for popular data sources and destinations.

### Source Connectors

| Connector                             | Type              | Description                                                                            | Documentation                                                                                                          |
| ------------------------------------- | ----------------- | -------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| **ActiveMQ**                          | Message Queue     | Integrate with Apache ActiveMQ message broker                                          | [📖 Docs](/connect/connectors/activemq-source/current/io-activemq-source)                                              |
| **AMQP 1.0**                          | Message Queue     | Connect to AMQP 1.0 compatible message brokers                                         | [📖 Docs](/connect/connectors/amqp-1-0-source/current/amqp-1-0-source)                                                 |
| **Canal**                             | Database CDC      | MySQL binlog change data capture using Alibaba Canal                                   | [📖 Docs](/connect/connectors/canal-source/current/canal-source)                                                       |
| **Debezium MongoDB**                  | Database CDC      | MongoDB change data capture using Debezium                                             | [📖 Docs](/connect/connectors/debezium-mongodb-source/current/debezium-mongodb-source)                                 |
| **Debezium SQL Server**               | Database CDC      | SQL Server change data capture using Debezium                                          | [📖 Docs](/connect/connectors/debezium-mssql-source/current/debezium-mssql-source)                                     |
| **Debezium MySQL**                    | Database CDC      | MySQL change data capture using Debezium                                               | [📖 Docs](/connect/connectors/debezium-mysql-source/current/debezium-MySQL-source)                                     |
| **Debezium PostgreSQL**               | Database CDC      | PostgreSQL change data capture using Debezium                                          | [📖 Docs](/connect/connectors/debezium-postgres-source/current/debezium-postgres-source)                               |
| **DynamoDB**                          | Database          | Read data from AWS DynamoDB                                                            | [📖 Docs](/connect/connectors/dynamodb-source/current/dynamodb-source)                                                 |
| **File**                              | File System       | Read data from local or remote files                                                   | [📖 Docs](/connect/connectors/file-source/current/file-source)                                                         |
| **Flume**                             | Data Collection   | Integrate with Apache Flume data collection service                                    | [📖 Docs](/connect/connectors/flume-source/current/flume-source)                                                       |
| **Google BigQuery**                   | Analytics         | Read data from Google BigQuery tables                                                  | [📖 Docs](/connect/connectors/google-bigquery-source/current/google-bigquery-source)                                   |
| **Google Pub/Sub**                    | Message Queue     | Consume messages from Google Cloud Pub/Sub                                             | [📖 Docs](/connect/connectors/google-pubsub-source/current/google-pubsub-source)                                       |
| **Kafka Connect Debezium MongoDB**    | Database CDC      | Debezium MongoDB source connector via Kafka Connect                                    | [📖 Docs](/connect/connectors/kafka-connect-debezium-mongodb/current/kafka-connect-debezium-mongodb)                   |
| **Kafka Connect Debezium Mysql**      | Database CDC      | Debezium Mysql source connector via Kafka Connect                                      | [📖 Docs](/connect/connectors/kafka-connect-debezium-mysql/current/kafka-connect-debezium-mysql)                       |
| **Kafka Connect Debezium PostgreSql** | Database CDC      | Debezium PostgreSql source connector via Kafka Connect                                 | [📖 Docs](/connect/connectors/kafka-connect-debezium-postgresql/current/kafka-connect-debezium-postgresql)             |
| **Kafka Connect Debezium Spanner**    | Database CDC      | Debezium Cloud Spanner source connector via Kafka Connect                              | [📖 Docs](/connect/connectors/kafka-connect-debezium-spanner/current/kafka-connect-debezium-spanner)                   |
| **Kafka Connect Debezium SqlServer**  | Database CDC      | Debezium Sql Server source connector via Kafka Connect                                 | [📖 Docs](/connect/connectors/kafka-connect-debezium-sqlserver/current/kafka-connect-debezium-sqlserver)               |
| **Kafka Connect Cosmos DB**           | Database          | Read data from Azure Cosmos DB change feed via Kafka Connect                           | [📖 Docs](/connect/connectors/kafka-connect-cosmosdb-source/current/kafka-connect-cosmosdb-source)                     |
| **Kafka Connect JDBC**                | Database          | Reads data from any JDBC-compliant database and writes data to Kafka topics            | [📖 Docs](/connect/connectors/kafka-connect-jdbc-source/current/kafka-connect-jdbc-source)                             |
| **Kafka Connect JR**                  | Testing           | Generate test data with JSON Schema Registry                                           | [📖 Docs](/connect/connectors/kafka-connect-jr-source/current/kafka-connect-jr-source)                                 |
| **Kafka Connect MongoDB**             | Database          | MongoDB source connector via Kafka Connect                                             | [📖 Docs](/connect/connectors/kafka-connect-mongo-source/current/kafka-connect-mongodb-source)                         |
| **Kafka Connect Google Pub/Sub**      | Message Queue     | Persists data from Google Cloud Pub/Sub into Apache Kafka topic via Kafka Connect      | [📖 Docs](/connect/connectors/kafka-connect-google-pubsub-source/current/kafka-connect-google-pubsub-source)           |
| **Kafka Connect Google Pub/Sub Lite** | Message Queue     | Persists data from Google Cloud Pub/Sub Lite into Apache Kafka topic via Kafka Connect | [📖 Docs](/connect/connectors/kafka-connect-google-pubsub-lite-source/current/kafka-connect-google-pubsub-lite-source) |
| **YugabyteDB CDC**                    | Database CDC      | YugabyteDB change data capture                                                         | [📖 Docs](/connect/connectors/kafka-connect-yugabyte-cdc-source/current/kafka-connect-yugabyte-cdc-source)             |
| **Kafka**                             | Message Queue     | Pull data from Apache Kafka topics                                                     | [📖 Docs](/connect/connectors/kafka-source/current/kafka-source)                                                       |
| **Kinesis**                           | Stream Processing | Amazon Kinesis data streams integration                                                | [📖 Docs](/connect/connectors/kinesis-source/current/kinesis-source)                                                   |
| **Lakehouse**                         | Data Lake         | Read from lakehouse storage systems                                                    | [📖 Docs](/connect/connectors/lakehouse-source/current/lakehouse-source)                                               |
| **Netty**                             | Network           | TCP/UDP network data ingestion                                                         | [📖 Docs](/connect/connectors/netty-source/current/netty-source)                                                       |
| **RabbitMQ**                          | Message Queue     | Consume from RabbitMQ message broker                                                   | [📖 Docs](/connect/connectors/rabbitmq-source/current/rabbitmq-source)                                                 |
| **Amazon SQS**                        | Message Queue     | Amazon Simple Queue Service integration                                                | [📖 Docs](/connect/connectors/sqs-source/current/sqs-source)                                                           |
| **Twitter Firehose**                  | Social Media      | Real-time Twitter data streaming                                                       | [📖 Docs](/connect/connectors/twitter-firehose-source/current/twitter-firehose-source)                                 |

### Sink Connectors

| Connector                              | Type              | Description                                                                                       | Documentation                                                                                                          |
| -------------------------------------- | ----------------- | ------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| **ActiveMQ**                           | Message Queue     | Send data to Apache ActiveMQ message broker                                                       | [📖 Docs](/connect/connectors/activemq-sink/current/io-activemq-sink)                                                  |
| **Aerospike**                          | Database          | High-performance NoSQL database integration                                                       | [📖 Docs](/connect/connectors/aerospike-sink/current/aerospike-sink)                                                   |
| **AMQP 1.0**                           | Message Queue     | Send to AMQP 1.0 compatible message brokers                                                       | [📖 Docs](/connect/connectors/amqp-1-0-sink/current/amqp-1-0-sink)                                                     |
| **AWS EventBridge**                    | Event Processing  | Amazon EventBridge event bus integration                                                          | [📖 Docs](/connect/connectors/aws-eventbridge-sink/current/aws-eventbridge-sink)                                       |
| **AWS Lambda**                         | Serverless        | Trigger AWS Lambda functions                                                                      | [📖 Docs](/connect/connectors/aws-lambda-sink/current/aws-lambda-sink)                                                 |
| **Amazon S3**                          | Object Storage    | Store data in Amazon S3 buckets                                                                   | [📖 Docs](/connect/connectors/aws-s3-sink/current/aws-s3-sink)                                                         |
| **Azure Blob Storage**                 | Object Storage    | Microsoft Azure blob storage integration                                                          | [📖 Docs](/connect/connectors/azure-blob-storage-sink/current/azure-blob-storage-sink)                                 |
| **Cassandra**                          | Database          | Apache Cassandra distributed database                                                             | [📖 Docs](/connect/connectors/cassandra-sink/current/cassandra-sink)                                                   |
| **Elasticsearch**                      | Search Engine     | Elasticsearch full-text search and analytics                                                      | [📖 Docs](/connect/connectors/elasticsearch-sink/current/elasticsearch-sink)                                           |
| **Flume**                              | Data Collection   | Send data to Apache Flume                                                                         | [📖 Docs](/connect/connectors/flume-sink/current/flume-sink)                                                           |
| **Google BigQuery**                    | Analytics         | Load data into Google BigQuery                                                                    | [📖 Docs](/connect/connectors/google-bigquery-sink/current/google-bigquery-sink)                                       |
| **Google Cloud Storage**               | Object Storage    | Store files in Google Cloud Storage                                                               | [📖 Docs](/connect/connectors/google-cloud-storage-sink/current/google-cloud-storage-sink)                             |
| **Google Pub/Sub**                     | Message Queue     | Publish to Google Cloud Pub/Sub                                                                   | [📖 Docs](/connect/connectors/google-pubsub-sink/current/google-pubsub-sink)                                           |
| **HBase**                              | Database          | Apache HBase distributed database                                                                 | [📖 Docs](/connect/connectors/hbase-sink/current/hbase-sink)                                                           |
| **HDFS3**                              | File System       | Hadoop Distributed File System v3                                                                 | [📖 Docs](/connect/connectors/hdfs3-sink/current/hdfs3-sink)                                                           |
| **InfluxDB**                           | Time Series       | Time series database for metrics and events                                                       | [📖 Docs](/connect/connectors/influxdb-sink/current/influxdb-sink)                                                     |
| **JDBC ClickHouse**                    | Analytics         | ClickHouse columnar database via JDBC                                                             | [📖 Docs](/connect/connectors/jdbc-clickhouse-sink/current/jdbc-sink)                                                  |
| **JDBC MariaDB**                       | Database          | MariaDB relational database via JDBC                                                              | [📖 Docs](/connect/connectors/jdbc-mariadb-sink/current/jdbc-sink)                                                     |
| **JDBC PostgreSQL**                    | Database          | PostgreSQL relational database via JDBC                                                           | [📖 Docs](/connect/connectors/jdbc-postgres-sink/current/jdbc-sink)                                                    |
| **JDBC SQLite**                        | Database          | SQLite embedded database via JDBC                                                                 | [📖 Docs](/connect/connectors/jdbc-sqlite-sink/current/jdbc-sink)                                                      |
| **Kafka Connect BigQuery**             | Analytics         | BigQuery integration via Kafka Connect                                                            | [📖 Docs](/connect/connectors/kafka-connect-bigquery/current/kafka-connect-bigquery)                                   |
| **Kafka Connect Cosmos DB**            | Database          | Write data to Azure Cosmos DB via Kafka Connect                                                   | [📖 Docs](/connect/connectors/kafka-connect-cosmosdb-sink/current/kafka-connect-cosmosdb-sink)                         |
| **Kafka Connect DataGen**              | Testing           | Generate test data for development                                                                | [📖 Docs](/connect/connectors/kafka-connect-datagen/current/kafka-connect-datagen-source)                              |
| **Kafka Connect Debezium JDBC**        | Database          | Write events to a relational database by using a JDBC driver.                                     | [📖 Docs](/connect/connectors/kafka-connect-debezium-jdbc-sink/current/kafka-connect-debezium-jdbc-sink)               |
| **Kafka Connect Elasticsearch**        | Search Engine     | Elasticsearch via Kafka Connect framework                                                         | [📖 Docs](/connect/connectors/kafka-connect-elasticsearch-sink/current/kafka-connect-elasticsearch-sink)               |
| **Kafka Connect Iceberg**              | Data Lake         | Apache Iceberg table format integration                                                           | [📖 Docs](/connect/connectors/kafka-connect-iceberg/current/kafka-connect-iceberg-sink)                                |
| **Kafka Connect JDBC**                 | Database          | Writes data to any JDBC-compliant database                                                        | [📖 Docs](/connect/connectors/kafka-connect-jdbc-sink/current/kafka-connect-jdbc-sink)                                 |
| **Kafka Connect Milvus**               | Vector Database   | Milvus vector database for AI/ML workloads                                                        | [📖 Docs](/connect/connectors/kafka-connect-milvus-sink/current/kafka-connect-milvus-sink)                             |
| **Kafka Connect MongoDB**              | Database          | MongoDB via Kafka Connect framework                                                               | [📖 Docs](/connect/connectors/kafka-connect-mongo-sink/current/kafka-connect-mongodb-sink)                             |
| **Kafka Connect Snowflake**            | Data Warehouse    | Snowflake cloud data warehouse                                                                    | [📖 Docs](/connect/connectors/kafka-connect-snowflake-sink/current/kafka-connect-snowflake-sink)                       |
| **Kafka Connect Google Pub/Sub**       | Message Queue     | Persists data from Apache Kafka topics as a data sink into Google Cloud Pub/Sub via Kafka Connect | [📖 Docs](/connect/connectors/kafka-connect-google-pubsub-sink/current/kafka-connect-google-pubsub-sink)               |
| **Kafka Connect Google Pub/Sub Lite**  | Message Queue     | Persists data from Apache Kafka topics as a data sink into Google Cloud Pub/Sub via Kafka Connect | [📖 Docs](/connect/connectors/kafka-connect-google-pubsub-lite-sink/current/kafka-connect-google-pubsub-lite-sink)     |
| **Kafka Connect Google Cloud Storage** | Object Storage    | Writes data from Kafka topics to a GCS(Google Cloud Storage) bucket                               | [📖 Docs](/connect/connectors/kafka-connect-google-cloud-storage-sink/current/kafka-connect-google-cloud-storage-sink) |
| **Kafka Connect Google Bigtable**      | Database          | Stream data from Kafka topics into Bigtable in real time                                          | [📖 Docs](/connect/connectors/kafka-connect-google-bigtable-sink/current/kafka-connect-google-bigtable-sink)           |
| **Kafka**                              | Message Queue     | Send data to Apache Kafka topics                                                                  | [📖 Docs](/connect/connectors/kafka-sink/current/kafka-sink)                                                           |
| **Kinesis**                            | Stream Processing | Amazon Kinesis data streams                                                                       | [📖 Docs](/connect/connectors/kinesis-sink/current/kinesis-sink)                                                       |
| **Lakehouse**                          | Data Lake         | Write to lakehouse storage systems                                                                | [📖 Docs](/connect/connectors/lakehouse-sink/current/lakehouse-sink)                                                   |
| **MongoDB**                            | Database          | MongoDB document database                                                                         | [📖 Docs](/connect/connectors/mongodb-sink/current/mongodb-sink)                                                       |
| **Pinecone**                           | Vector Database   | Pinecone vector database for ML applications                                                      | [📖 Docs](/connect/connectors/pinecone-sink/current/pinecone-sink)                                                     |
| **RabbitMQ**                           | Message Queue     | Send to RabbitMQ message broker                                                                   | [📖 Docs](/connect/connectors/rabbitmq-sink/current/rabbitmq-sink)                                                     |
| **Redis**                              | Cache             | Redis in-memory data structure store                                                              | [📖 Docs](/connect/connectors/redis-sink/current/redis-sink)                                                           |
| **Snowflake**                          | Data Warehouse    | Snowflake cloud data platform                                                                     | [📖 Docs](/connect/connectors/snowflake-sink/current/snowflake-sink)                                                   |
| **Snowflake Streaming**                | Data Warehouse    | Snowflake real-time data ingestion                                                                | [📖 Docs](/connect/connectors/snowflake-streaming-sink/current/snowflake-streaming)                                    |
| **Solr**                               | Search Engine     | Apache Solr search platform                                                                       | [📖 Docs](/connect/connectors/solr-sink/current/solr-sink)                                                             |
| **Amazon SQS**                         | Message Queue     | Amazon Simple Queue Service                                                                       | [📖 Docs](/connect/connectors/sqs-sink/current/sqs-sink)                                                               |

## What's Next?

<CardGroup>
  <Card title="Deploy Your First Connector" icon="rocket" href="/cloud/connect/pulsar-io/deploy-connectors/deploy-connector-index">
    Follow our step-by-step guide to deploy your first connector.
  </Card>

  <Card title="Configuration Reference" icon="settings" href="/cloud/connect/pulsar-io/connector-config">
    Learn about connector configuration options and best practices.
  </Card>

  <Card title="Monitor & Troubleshoot" icon="chart-line" href="/cloud/connect/pulsar-io/connector-monitoring">
    Set up monitoring and troubleshooting for your connectors.
  </Card>
</CardGroup>


# snctl Command References
Source: https://docs.streamnative.io/tools/cli/snctl/snctl-command-references



## Available releases

The following table lists the available releases of `snctl` and their reference documentation.

| Version | Reference                                                                            |
| ------- | ------------------------------------------------------------------------------------ |
| latest  | [Command reference](https://doc-references.streamnative.io/snctl/latest/index.html)  |
| v1.1.0  | [Command reference](https://doc-references.streamnative.io/snctl/v1.1.0/index.html)  |
| v1.0.0  | [Command reference](https://doc-references.streamnative.io/snctl/v1.0.0/index.html)  |
| v0.22.1 | [Command reference](https://doc-references.streamnative.io/snctl/v0.22.1/index.html) |
| v0.22.0 | [Command reference](https://doc-references.streamnative.io/snctl/v0.22.0/index.html) |
| v0.21.0 | [Command reference](https://doc-references.streamnative.io/snctl/v0.21.0/index.html) |
| v0.20.0 | [Command reference](https://doc-references.streamnative.io/snctl/v0.20.0/index.html) |
| v0.19.0 | [Command reference](https://doc-references.streamnative.io/snctl/v0.19.0/index.html) |
| v0.18.0 | [Command reference](https://doc-references.streamnative.io/snctl/v0.18.0/index.html) |
| v0.17.0 | [Command reference](https://doc-references.streamnative.io/snctl/v0.17.0/index.html) |
| v0.16.0 | [Command reference](https://doc-references.streamnative.io/snctl/v0.16.0/index.html) |
| v0.15.0 | [Command reference](https://doc-references.streamnative.io/snctl/v0.15.0/index.html) |
| v0.14.2 | [Command reference](https://doc-references.streamnative.io/snctl/v0.14.2/index.html) |
| v0.14.1 | [Command reference](https://doc-references.streamnative.io/snctl/v0.14.1/index.html) |
| v0.14.0 | [Command reference](https://doc-references.streamnative.io/snctl/v0.14.0/index.html) |
| v0.13.3 | [Command reference](https://doc-references.streamnative.io/snctl/v0.13.3/index.html) |
| v0.13.2 | [Command reference](https://doc-references.streamnative.io/snctl/v0.13.2/index.html) |
| v0.13.1 | [Command reference](https://doc-references.streamnative.io/snctl/v0.13.1/index.html) |
| v0.13.0 | [Command reference](https://doc-references.streamnative.io/snctl/v0.13.0/index.html) |
| v0.12.1 | [Command reference](https://doc-references.streamnative.io/snctl/v0.12.1/index.html) |
| v0.12.0 | [Command reference](https://doc-references.streamnative.io/snctl/v0.12.0/index.html) |
| v0.11.1 | [Command reference](https://doc-references.streamnative.io/snctl/v0.11.1/index.html) |
| v0.10.1 | [Command reference](https://doc-references.streamnative.io/snctl/v0.10.1/index.html) |
| v0.9.0  | [Command reference](https://doc-references.streamnative.io/snctl/v0.9.0/index.html)  |
| v0.8.0  | [Command reference](https://doc-references.streamnative.io/snctl/v0.8.0/index.html)  |
| v0.7.0  | [Command reference](https://doc-references.streamnative.io/snctl/v0.7.0/index.html)  |
| v0.6.0  | [Command reference](https://doc-references.streamnative.io/snctl/v0.6.0/index.html)  |
| v0.5.0  | [Command reference](https://doc-references.streamnative.io/snctl/v0.5.0/index.html)  |


# StreamNative CLI (snctl)
Source: https://docs.streamnative.io/tools/cli/snctl/snctl-overview



StreamNative Cloud provides a command line tool for communicating with StreamNative Cloud's control plane, using the StreamNative Cloud API. This tool is named `snctl`. This overview includes how to install and configure `snctl`, covers `snctl` syntax, describes the command operations, and provides common examples. For details about each command, including all the supported flags and subcommands, see the [snctl](https://doc-references.streamnative.io/snctl/latest/index.html) reference documentation.

<Note title="Note">
  To resolve ambiguity with Apache Pulsar's concept of namespaces, particularly when using `snctl pulsar` subcommands, the flags `--namespace` (`-n`) were updated to `--organization` (`-O`) beginning in snctl v1.0.0.
</Note>

## Prerequisites

Before moving on to the subsequent steps, ensure you review the following requirements.

### Operating systems

The StreamNative CLI is compatible with the following operating systems and architectures only:

* macOS with 64-bit Intel chips (Darwin AMD64)
* macOS with Apple chips (Darwin ARM64)
* Windows with 64-bit Intel or AMD chips (Microsoft Windows AMD64)
* Linux with 64-bit Intel or AMD chips (Linux AMD64)
* Linux with 64-bit ARM chips (Linux ARM64)

### Network access

When the StreamNative CLI interacts with StreamNative Cloud, it requires network access to the following domains:

* `api.streamnative.cloud`
* `auth.streamnative.cloud`
* `log.streamnative.cloud`

## Install snctl

This section describes how to install snctl on Linux, MAC, and Windows Operating System (OS).

<Tabs>
  <Tab title="Linux">
    You can use the `curl` command or use Homebrew to install snctl on Linux.

    #### Install snctl with curl command

    <span />

    1. Execute the following command to download and install the latest snctl.

       ```bash theme={null}
       bash -c "$(curl -fsSL https://storage.googleapis.com/downloads.streamnative.cloud/snctl/install.sh)"
       ```

       If you want to download a specific version, use the flag `-v` or `--version` to specify the version. For example, you can use the following command to download snctl `v1.0.0`.

       ```bash theme={null}
       bash -c "$(curl -fsSL https://storage.googleapis.com/downloads.streamnative.cloud/snctl/install.sh) -v v1.0.0"
       ```

    2. Check whether snctl is installed successfully.

       ```bash theme={null}
       snctl version --client
       ```
  </Tab>

  <Tab title="Mac">
    You can use the curl command or use Homebrew to install snctl on a Mac.

    #### Install snctl with curl command

    <span />

    1. Execute the following command to download and install the latest snctl.

       ```bash theme={null}
       bash -c "$(curl -fsSL https://storage.googleapis.com/downloads.streamnative.cloud/snctl/install.sh)"
       ```

       If you want to download a specific version, use the flag `-v` or `--version` to specify the version. For example, you can use the following command to download snctl `v1.0.0`.

       ```bash theme={null}
       bash -c "$(curl -fsSL https://storage.googleapis.com/downloads.streamnative.cloud/snctl/install.sh) -v v1.0.0"
       ```

    2. Check whether snctl is installed successfully.

       ```bash theme={null}
       snctl version --client
       ```

    #### Install snctl with Homebrew

    <span />

    1. Add the repository.

       ```bash theme={null}
       brew tap streamnative/streamnative
       ```

    2. Install snctl.

       ```bash theme={null}
       brew install snctl
       ```
  </Tab>

  <Tab title="Windows">
    1. Download the latest release package.

       ```bash theme={null}
       curl https://storage.googleapis.com/downloads.streamnative.cloud/snctl/v1.0.0/snctl_1.0.0_windows_amd64.zip -o-output <your_file_name>
       ```

    2. Extract the snctl `.zip` package using Windows Explorer. For details, see the [instructions](https://support.microsoft.com/en-us/windows/zip-and-unzip-files-f6dde0a7-0fec-8294-e1d3-703ed85e7ebc).

    3. (Optional) Copy the `snctl.exe` binary to a directory on your `PATH`.

    4. Check whether snctl is installed successfully.

       ```bash theme={null}
       snctl version --client
       ```
  </Tab>
</Tabs>

<Note title="Note">
  When upgrading from `snctl` `v0.x.x` to `v1.x.x`, please run `snctl config init` again to ensure all newly introduced configuration settings are applied to your local configuration file.
</Note>

## Configure snctl

This section describes how to configure `snctl`.

### Initialize snctl configuration

Before logging in to `snctl`, you need to configure `snctl`. You can either use the `snctl config init` command to initialize snctl with default configurations or you can use the `snctl config set` command to update the `.snctl/config` snctl configuration file.

### Set snctl configuration

The `.snctl/config` file contains configurations about snctl, including OAuth2 configurations, service URL for the StreamNative Cloud API, and so on. You can use the `snctl config set` command to set snctl configurations.

This example sets a target organization as the default organization. Consequently, in this organization, you can perform other operations on StreamNative Cloud resources without specifying the organization name every time.

<Note title="Note">
  With the `organization` option, you need to use the organization id, not the descriptive organization name. To find the organization id, see [Cloud Organization ID](/cloud/security/access/resource-hierarchy/organizations#cloud-organization-id).
</Note>

```bash theme={null}
snctl config set --organization <organization_id>
```

## Sign in to an organization

You need to sign in to an organization before executing any `snctl` commands. You can sign in either as a specific user or as a service account.

<Tabs>
  <Tab title="User account">
    ### Sign in as a user account

    This example shows how to sign in as a user account. In this way, `snctl` is given a token to impersonate the user.

    ```bash theme={null}
    snctl auth login
    ```

    **Output**

    ```shell theme={null}
    Logged in as example@streamnative.io.
    Welcome to StreamNative Cloud!
    ```
  </Tab>

  <Tab title="Service account">
    ### Log in to snctl as a service account

    This example shows how to sign in as a service account.

    1. Download the credentials file of the service account.

       ```bash theme={null}
       snctl auth export-service-account <service_account_name> --key-file /path/to/service_account_credentials.json
       ```

    2. Sign in as a service account.

       ```bash theme={null}
       snctl auth activate-service-account <service_account_name> --key-file /path/to/service_account_credentials.json
       ```

       **Output**

       ```shell theme={null}
       Logged in as bot@test.auth.streamnative.cloud.
       Welcome to StreamNative Cloud!
       ```

    Notes:

    * Replace `<service_account_name>` with the service account name you use.

    * Replace `/path/to/service_account_credentials.json` with the right file path to save the credentials of the service account. Please make sure the credentials are saved in a safe location.
  </Tab>
</Tabs>

## Subscribe to StreamNative Cloud

<Tip title="Tips">
  If your organization already has a valid subscription, you can skip this section.
</Tip>

Before you provision a cluster using `snctl`, you need to set up a subscription to StreamNative Cloud. If you have a legacy cluster, [submit a ticket](https://support.streamnative.io/support/login) to get assistance with moving your cluster to the updated subscription plan.

<Note title="Note">
  * The `snctl create subscription` command is available for `snctl` version `0.14.1` and above.
  * Ensure to add a payment method if you have not done so already. `snctl` does not support setting up a payment method. You can add a payment method for your organization through StreamNative Cloud Console. For details, see [manage billing using StreamNative Cloud Console](/cloud/billing/billing).
</Note>

You can use the `snctl create subscription` command to create a subscription.

```bash theme={null}
snctl create subscription <subscription_name> -O <organization_name> --offer-type <type_of_offer> --offer <offer_name>
```

Notes:

* `--offer-type`: the offer type. Two options are available:
  * `private`: the private offer that is made to a specific customer. It is a commitment to a minimum amount of spend over a specified time period.
  * `public`: the public offer that anyone may subscribe to through StreamNative or Marketplace channels. It is a Pay-As-You-Go subscription with usage-based pricing.
* `--offer`: the offer name.
  * For a public offer, there is a well-known name (`SN2_OD_HOSTED_CLOUD`) that `snctl` uses by default.
  * For a private offer, you will get the offer name from StreamNative sales.

**Output**

* You should see the following output if you create a public offer subscription:

  ```bash theme={null}
  Preparing the subscription, please wait...
  Subscription has been activated.
  ```

* You should see the following output if you create a private offer subscription:

  ```bash theme={null}
  Preparing the subscription, please wait...
  To activate your subscription, please pay the initial invoice.
  Open the following URL:
  https://stripe.com/...
  Subscription has been activated.
  ```

## Use snctl to provision a Pulsar cluster

This section describes how to provision Pulsar clusters through snctl.

<Note title="Note">
  In this section, the target organization name is already set in the `.snctl/config` file. Therefore, you do not need to use the `--organization <organization_name>` flag to specify the target organization every time when executing `snctl` commands. For details about how to set snctl configurations, see [set snctl configuration](#set-snctl-configuration).
</Note>

### Create a Pulsar cluster

This section describes how to create a Dedicated cluster through snctl.

1. Create a Pulsar instance named `neo`.

   This example shows how to create the `neo` instance on the AWS cloud platform. To create a Pulsar instance on Google Cloud, set the infrastructure pool name to `streamnative/shared`. To create a BYOC instance/cluster, the StreamNative team needs to provision an infrastructure pool in your cloud account before you can use it.

   ```bash theme={null}
   snctl create pulsarinstance neo --pool streamnative/shared-aws
   ```

   **Output**

   ```shell theme={null}
   pulsarinstance.cloud.streamnative.io/neo created
   ```

2. Find the available locations of a given infrastructure pool to deploy a Pulsar cluster.

   This example shows how to find the available locations for the infrastructure pool `streamnative/shared-aws`.

   ```bash theme={null}
   snctl get pooloptions streamnative-shared-aws -o yaml
   ```

   You can find the locations shown up in the `spec.locations` section of the output.

   ```shell theme={null}
   spec:
     cloudType: aws
     deploymentType: ""
     features:
       AutoScaling: true
       Function: true
       Istio: true
       KOP: true
       MOP: true
       Transaction: true
       WebSocket: true
     locations:
     - location: ap-southeast-2
     - location: eu-central-1
     - location: eu-west-1
     - location: us-east-2
   ```

3. Create a Pulsar cluster named `neo-1`, consisting of 2 brokers, each using 0.5 CU, and 3 bookies, each using 0.5 SU, to be deployed in `us-east-1`.

<Note title="Note">
  The following command is available for `snctl` version `0.15.0` and above.
</Note>

```bash theme={null}
snctl create pulsarcluster neo-1 --instance-name neo \
  --bookie-replicas 3 \
  --storage-unit 0.5 \
  --broker-replicas 3 \
  --compute-unit 0.5 \
  --location us-east-2
```

**Output**

```shell theme={null}
pulsarcluster.cloud.streamnative.io/neo-1 created
```

### Get the details of a Pulsar cluster

You can use the `snctl describe pulsarcluster <cluster_name>` command to get details of a Pulsar cluster.

This example gets details of the `neo-1` cluster.

```bash theme={null}
snctl describe pulsarcluster neo-1
```

**Output**

```shell theme={null}
Cluster neo-1
Name:         neo-1
Namespace:    matrix
Labels:       <none>
Annotations:  <none>
API Version:  cloud.streamnative.io/v1alpha1
Kind:         PulsarCluster
Metadata:
  Creation Timestamp:  2021-01-25T14:48:27Z
  Finalizers:
    pulsarcluster.finalizers.cloud.streamnative.io
  Generation:  2
  Managed Fields:
    API Version:  cloud.streamnative.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:bookkeeper:
          .:
          f:replicas:
          f:resourceSpec:
        f:broker:
          f:resourceSpec:
            .:
            f:nodeType:
        f:instanceName:
        f:location:
    Manager:      snctl
    Operation:    Update
    Time:         2021-01-25T14:48:27Z
    API Version:  cloud.streamnative.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:broker:
          f:replicas:
    Manager:      kubectl
    Operation:    Update
    Time:         2021-01-29T09:51:58Z
    API Version:  cloud.streamnative.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"pulsarcluster.finalizers.cloud.streamnative.io":
      f:status:
        f:conditions:
          .:
          k:{"type":"BookKeeperReady"}:
            .:
            f:lastTransitionTime:
            f:status:
            f:type:
          k:{"type":"PulsarBrokerReady"}:
            .:
            f:lastTransitionTime:
            f:reason:
            f:status:
            f:type:
          k:{"type":"PulsarProxyReady"}:
            .:
            f:lastTransitionTime:
            f:reason:
            f:status:
            f:type:
          k:{"type":"Ready"}:
            .:
            f:lastTransitionTime:
            f:reason:
            f:status:
            f:type:
          k:{"type":"ZookeeperReady"}:
            .:
            f:lastTransitionTime:
            f:status:
            f:type:
    Manager:         controller-manager
    Operation:       Update
    Time:            2021-02-05T03:16:49Z
  Resource Version:  18740114
  Self Link:         /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/pulsarclusters/neo-1
  UID:               774c4774-47c7-4913-b9b7-58c47c2f5e43
Spec:
  Bookkeeper:
    Image:     docker.cloudsmith.io/streamnative/cloud-pulsar/pulsar-cloud:3.0.1.6
    Replicas:  3
    Resources:
      Cpu:                1
      Direct Percentage:  0
      Heap Percentage:    0
      Journal Disk:       8G
      Ledger Disk:        64G
      Memory:             4294967296
  Broker:
    Image:     docker.cloudsmith.io/streamnative/cloud-pulsar/pulsar-cloud:3.0.1.6
    Replicas:  3
    Resources:
      Cpu:                1
      Direct Percentage:  0
      Heap Percentage:    0
      Memory:             4294967296
  Instance Name:          neo
  Location:               us-east-1
  Pool Member Ref:
    Name:       aws-use2-production-snci-pool-kid
    Namespace:  streamnative
  Service Endpoints:
    Dns Name:  neo-1.matrix.aws-us-east-1.streamnative.aws.snio.cloud
    Type:      service
Status:
  Bookkeeper:
    Ready Replicas:    3
    Replicas:          3
    Updated Replicas:  3
  Broker:
    Ready Replicas:    2
    Replicas:          2
    Updated Replicas:  2
  Conditions:
    Last Transition Time:  2023-12-28T17:12:00Z
    Reason:                Deploy
    Status:                True
    Type:                  ZookeeperReady
    Last Transition Time:  2023-12-28T17:12:20Z
    Reason:                Deploy
    Status:                True
    Type:                  BookKeeperReady
    Last Transition Time:  2023-12-28T17:11:56Z
    Reason:                Deploy
    Status:                True
    Type:                  PulsarBrokerReady
    Last Transition Time:  2023-12-28T17:12:20Z
    Reason:                AllConditionStatusTrue
    Status:                True
    Type:                  Ready
    Last Transition Time:  2023-12-28T17:10:05Z
    Reason:                Ready
    Status:                True
    Type:                  PulsarInstanceReady
  Zookeeper:
    Ready Replicas:    3
    Replicas:          3
    Updated Replicas:  3
Events:                    <none>
```

From the outputs, you can see that the `status` and `type` parameters for items under `Conditions` are set to `true` and `ready` respectively. This means that the `neo-1` cluster is created successfully.

### Configure the Service Context to interact with Pulsar and Kafka protocol

Starting from version `v1.0.0`, `snctl` introduces the concept of a **Service Context**. This feature aims to make `snctl` a unified tool not only for managing StreamNative Cloud resources (like creating and managing clusters) but also for directly interacting with the data plane of your Pulsar clusters using both the native Pulsar protocol and the Kafka protocol (via KSN or Ursa Engine).

A Service Context is essentially a named configuration profile stored within your `snctl` configuration (`~/.snctl/config`). It bundles the necessary connection details —- primarily the service URL and authentication information —- required to connect to a specific Pulsar cluster managed by StreamNative Cloud.

**Automatic Context Discovery**

Typically, you don't need to manually create contexts for your StreamNative Cloud Pulsar clusters. After successfully logging in using `snctl auth login`, `snctl` can often automatically detect the Pulsar clusters you have access to within your organization. It makes these clusters available as selectable contexts, usually using the instance name and cluster name as the context.

**(Note:** For managing contexts related to external, non-StreamNative Cloud clusters, see commands like `snctl context add-external-context`, `snctl context list-external-context`, etc.)

**Switching and Using Contexts**

To interact with a specific cluster using `snctl pulsar` or `snctl kafka` commands, you first need to activate its corresponding context.

1. **Set the Active Context:** Use the `snctl context use` command, will likely prompt you to choose from a list of all available Pulsar Instances and Pulsar Clusters, select the target cluster by keyboard, and additional authentication may required if no valid token cached.

   ```bash theme={null}
   snctl context use
   ```

   **Non-interactive**

   ```shell theme={null}
   snctl context use --pulsar-instance $instance --pulsar-cluster $cluster
   ```

2. **Verify the Current Context:** You can check which context is currently active using:

   ```bash theme={null}
   snctl context current
   ```

   **Output (Example)**

   ```shell theme={null}
   Using cloud service context
   Pulsar instance: xxx
   Pulsar cluster: xxx
   Configured in organization: xxx
   Current organization: xxx
   ```

**Interacting with Pulsar and Kafka Protocols**

Once a service context is active, all subsequent `snctl pulsar client ...`, `snctl pulsar admin ...`, `snctl kafka client ...`, and `snctl kafka admin ...` commands will automatically use the connection details (service URL, authentication) defined in that active context. You no longer need to specify connection flags repeatedly for these commands.

**Running commands by using specific Service Account**

While the active context usually derives its authentication from your user login (`snctl auth login`), you can override this for individual command executions to run as a specific service account. This is useful for automation or when specific permissions granted to a service account are needed. Another use case is to submit Pulsar Functions, Pulsar IO Connectors, and Kafka Connect connectors in StreamNative Cloud, this will makes the submitted instances running under given permission by the service account.

Crucially, `snctl` will verify if your currently logged-in user account has the necessary permissions to impersonate the chosen service account before executing the command.

* **Specify Service Account by Name:** Use the `--as-service-account` flag followed by the service account name. `snctl` will use the credentials associated with this service account for the command.

  ```bash theme={null}
  # Run a pulsar command as 'my-automation-sa' service account
  snctl pulsar admin tenants list --as-service-account my-automation-sa

  # Run a kafka command as 'my-kafka-processor-sa' service account
  snctl kafka admin topics list --as-service-account my-kafka-processor-sa
  ```

* **Interactively Select Service Account:** Use the `--use-service-account` flag without specifying a name. `snctl` will likely prompt you to choose from a list of available service accounts you have access to. This will shows you the details of each Service Account, as well as the status of each Service Account Binding, which is useful on manage Pulsar Functions, Pulsar IO Connectors, and Kafka Connect connectors.

  ```bash theme={null}
  # Interactively select a service account for the command
  snctl pulsar admin namespaces list --use-service-account
  ```

**Example Workflow:**

```bash theme={null}
# Log in to snctl (if not already)
snctl auth login

# Switch to the desired cluster context
snctl context use my-production-cluster

# Verify the current context (optional)
# snctl context current-context

# Now interact with the cluster using Pulsar client commands
snctl pulsar client produce my-tenant/my-namespace/my-topic --message "Hello from snctl!"

# Or use Pulsar admin commands
snctl pulsar admin tenants list

# Or interact using Kafka client commands (if KSN is enabled, or if Ursa engine is enabled)
snctl kafka client consume my-kafka-topic --group my-group --from-beginning

# Or use Kafka admin commands
snctl kafka admin topics list
```

**Note on `pulsarctl` Configuration:**

This Service Context mechanism is internal to `snctl` for unifying its own commands. It is distinct from the `snctl x update-pulsar-config` command (described later). The `update-pulsar-config` command is specifically designed to configure the separate `pulsarctl` tool, allowing it to connect to your StreamNative Cloud cluster. The `snctl context` commands enable `snctl` itself to perform data plane operations directly via its `pulsar` and `kafka` subcommands.

### Add clusters to the Pulsar configuration file

<span />

This section describes how to add a cluster to the Pulsar client configuration file. After adding a cluster to the Pulsar client configuration file, you can manage the cluster using the `pulsarctl` CLI tool. To get details about the access and managing Apache Pulsar resources through `pulsarctl`, see [`pulsarctl` command reference](https://doc-references.streamnative.io/pulsarctl/latest/index.html).

<Note title="Note">
  It is recommended to use the [`pulsarctl context get`](https://doc-references.streamnative.io/pulsarctl/latest/index.html#-em-set-em--54) command to set the context (cluster) in advance.
</Note>

1. Add a cluster to the Pulsar client configuration file.

   This example adds a `neo-1` cluster to the Pulsar client configuration file.

   **Input**

   ```
   snctl x update-pulsar-config --cluster-name <neo-1>
   ```

   **Output**

   ```
   Updated Pulsar client configuration for context 'neo-1'.
   ```

2. Verify that the current context has been changed.

   **Input**

   ```
   pulsarctl context current
   ```

   **Output**

   ```
   neo-1
   ```

After verifying that the change was made, you can use the `pulsarctl` CLI tool to interact with the target cluster. For details, see the [pulsarctl reference docs](https://doc-references.streamnative.io/pulsarctl/latest/index.html). To work with [tenants](/cloud/manage-data-streams/tenant), [namespaces](/cloud/manage-data-streams/namespace), and [topics](/cloud/manage-data-streams/topic), you can use the StreamNative Cloud Console. snctl does not currently support working with these features.


# snctl Quick Reference
Source: https://docs.streamnative.io/tools/cli/snctl/snctl-quick-reference



This page contains a list of commonly used `snctl` commands and flags.

## `snctl` autocomplete

### BASH

```bash theme={null}
source <(snctl completion bash) # set up autocomplete in bash into the current shell, bash-completion package should be installed first.
echo "source <(snctl completion bash)" >> ~/.bashrc # add autocomplete permanently to your bash shell.
```

You can also use a shorthand alias for snctl that also works with completion:

```bash theme={null}
alias sn=snctl
complete -o default -F \_\_start_snctl sn
```

### ZSH

```zsh theme={null}
source <(snctl completion zsh)  # set up autocomplete in zsh into the current shell
echo '[[ $commands[snctl] ]] && source <(snctl completion zsh)' >> ~/.zshrc # add autocomplete permanently to your zsh shell
```

## `snctl` apply

`apply` manages resources through files defining StreamNative Cloud resources. It creates and updates StreamNative Cloud resources through running `snctl apply`. This is recommended way of managing StreamNative Cloud resources in production.

## Creating and updating resources

The manifests of StreamNative Cloud resources can be defined in YAML or JSON. The file extension `.yaml`, `.yml`, and `.json` can be used.

```bash theme={null}
snctl apply -f ./my-manifest.yaml         # create resource(s)
snctl apply -f ./my1.yaml -f ./my2.yaml   # create from multiple files
snctl apply -f ./dir                      # create resource(s) in all manifest files in dir

# Create multiple resource(s) from stdin
snctl apply -f - <<EOF
apiVersion: cloud.streamnative.io/v1alpha1
kind: User
metadata:
  name: john.doe@test.local
  namespace: myorg
spec:
  email: john.doe@test.local
  name:
    first: John
    last: Doe
  type: external

---

apiVersion: cloud.streamnative.io/v1alpha1
kind: RoleBinding
metadata:
  name: john.doe@test.local
  namespace: myorg
spec:
  roleRef:
    apiGroup: cloud.streamnative.io
    kind: ClusterRole
    name: admin
  subjects:
    - apiGroup: cloud.streamnative.io
      kind: User
      name: john.doe@test.local
EOF
```

## Viewing and finding resources

```bash theme={null}
snctl get pulsarclusters                      # List all pulsar clusters in the organization
snctl get pulsarclusters mycluster            # List a particular pulsar cluster
snctl get puslarclusters mycluster -o yaml    # Get the manifest of a pulsar cluster

# Describe commands with verbose output
snctl describe pulsarcluster mycluster

# List Pulsar Clusters Sorted by Creation Time
snctl get pulsarclusters --sort-by=.metadata.creationTimestamp

# Show labels for all clusters (or any other StreamNative Cloud object that supports labelling)
snctl get pulsarclusters --show-labels

# Get the Pulsar Clusters with label app=myapp
snctl get pulsarclusters --selector=app=myapp

# Check which Pulsar Clusters are ready and their locations with custom-columns
snctl get pulsarclusters -o custom-columns='CLUSTER NAME:.metadata.name,LOCATION:.spec.location,STATUS:.status.conditions[?(@.type=="Ready")].status'
```

## Deleting resources

```bash theme={null}
snctl delete -f ./sa.yaml                     # Delete a service account using the type and name specified in sa.yaml
snctl delete pulsarcluster unwanted --now     # Delete a pulsar cluster with no grace period
snctl delete user,rolebinding baz foo         # Delete user and rolebinding with same names "baz" and "foo"
snctl delete user,rolebinding -l name=myLabel # Delete usersand rolebindings with label name=myLabel
```

## Resource types

List all supported resource types along with their shortnames, API group, whether they are namespaced (organizational scoped), and kind.

```bash theme={null}
snctl api-resources
```

## Service Context Management

Manage and use service contexts to interact with Pulsar and Kafka clusters via `snctl`. The active context determines the target cluster for `pulsar` and `kafka` commands.

```bash theme={null}
# Context Switching & Viewing
snctl context use <cluster-name>          # Set the active Service Context (typically the Pulsar cluster name)
snctl context current                     # Show the currently active Service Context

# Interacting with the Active Context's Cluster
snctl pulsar client ...   # Act as pulsar client on the active cluster
snctl pulsar admin ...            # Run pulsar admin commands on the active cluster
snctl kafka client ...    # Act as kafka client on the active cluster
snctl kafka admin ...             # Run kafka admin commands on the active cluster

# Running Commands as a Service Account (Overrides context's user auth)
snctl pulsar admin namespaces list --as-service-account <sa-name> # Run command as specific SA
snctl kafka admin groups list --use-service-account              # Interactively select an SA to run as
```


# Tutorials: Manage StreamNative Cloud resources using snctl
Source: https://docs.streamnative.io/tools/cli/snctl/snctl-tutorials



## Work with organizations

An **organization** is a team account configured perfectly for your use case on the cloud provider of your choice.

Currently, snctl does not support creating organizations. You can create an organization through [StreamNative Cloud Console](https://console.streamnative.cloud). For details about how to create an organization, see [create an organization](/cloud/security/access/resource-hierarchy/organizations#create-an-organization).

After you have created an organization through StreamNative Cloud Console, you can use the `snctl config set --organization` command to set your default organization.

<Note title="Note">
  With the `organization` option, you need to use the random string for the organization name, not the descriptive name. You can find the random string on the Dashboard, next to the organization's descriptive name. For an example, see [organizations](/cloud/references/glossary#organization).
</Note>

After setting the default organization, you can use snctl commands to perform other operations on StreamNative Cloud resources in the default organization without specifying the organization name.

## Work with instances

<Note title="Note">
  In this section, we named the organization `matrix` for an example.
</Note>

### Create an instance

A **Pulsar instance** is a group of clusters that acts together as a single unit.

To create an instance through snctl, follow these steps.

1. Define an instance named `neo` by using a manifest file and save the manifest file `instance-neo.yaml`.

   * This example shows how to create an instance on Google Cloud.

     ```yaml theme={null}
     apiVersion: cloud.streamnative.io/v1alpha1
     kind: PulsarInstance
     metadata:
       namespace: matrix
       name: neo
     spec:
       availabilityMode: zonal
       poolRef:
         namespace: streamnative
         name: shared
     ```

   * This example shows how to create an instance on AWS.

     ```yaml theme={null}
     apiVersion: cloud.streamnative.io/v1alpha1
     kind: PulsarInstance
     metadata:
       namespace: matrix
       name: neo
     spec:
       availabilityMode: zonal
       poolRef:
         namespace: streamnative
         name: shared-aws
     ```

   The following table lists fields in the manifest file.

   | Field      | Description                               |
   | ---------- | ----------------------------------------- |
   | apiVersion | Specify the version of Pulsar API server. |
   | kind       | Specify the component to be created.      |

<br />

<br />

2. Apply the manifest file to create the instance.

   ```bash theme={null}
   snctl apply -f /path/to/instance-neo.yaml
   ```

3. Check whether the instance is created successfully. This example shows whether the instance is successfully created on AWS.

   ```
   snctl describe pulsarinstance neo
   ```

   **Output**

   ```
   Name:         neo
   Namespace:    matrix
   Labels:       <none>
   Annotations:  <none>
   API Version:  cloud.streamnative.io/v1alpha1
   Kind:         PulsarInstance
   Metadata:
     Creation Timestamp:  2020-08-11T07:37:49Z
     Finalizers:
       pulsarinstance.finalizers.cloud.streamnative.io
     Generation:        1
     Resource Version:  367947
     Self Link:         /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/pulsarinstances/neo
     UID:               53f2958d-c7c9-4628-9290-a34e36cdca3b
   Spec:
     Availability Mode:  zonal
     poolRef:
       namespace: streamnative
       name: shared-aws
   Status:
     Auth:
       oauth2:
         Audience:    urn:sn:pulsar:test:test
         Issuer URL:  https://auth.streamnative.cloud
       Type:          oauth2
     Conditions:
       Last Transition Time:  2020-08-11T07:38:00Z
       Status:                True
       Type:                  SubscriptionReady
       Last Transition Time:  2020-08-11T07:37:51Z
       Reason:                Created
       Status:                True
       Type:                  ResourceServerReady
       Last Transition Time:  2020-08-11T07:37:51Z
       Reason:                Created
       Status:                True
       Type:                  ServiceAccountReady
       Last Transition Time:  2020-08-11T07:38:01Z
       Reason:                AllConditionStatusTrue
       Status:                True
       Type:                  Ready
   Events:                    <none>
   ```

   From the outputs, you can see that the `status` and `type` parameters for items under `Conditions` are set to `true` and `ready`. This means that the instance `neo` is created successfully.

In addition, you can use the `snctl create pulsarinstance PULSAR_INSTANCE_NAME` command to create an instance. For details, see [snctl reference](https://doc-references.streamnative.io/snctl/latest/index.html#create).

### Check the instance

You can use the following command to list all created instances.

```bash theme={null}
snctl get pulsarinstance
```

**Output**

```shell theme={null}
NAME            CREATED AT
instance-test   2020-08-13T14:14:43Z
neo            2020-08-11T07:37:49Z
test-orga-a     2020-08-12T05:32:33Z
```

From the output of this command, you can see all created instances and the time when these instances were created.

### Check instance details

Before checking details about an instance, you should use the following command to confirm whether the target instance is available.

```bash theme={null}
snctl get pulsarinstance
```

Then, you can use the following command to check details about a specific instance.

```bash theme={null}
snctl describe pulsarinstance PULSAR_INSTANCE_NAME
```

The following example checks details about the instance `neo`.

```bash theme={null}
snctl describe pulsarinstance neo
```

**Output**

```
Name:         neo
Namespace:    matrix
Labels:       <none>
Annotations:  <none>
API Version:  cloud.streamnative.io/v1alpha1
Kind:         PulsarInstance
Metadata:
  Creation Timestamp:  2020-08-11T07:37:49Z
  Finalizers:
    pulsarinstance.finalizers.cloud.streamnative.io
  Generation:        1
  Resource Version:  367947
  Self Link:         /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/pulsarinstances/neo
  UID:               53f2958d-c7c9-4628-9290-a34e36cdca3b
Spec:
  Availability Mode:  zonal
Status:
  Auth:
    oauth2:
      Audience:    urn:sn:pulsar:test:test
      Issuer URL:  https://auth.streamnative.cloud
    Type:          oauth2
  Conditions:
    Last Transition Time:  2020-08-11T07:38:00Z
    Status:                True
    Type:                  SubscriptionReady
    Last Transition Time:  2020-08-11T07:37:51Z
    Reason:                Created
    Status:                True
    Type:                  ResourceServerReady
    Last Transition Time:  2020-08-11T07:37:51Z
    Reason:                Created
    Status:                True
    Type:                  ServiceAccountReady
    Last Transition Time:  2020-08-11T07:38:01Z
    Reason:                AllConditionStatusTrue
    Status:                True
    Type:                  Ready
Events:                    <none>
```

### Delete an instance

You can use the following command to delete a specific instance based on the instance name.

```bash theme={null}
snctl delete pulsarinstance PULSAR_INSTANCE_NAME
```

In addition, you can use the following command to delete a instance based on the type and name specified in the manifest file.

```
snctl delete -f ./instance-neo.yaml
```

## Work with clusters

A cluster is a secure messaging environment within Pulsar. Each Pulsar cluster consists a set of 3 components in a geographical location.

* **Pulsar brokers** - set of brokers handling all the data going in and out of Pulsar (or client requests)
* **Metadata storage** - providing coordination and service discovery between services
* **Bookie ensemble** - set of bookies that retain copies of the messages

A cluster has two layers: a stateless serving layer (made up of brokers) and a stateful storage layer (made up of bookies). See also [Pulsar Architecture and Design](https://pulsar.apache.org/docs/4.0.x/concepts-architecture-overview/).

<Note title="Note">
  In StreamNative Console, you can create one and only one cluster for an instance.
</Note>

The code examples in this section use an organization called, `matrix`.

### Create a cluster

<Note title="Note">
  Before you can create a cluster through `snctl`, you need to create a billing subscription. For more information, see [subscribe to StreamNative Cloud](/tools/cli/snctl/snctl-overview#subscribe-to-stream-native-cloud).
</Note>

To create a cluster, follow these steps.

1. Define a cluster named `neo-1` by using a manifest file and save the manifest file `clusterneo1.yaml`.

   ```yaml theme={null}
   apiVersion: cloud.streamnative.io/v1alpha1
   kind: PulsarCluster
   metadata:
     namespace: matrix
     name: neo-1
   spec:
     instanceName: neo
     location: us-east4
     bookkeeper:
       replicas: 3
       resourceSpec:
         nodeType: tiny-1
     broker:
       replicas: 1
       resourceSpec:
         nodeType: tiny-1
     config:
       custom:
         backlogQuotaDefaultLimitBytes: '1000000000'
       websocketEnabled: true
   ```

   The following table lists fields in the manifest file.

   | Field      | Description                               |
   | ---------- | ----------------------------------------- |
   | apiVersion | Specify the version of Pulsar API server. |
   | kind       | Specify the component to be created.      |

<br />

<br />

2. Apply the manifest file to create the cluster.

   ```bash theme={null}
   snctl apply -f /path/to/clusterneo1.yaml
   ```

3. Check whether the cluster is created successfully.

   ```bash theme={null}
   snctl describe pulsarcluster neo-1
   ```

   **Output**

   ```
   Name:         neo-1
   Namespace:    matrix
   Labels:       <none>
   Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                   {"apiVersion":"cloud.streamnative.io/v1alpha1","kind":"PulsarCluster","metadata":{"annotations":{},"name":"neo-1","namespace":"matrix"},...
   API Version:  cloud.streamnative.io/v1alpha1
   Kind:         PulsarCluster
   Metadata:
     Creation Timestamp:  2021-10-12T01:49:30Z
     Finalizers:
       pulsarcluster.finalizers.cloud.streamnative.io
     Generation:  1
     Managed Fields:
       API Version:  cloud.streamnative.io/v1alpha1
       Fields Type:  FieldsV1
       fieldsV1:
         f:metadata:
           f:annotations:
             .:
             f:kubectl.kubernetes.io/last-applied-configuration:
         f:spec:
           f:bookkeeper:
             .:
             f:replicas:
             f:resourceSpec:
               .:
               f:nodeType:
           f:broker:
             f:replicas:
             f:resourceSpec:
               .:
               f:nodeType:
           f:config:
             .:
             f:custom:
               .:
               f:backlogQuotaDefaultLimitBytes:
             f:websocketEnabled:
           f:instanceName:
           f:location:
       Manager:      Go-http-client
       Operation:    Update
       Time:         2021-10-12T01:49:29Z
       API Version:  cloud.streamnative.io/v1alpha1
       Fields Type:  FieldsV1
       fieldsV1:
         f:metadata:
           f:finalizers:
             .:
             v:"pulsarcluster.finalizers.cloud.streamnative.io":
       Manager:         controller-manager
       Operation:       Update
       Time:            2021-10-12T01:49:30Z
     Resource Version:  11761383
     Self Link:         /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/pulsarclusters/neo-1
     UID:               b1409658-e9f5-4056-9dae-0d484b230607
   Spec:
     Bookkeeper:
       Image:     gcr.io/affable-ray-226821/pulsar-cloud:2.8.0.9
       Replicas:  3
       Resource Spec:
         Node Type:  tiny-1
     Broker:
       Image:     gcr.io/affable-ray-226821/pulsar-cloud:2.8.0.9
       Replicas:  1
       Resource Spec:
         Node Type:  tiny-1
     Config:
       Custom:
         Backlog Quota Default Limit Bytes:  1000000000
       Websocket Enabled:                    true
     Instance Name:                          ins-2
     Location:                               us-east4
     Pool Member Ref:
       Name:       us-east4
       Namespace:  streamnative
     Service Endpoints:
       Dns Name:  neo-1.matrix.sn2.dev
       Type:      service
       Dns Name:  neo-1.matrix.us-east4.streamnative.g.sn2.dev
   Status:
   Events:  <none>
   ```

### Check the cluster

You can use the following command to list all created clusters.

```bash theme={null}
snctl get pulsarcluster
```

**Output**

```shell theme={null}
NAME            CREATED AT
cluster-test   2020-08-14T01:48:46Z
neo-1          2021-10-12T01:49:29Z
test11         2020-08-12T13:24:19Z
```

From the output of this command, you can see all created clusters and the time when these clusters were created.

### Check cluster details

Before checking the details about a cluster, you should use the following command to confirm whether the target cluster is available.

```bash theme={null}
snctl get pulsarcluster
```

Then, you can use the following command to check details about a specific cluster.

```bash theme={null}
snctl describe pulsarcluster <cluster_name>
```

The following example checks details about the instance `neo-1`.

```bash theme={null}
snctl describe pulsarcluster neo-1
```

**Output**

```
Name:         neo-1
Namespace:    matrix
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"cloud.streamnative.io/v1alpha1","kind":"PulsarCluster","metadata":{"annotations":{},"name":"neo-1","namespace":"matrix"},...
API Version:  cloud.streamnative.io/v1alpha1
Kind:         PulsarCluster
Metadata:
  Creation Timestamp:  2021-10-12T01:49:30Z
  Finalizers:
    pulsarcluster.finalizers.cloud.streamnative.io
  Generation:  1
  Managed Fields:
    API Version:  cloud.streamnative.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        f:bookkeeper:
          .:
          f:replicas:
          f:resourceSpec:
            .:
            f:nodeType:
        f:broker:
          f:replicas:
          f:resourceSpec:
            .:
            f:nodeType:
        f:config:
          .:
          f:custom:
            .:
            f:backlogQuotaDefaultLimitBytes:
          f:websocketEnabled:
        f:instanceName:
        f:location:
    Manager:      Go-http-client
    Operation:    Update
    Time:         2021-10-12T01:49:29Z
    API Version:  cloud.streamnative.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"pulsarcluster.finalizers.cloud.streamnative.io":
    Manager:         controller-manager
    Operation:       Update
    Time:            2021-10-12T01:49:30Z
  Resource Version:  11761383
  Self Link:         /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/pulsarclusters/neo-1
  UID:               b1409658-e9f5-4056-9dae-0d484b230607
Spec:
  Bookkeeper:
    Image:     gcr.io/affable-ray-226821/pulsar-cloud:2.8.0.9
    Replicas:  3
    Resource Spec:
      Node Type:  tiny-1
  Broker:
    Image:     gcr.io/affable-ray-226821/pulsar-cloud:2.8.0.9
    Replicas:  1
    Resource Spec:
      Node Type:  tiny-1
  Config:
    Custom:
      Backlog Quota Default Limit Bytes:  1000000000
    Websocket Enabled:                    true
  Instance Name:                          ins-2
  Location:                               us-east4
  Pool Member Ref:
    Name:       us-east4
    Namespace:  streamnative
  Service Endpoints:
    Dns Name:  neo-1.matrix.sn2.dev
    Type:      service
    Dns Name:  neo-1.matrix.us-east4.streamnative.g.sn2.dev
Status:
Events:  <none>
```

### Interact with Clusters using Service Context

Starting from `snctl` version `v1.0.0`, the **Service Context** feature provides a streamlined way to interact directly with your Pulsar clusters (using both Pulsar and Kafka protocols) without needing to manually specify connection details for every command.

A Service Context stores the necessary service URL and authentication information for connecting to a specific cluster. `snctl` typically automatically discovers your StreamNative Cloud Pulsar clusters after you log in (`snctl auth login`) and makes them available as contexts.

**1. Viewing and Switching Contexts**

You usually don't need to manually create contexts for your cloud clusters. To see which context is currently active and switch between available contexts:

* Check the current active context:

  ```bash theme={null}
  snctl context current-context
  ```

  **Output (Example)**

  ```shell theme={null}
  # Assumes 'neo-1' was the last used or default context
  neo-1
  ```

* Switch to a different context (e.g., your cluster named `neo-1`):

  ```bash theme={null}
  # Replace 'neo-1' with your actual cluster name if different
  snctl context use neo-1
  ```

  **Output (Example)**

  ```shell theme={null}
  Context "neo-1" set as current context.
  ```

  Now, all subsequent `pulsar` and `kafka` commands will target the `neo-1` cluster.

**(Note:** For managing contexts related to external, non-StreamNative Cloud clusters, see commands like `snctl context add-external-context`, `snctl context list-external-context`, etc.)

**2. Using `pulsar` and `kafka` Commands with the Active Context**

Once a context is active, you can run `pulsar` and `kafka` commands directly:

* Produce a message to a topic using the Pulsar protocol:

  ```bash theme={null}
  # Assumes 'neo-1' context is active
  # Replace my-tenant/my-namespace/my-topic with your actual topic path
  snctl pulsar client produce my-tenant/my-namespace/my-topic --message "Hello from snctl context!"
  ```

  *(Note: Replace `<organization_name>` with your actual organization name if not set as default)*

* List Pulsar tenants using the admin API:

  ```bash theme={null}
  # Assumes 'neo-1' context is active
  snctl pulsar admin tenants list
  ```

* Consume messages from a topic using the Kafka protocol (requires KoP enabled on the cluster):

  ```bash theme={null}
  # Assumes 'neo-1' context is active
  # Replace my-kafka-topic with your actual topic name
  snctl kafka client consume my-kafka-topic --group my-consumer-group --from-beginning
  ```

* List Kafka topics using the admin API (requires KoP):

  ```bash theme={null}
  # Assumes 'neo-1' context is active
  snctl kafka admin topics list
  ```

**3. Running Commands as a Specific Service Account**

Sometimes you need to perform actions as a specific service account rather than your logged-in user. `snctl` allows this using flags, and it will verify your user has permission to impersonate the specified service account.

* Run a command using a named service account (`--as-service-account`):

  ```bash theme={null}
  # List namespaces as 'my-automation-sa', checking user permission first
  snctl pulsar admin namespaces list --as-service-account my-automation-sa
  ```

* Interactively choose a service account to run as (`--use-service-account`):
  This will prompt you to select from service accounts your user is allowed to impersonate.

  ```bash theme={null}
  # Interactively select an authorized SA to list Kafka consumer groups
  snctl kafka admin groups list --use-service-account
  ```

Using Service Contexts simplifies interactions with your clusters directly through `snctl`, making it a more powerful and unified tool.

### Add clusters to the Pulsar configuration file

This section describes how to add a cluster to the Pulsar client configuration file by defining a `context` for that cluster. Then, you can access the cluster using the `pulsarctl` CLI tool.

1. Add a cluster to the Pulsar client configuration file.

   This example adds a `neo-1` cluster to the Pulsar client configuration file.

   **Input**

   ```
   snctl x update-pulsar-config --cluster-name <neo-1>
   ```

   **Output**

   ```
   Updated Pulsar client configuration for context 'neo-1'.
   ```

2. Verify that the current context has been changed.

   **Input**

   ```
   pulsarctl context current
   ```

   **Output**

   ```
   neo-1
   ```

Then, you can use the `pulsarctl` CLI tool to interact with the target cluster. For details, see [Connect to cluster through pulsarctl](/tools/cli/pulsarctl/pulsarctl-overview) and perform other operations.

### Delete a cluster

<Note title="Note">
  You cannot delete a cluster if there are resources associated with the cluster.
</Note>

You can use the following command to delete a specific cluster based on the cluster name.

```bash theme={null}
snctl delete pulsarcluster <cluster_name>
```

In addition, you can use the following command to delete the cluster based on the cluster name specified in the manifest file.

```bash theme={null}
snctl delete -f ./clusterneo1.yaml
```

## Work with service accounts

<Note title="Note">
  In this section, the organization has the name `matrix` as an example name.
</Note>

### Create a service account

To create a service account through snctl, follow these steps.

1. Define a service account resource named `bot` by using a manifest file and save the manifest file `sa-bot.yaml`.

   ```yaml theme={null}
   apiVersion: cloud.streamnative.io/v1alpha1
   kind: ServiceAccount
   metadata:
     namespace: matrix
     name: bot
   ```

   The following table lists fields in the manifest file.

   | Field      | Description                               |
   | ---------- | ----------------------------------------- |
   | apiVersion | Specify the version of Pulsar API server. |
   | kind       | Specify the component to be created.      |

<br />

2. Apply the manifest file to create the service account.

   ```bash theme={null}
   snctl apply -f /path/to/sa-bot.yaml
   ```

   **Output**

   ```
   serviceaccount.cloud.streamnative.io/bot created
   ```

3. Check whether the service account was created successfully.

   ```bash theme={null}
   snctl describe serviceaccount bot
   ```

   **Output**

   ```
   Name:         bot
   Namespace:    matrix
   Labels:       <none>
   Annotations:  <none>
   API Version:  cloud.streamnative.io/v1alpha1
   Kind:         ServiceAccount
   Metadata:
     Creation Timestamp:  2020-08-11T16:25:10Z
     Finalizers:
       serviceaccount.finalizers.cloud.streamnative.io
     Generation:        1
     Resource Version:  396516
     Self Link:         /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/serviceaccounts/bot
     UID:               874b226b-ea01-41c2-9a7b-059fdcc0d5c1
   Spec:
   Status:
     Conditions:
       Last Transition Time:  2020-08-14T06:25:52Z
       Reason:                Provisioned
       Status:                True
       Type:                  Ready
     Private Key Data:
     Private Key Type:        TYPE_SN_CREDENTIALS_FILE
   Events:                    <none>
   ```

   From the output, you can see that the `status` and `type` parameters for items under `Conditions` are set to `true` and `ready`. This means that the service account `bot` was created successfully.

In addition, you can use the `snctl create serviceaccount SERVICE_ACCOUNT_NAME` command to create a service account. For details, see [snctl reference](https://doc-references.streamnative.io/snctl/latest/index.html#create).

### Download service account credentials

To use a service account, you first download its associated credentials to a JSON file. The information is made available through the `status` block of the `ServiceAccount` resource.

The following example shows how to download the service account credentials to a file called `bot.json`.

```bash theme={null}
snctl auth export-service-account bot --key-file bot.json
```

The file contents will be similar to the following:

```json theme={null}
{
  "type": "SN_SERVICE_ACCOUNT",
  "client_id": "CLIENT_ID",
  "client_secret": "CLIENT_SECRET",
  "client_email": "bot@matrix.auth.streamnative.cloud"
}
```

The following table lists two fields in the JSON file.

| Field           | Description                                              |
| --------------- | -------------------------------------------------------- |
| `client_id`     | It is an Auth0 Application that has been created.        |
| `client_secret` | It is used to authenticate to Auth0 for accessing snctl. |

<Note title="Note">
  The file contains credentials information and should be well protected.
</Note>

### Download service account tokens

To get a token using the StreamNative CLI `snctl`, run the following command.

```shell script theme={null}
snctl auth get-token [PULSAR_INSTANCE_NAME]
```

### Activate a service account

This example shows how to activate a service account through a key file called `bot.json`.

```bash theme={null}
snctl auth activate-service-account --key-file bot.json -a https://api.streamnative.cloud -i https://auth.streamnative.cloud/
```

**Output**

```plain theme={null}
Logged in as bot@matrix.auth.streamnative.cloud.
Welcome to StreamNative Cloud!
```

### Access the StreamNative Cloud API

This example shows how to access the StreamNative Cloud API through a service account.

1. Log in to snctl.

   ```
   snctl auth login
   ```

2. Create a service account.

   This example creates a service account named *bot*.

   ```
   snctl create serviceaccount bot.
   ```

   **Output**

   ```
   serviceaccount.cloud.streamnative.io/bot created
   ```

3. Download the associated credentials of the service account to a JSON file。

   ```
   snctl auth export-service-account bot -f bot.json
   ```

   **Output**

   ```
   Wrote private key file 'bot.json'.
   ```

4. Bind the service account with an "admin" role.

   ```
   snctl create rolebinding bot-cluster-admin --role admin --serviceaccount bot
   ```

   **Output**

   ```
   rolebinding.cloud.streamnative.io/bot-cluster-admin created
   ```

5. Log out from snctl.

   ```
   snctl auth logout
   ```

6. Log in to snctl with the service account.

   ```
   snctl auth activate-service-account --key-file bot.json
   ```

   **Output**

   ```
   Logged in as bot@matrix.auth.streamnative.cloud.
   Welcome to StreamNative Cloud!
   ```

### Check service accounts

This example shows how to check the service accounts of an organization.

```bash theme={null}
snctl get serviceaccount
```

**Output**

```
NAME   CREATED AT
bot   2020-08-11T16:25:10Z
```

From the output of this command, you can see all created service accounts and the time when these service accounts were created.

### Check service account details

Before checking the details about a service account, you should use the following command to confirm whether the service account is available.

```bash theme={null}
snctl get serviceaccount
```

Then, you can use the following command to check details about a service account.

```bash theme={null}
snctl describe serviceaccount SERVICE_ACCOUNT_NAME
```

The following example checks the details about the service account `bot`.

```bash theme={null}
snctl describe serviceaccount bot
```

**Output**

```
Name:         bot
Namespace:    matrix
Labels:       <none>
Annotations:  <none>
API Version:  cloud.streamnative.io/v1alpha1
Kind:         ServiceAccount
Metadata:
  Creation Timestamp:  2020-08-11T16:25:10Z
  Finalizers:
    serviceaccount.finalizers.cloud.streamnative.io
  Generation:        1
  Resource Version:  396516
  Self Link:         /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/serviceaccounts/bot
  UID:               874b226b-ea01-41c2-9a7b-059fdcc0d5c1
Spec:
Status:
  Conditions:
    Last Transition Time:  2020-08-14T06:25:52Z
    Reason:                Provisioned
    Status:                True
    Type:                  Ready
  Private Key Data:
  Private Key Type:        TYPE_SN_CREDENTIALS_FILE
Events:                    <none>
```

### Delete a service account

You can use the following command to delete a service account based on the service account name.

```bash theme={null}
snctl delete serviceaccount SERVICE_ACCOUNT_NAME
```

In addition, you can use the following command to delete the service account based on the name specified in the manifest file.

```
snctl delete -f ./sa-bot.yaml
```

## Work with API Keys

<Note title="Note">
  To work with API keys using `snctl`, you need to snctl version 0.16.0 or later.
</Note>

### Create an API Key

To create an API key that doesn't have an expiration date, do the following:

```bash theme={null}
snctl create apikey test-apikey --service-account-name test-api-key --expiration-time 0 --description 'this is test' -O sndev
```

To create an API key that expires in 30 days, do the following:

```bash theme={null}
snctl create apikey test-apikey2 --service-account-name test-api-key --expiration-time 30d --description 'this is test' -O sndev
```

To create an API key that expires in at a specific time, use the following:

```bash theme={null}
snctl create apikey test-apikey3 --service-account-name test-api-key --expiration-time "2025-02-08T15:38:40Z" --description 'this is test' -O your-org-name
```

### View API Keys

To view API keys, use the following command:

```bash theme={null}
snctl get apikey -O your-org-name
```

### Revoke an API Key

To revoke an API key, use the following:

```bash theme={null}
snctl revoke apikey test-apikey3 -O your-org-name
```

### Delete an API Key

To delete an API key, use the following:

```bash theme={null}
snctl delete apikey test-apikey-name -O your-org-name
```

Deleting an API key will also revoke it.

## Work with users

<Note title="Note">
  In this document, we named the organization `matrix` for an example
</Note>

### Create a user

To create a user, follow these steps.

1. Define a user named `ironman` by using a manifest file and save the manifest file `user-ironman.yaml`.

   ```yaml theme={null}
   apiVersion: cloud.streamnative.io/v1alpha1
   kind: User
   metadata:
     namespace: matrix
     name: ironman
   spec:
     email: ironman@matrix.local
   ```

   The following table lists fields in the manifest file.

   | Field      | Description                               |
   | ---------- | ----------------------------------------- |
   | apiVersion | Specify the version of Pulsar API server. |
   | kind       | Specify the component to be created.      |

<br />

\| spec       | Specify the email address for the user.                                                                                                                             |

2. Apply the manifest file to create the user.

   ```bash theme={null}
   snctl apply -f /path/to/user-ironman.yaml
   ```

3. Check whether the user is created successfully.

   ```bash theme={null}
   snctl describe users ironman@matrix.local
   ```

   **Output**

   ```
   Name:         ironman@matrix.local
   Namespace:    matrix
   Labels:       <none>
   Annotations:  <none>
   API Version:  cloud.streamnative.io/v1alpha1
   Kind:         User
   Metadata:
     Creation Timestamp:  2020-08-10T14:31:12Z
     Finalizers:
       user.finalizers.cloud.streamnative.io
     Generation:        1
     Resource Version:  104126
     Self Link:         /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/users/ironman@matrix.local
     UID:               5f6297c6-8e64-4175-b866-bb6362437f29
   Spec:
     Email:  ironman@matrix.local
   Status:
   Events:  <none>
   ```

### Check users

You can use the following command to check users created for an organization.

```bash theme={null}
snctl get users
```

**Output**

```
NAME                     CREATED AT
ironman@matrix.local   2020-08-10T14:31:12Z
```

### Check user details

Before checking details about a user, you should use the following command to confirm whether the target user is available.

```bash theme={null}
snctl get users
```

Then, you can use the following command to check details about a specific user.

```
snctl describe users USER_NAME
```

The following example checks details about the user `ironman@matrix.local`.

```bash theme={null}
snctl describe users ironman@matrix.local
```

**Output**

```
Name:         ironman@matrix.local
Namespace:    matrix
Labels:       <none>
Annotations:  <none>
API Version:  cloud.streamnative.io/v1alpha1
Kind:         User
Metadata:
  Creation Timestamp:  2020-08-10T14:31:12Z
  Finalizers:
    user.finalizers.cloud.streamnative.io
  Generation:        1
  Resource Version:  104126
  Self Link:         /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/users/ironman@matrix.local
  UID:               5f6297c6-8e64-4175-b866-bb6362437f29
Spec:
  Email:  ironman@matrix.local
Status:
Events:  <none>
```

### Delete a user

You can use the following command to delete a specific user based on the user name.

```bash theme={null}
snctl delete users USER_NAME
```

In addition, you can use the following command to delete the cluster based on the type and name specified in the manifest file.

```
snctl delete -f ./user-ironman.yaml
```


# snctl Usage Conventions
Source: https://docs.streamnative.io/tools/cli/snctl/snctl-usage-conventions



Recommended usage conventions for `snctl`.

## Using `snctl` in Reusable Scripts

For a stable output in a script:

* Request one of the machine-oriented output forms, such as `-o name`, `-o json`, `-o yaml`, or `-o jsonpath`.
* Don't rely on context, preferences, or other implicit states.


# StreamNative CLI Overview
Source: https://docs.streamnative.io/tools/cli/streamnative-cli-overview



StreamNative provides powerful command-line interfaces (CLIs) for managing your messaging infrastructure.

The primary tool is the [StreamNative Cloud CLI (`snctl`)](/tools/cli/snctl/snctl-overview). It serves as a unified interface to:

* Deploy and manage StreamNative Cloud infrastructure (instances, clusters, service accounts, API keys, etc.).
* Interact directly with your Pulsar clusters using the `snctl pulsar` commands (for managing tenants, namespaces, topics, producing/consuming messages, running admin tasks, etc.).
* Interact with Kafka-protocol endpoints (KSN / Ursa) using the `snctl kafka` commands (for managing topics, consumer groups, producing/consuming messages, etc.).

Additionally, the traditional [Pulsar CLI (`pulsarctl`)](/tools/cli/pulsarctl/pulsarctl-overview) remains available. While `snctl` now covers most common Pulsar management tasks for StreamNative Cloud clusters via its Service Context feature, `pulsarctl` is still relevant for:

* Managing self-managed, open-source Apache Pulsar clusters.
* Users who prefer its specific command set or have existing scripts based on it.
* Potentially accessing advanced Pulsar features or configurations not yet exposed through `snctl pulsar admin`.

By primarily leveraging `snctl`, you can automate the complete workflow of provisioning StreamNative Cloud resources and managing the Pulsar/Kafka resources within them. You can supplement with `pulsarctl` for specific OSS or advanced use cases if needed.

## Resources you can manage

### by `snctl`

You can provision or manage the following StreamNative Cloud resources by using `snctl`:

**Generic resources**

* Organization

**Infrastructure resources**

* CloudConnection
* CloudEnvironment
* PulsarGateway
* Pool
* PoolMember

**Instance & Cluster resources**

* PulsarInstance
* PulsarCluster

**Security, access control, and observability resources**

* ServiceAccount
* Role
* RoleBinding
* User
* APIKey

**Pulsar resources on PulsarCluster**

* Cluster
* Broker
* Tenant
* Namespace
* Topic
* Subscription
* Schema
* Source
* Sink
* Function
* Packages

**Kafka resources on KSN enabled PulsarCluster or Ursa engine PulsarCluster**

* Topics
* Partitions
* Groups
* Schema Registry
* Kafka Connect

### by `pulsarctl`

You can provision or manage the following Pulsar resources by using `pulsarctl`:

* Cluster
* Broker
* Tenant
* Namespace
* Topic
* Subscription
* Schema
* Source
* Sink
* Function
* Packages

## Other tools

You can use the other existing tools to interact with your StreamNative clusters.

### Pulsar CLI tools

You can use the existing [OSS Pulsar CLI tools](https://pulsar.apache.org/reference/#/next/cli), such as `pulsar-admin`, `pulsar-client`, and `pulsar-perf`, to manage the Pulsar resources or interact with your StreamNative clusters through the Pulsar protocol.

See [Use Pulsar Tools With StreamNative Cloud](/tools/cli/other-tools/use-pulsar-tools-with-streamnative-cloud) for more information.

### Kafka CLI tools

**Apache Kafka CLI tools**

Apache Kafka provides a suite of command-line interface (CLI) tools that can be accessed from the `/bin` directory after [downloading](https://kafka.apache.org/downloads) and extracting the Kafka distribution. These tools offer a range of capabilities, including starting and stopping Kafka, managing topics, and handling partitions. You can use these tools to interact with your StreamNative clusters through the Kafka protocol.

See [Use Kafka Tools With StreamNative Cloud](/tools/cli/other-tools/use-kafka-tools-with-streamnative-cloud) for more information.

**`kcat`**

[`kcat`](https://github.com/edenhill/kcat) is a popular CLI tool to interact with Apache Kafka. You can use it to interact with your StreamNative clusters as well.

See [Use kcat to interact with StreamNative Cloud](/tools/cli/other-tools/use-kcat-with-streamnative-cloud) for more information.


# StreamNative CLI Tutorial
Source: https://docs.streamnative.io/tools/cli/streamnative-cli-tutorial



This tutorial demonstrates how to use the StreamNative command-line tools to deploy a Serverless cluster and manage Pulsar resources within it. We will primarily focus on the modern, unified approach using [StreamNative Cloud CLI (`snctl`)](/tools/cli/snctl/snctl-overview) for both cloud infrastructure and Pulsar resource management/interaction. We will also show alternative methods using the traditional [`pulsarctl`](/tools/cli/pulsarctl/pulsarctl-overview) and `pulsar-client` tools.

This tutorial covers:

1. Provisioning a Serverless Instance (`snctl`).
2. Provisioning a Serverless Cluster (`snctl`).
3. Provisioning an Application Service Account (`snctl`).
4. Creating an API Key for the Service Account (`snctl`).
5. **Method 1 (Unified `snctl`):**
   * Configuring `snctl` Service Context.
   * Creating Pulsar resources (tenant, namespace, topic) using `snctl pulsar admin`.
   * Granting permissions using `snctl pulsar admin`.
   * Producing/Consuming messages using `snctl pulsar client`.
6. **Method 2 (Traditional Tools - Alternatives):**
   * Configuring `pulsarctl` context using the API Key.
   * Creating Pulsar resources using `pulsarctl`.
   * Granting permissions using `pulsarctl`.
   * Producing/Consuming messages using `pulsar-client` configured with the API Key.

## 0. Prerequisites

### Install Required CLIs

Make sure you have the following installed:

* [snctl](/tools/cli/snctl/snctl-overview) (v1.0.0+ recommended)
* [pulsarctl](/tools/cli/pulsarctl/pulsarctl-overview) (Needed for the alternative method)
* An Apache Pulsar distribution (e.g., from [Pulsar Downloads](https://pulsar.apache.org/download/)) for the `pulsar-client` alternative method.

### Create a new directory for the tutorial

Create a new directory for the tutorial.

```bash theme={null}
mkdir snctl-getting-started && cd snctl-getting-started
```

### Create and Activate a Super-Admin Service Account

You need to [create a service account](/cloud/security/authentication/service-accounts/manage-service-accounts#create-a-service-account) with **Super Admin** access in StreamNative Cloud Console. Let's name it `snctl-super-admin`. After creating the service account, download and save its OAuth2 credentials file (e.g., `snctl-super-admin-credentials.json`).

Activate this service account for `snctl`. This identity will be used for provisioning cloud resources (instances, clusters, other service accounts). Make sure to replace `/path/to/snctl-super-admin-credentials.json` with the actual path.

```bash theme={null}
# Activate the super-admin service account
snctl auth activate-service-account --key-file=/path/to/snctl-super-admin-credentials.json
```

After the service account is activated, you should see a similar message:

```
Logged in as snctl-super-admin@<your-org-id>.auth.streamnative.cloud.
Welcome to StreamNative Cloud!
```

Set the target organization as the default organization for `snctl` to avoid specifying `-n` or `-O` repeatedly. Replace `<your-org-id>` with your actual organization ID.

```bash theme={null}
snctl config set --organization <your-org-id>
```

## 1. Provision a Serverless Instance

Edit a file named `001-instance.yaml` with the following content:

```yaml theme={null}
apiVersion: cloud.streamnative.io/v1alpha1
kind: PulsarInstance
metadata:
  name: <your-instance-name>
  namespace: <your-org-id>
spec:
  availabilityMode: regional
  poolRef:
    name: shared-gcp
    namespace: streamnative
  type: serverless
```

This yaml file defines a Serverless Instance running in GCP with regional availability mode. Replace the following placeholders with your actual values:

* `<your-instance-name>`: The name of the Serverless Instance.
* `<your-org-id>`: The organization ID.

Run the following command to provision the Serverless Instance:

```bash theme={null}
snctl create -f 001-instance.yaml
```

You should see the following message:

```bash theme={null}
pulsarinstance.cloud.streamnative.io/<your-instance-name> created
```

Query the instance to verify the instance is created.

```bash theme={null}
snctl get PulsarInstance <your-instance-name> -o yaml
```

You will be able to see a similar status block of this instance in the output:

```yaml theme={null}
status:
  auth:
    oauth2:
      audience: urn:sn:pulsar:<your-org-id>:<your-instance-name>
      issuerURL: https://auth.streamnative.cloud/
    type: oauth2
  conditions:
    - lastTransitionTime: '...'
      message: a payment method is not required because discount is active
      reason: HasActiveDiscount
      status: 'True'
      type: SubscriptionReady
    - lastTransitionTime: '...'
      reason: Created
      status: 'True'
      type: ResourceServerReady
    - lastTransitionTime: '...'
      reason: Created
      status: 'True'
      type: ServiceAccountReady
    - lastTransitionTime: '...'
      reason: AllConditionStatusTrue
      status: 'True'
      type: Ready
```

When all the conditions are `True`, the instance is ready.

## 2. Provision a Serverless Cluster

Edit a file named `002-cluster.yaml` with the following content:

```yaml theme={null}
apiVersion: cloud.streamnative.io/v1alpha1
kind: PulsarCluster
metadata:
  namespace: <your-org-id>
spec:
  # currently the `broker` section is still required despite the fact that
  # settings are not used by serverless
  broker:
    replicas: 2
    resources:
      cpu: '1'
      memory: 4Gi
  displayName: serverless-cluster
  instanceName: <your-instance-name>
  location: us-central1
```

This yaml file defines a Serverless Cluster in `us-central1` region. Replace the following placeholders with your actual values:

* `<your-instance-name>`: The name of the Serverless Instance.
* `<your-org-id>`: The organization ID.

Run the following command to provision the Serverless Instance:

```bash theme={null}
snctl create -f 002-cluster.yaml
```

You should see the following message:

```bash theme={null}
pulsarcluster.cloud.streamnative.io/<your-cluster-name> created
```

<Note title="Note">
  Please note the `<your-cluster-name>` in the output message because the cluster name of a Serverless Cluster is generated by StreamNative Cloud. You will need this cluster name in the future steps.
</Note>

Query the cluster to verify the cluster is created.

```bash theme={null}
snctl get PulsarCluster <your-cluster-name> -o yaml
```

You will be able to see a similar status block of this cluster in the output:

```yaml theme={null}
status:
  broker:
    readyReplicas: 2
    replicas: 2
    updatedReplicas: 2
  conditions:
    - lastTransitionTime: '...'
      reason: Deploy
      status: 'True'
      type: PulsarBrokerReady
    - lastTransitionTime: '...'
      reason: AllConditionStatusTrue
      status: 'True'
      type: Ready
    - lastTransitionTime: '...'
      reason: Ready
      status: 'True'
      type: PulsarInstanceReady
  deploymentType: hosted
  instanceType: serverless
```

Wait for all the conditions to be `True`, then the cluster is ready. A Serverless Cluster is usually ready within 1\~2 minutes.

## 3. Provision a Service Account

Edit a file named `003-sa.yaml` with the following content:

```yaml theme={null}
apiVersion: cloud.streamnative.io/v1alpha1
kind: ServiceAccount
metadata:
  name: <your-service-account-name>
  namespace: <your-org-id>
spec: {}
```

This yaml file defines a Service Account with a name `<your-service-account-name>`. Replace the following placeholders with your actual values:

* `<your-service-account-name>`: The name of the Service Account.
* `<your-org-id>`: The organization ID.

Run the following command to provision the Service Account:

```bash theme={null}
snctl create -f 003-sa.yaml
```

You should see the following message:

```bash theme={null}
serviceaccount.cloud.streamnative.io/<your-service-account-name> created
```

Query the service account to verify the service account is created.

```bash theme={null}
snctl get ServiceAccount <your-service-account-name> -o yaml
```

You will be able to see a similar status block of this service account in the output:

```yaml theme={null}
status:
  conditions:
    - lastTransitionTime: '...'
      reason: Provisioned
      status: 'True'
      type: Ready
  privateKeyData: ...
  privateKeyType: TYPE_SN_CREDENTIALS_FILE
```

Wait until the `Ready` condition is `True`, then the service account is ready.

## 4. Create an API Key for the Service Account (Optional but shown for completeness)

While `snctl` typically uses OAuth2 via `auth activate-service-account` or context impersonation (`--as-service-account`), you might need an API Key for external clients or tools that only support token authentication. This API key will be used in the *alternative* `pulsarctl` and `pulsar-client` method later.

Edit a file named `004-api-key.yaml`:

```yaml theme={null}
apiVersion: cloud.streamnative.io/v1alpha1
kind: APIKey
metadata:
  name: <your-api-key-name>
  namespace: <your-org-id>
spec:
  description: This is a test api key for <your-service-account-name> in running the snctl tutorial
  instanceName: <your-instance-name>
  serviceAccountName: <your-service-account-name>
```

This yaml file defines an API Key for the Service Account `<your-service-account-name>`. Replace the following placeholders with your actual values:

* `<your-api-key-name>`: The name of the API Key.
* `<your-service-account-name>`: The name of the Service Account.
* `<your-instance-name>`: The name of the Serverless Instance.
* `<your-org-id>`: The organization ID.

Run the following command to create the API Key:

```bash theme={null}
snctl create -f 004-api-key.yaml
```

You should see the following message:

```bash theme={null}
apikey.cloud.streamnative.io/<your-api-key-name> created
```

Query the API Key to verify the API Key is created and optionally retrieve the token.

```bash theme={null}
snctl get apikey <your-api-key-name> -o yaml
```

You will be able to see a similar status block of this API Key in the output:

```yaml theme={null}
status:
  conditions:
    - lastTransitionTime: '...'
      message: ''
      reason: API Key has been provisioned
      status: 'True'
      type: Issued
    - lastTransitionTime: '...'
      message: ''
      reason: API Key is not revoked
      status: 'False'
      type: Revoked
    - lastTransitionTime: '...'
      message: ''
      reason: API Key will never expire
      status: 'False'
      type: Expired
  expiresAt: '1970-01-01T00:00:00Z'
  issuedAt: '...'
  keyId: <key-id>
  token: <token>
```

Wait until the `Issued` condition is `True`, then the API Key is issued and ready to use. You can obtain the token from the `token` field in the status block.

You can use the following command to obtain the token and export it as an environment variable `API_KEY_TOKEN`:

```bash theme={null}
export API_KEY_TOKEN=$(snctl get apikey <your-api-key-name> -o jsonpath='{.status.token}')
```

***

## Method 1: Unified Management with `snctl`

This section demonstrates using `snctl` for configuring access, managing Pulsar resources, and interacting with the data plane.

## 5. Configure `snctl` Service Context for Pulsar Interaction

`snctl` uses Service Contexts to manage connections to Pulsar/Kafka clusters. After creating a cluster, `snctl` usually discovers it automatically. Let's explicitly set the context for the cluster we created to ensure subsequent commands target it correctly.

Set the active context to your newly created cluster. Replace `<your-instance-name>` and `<your-cluster-name>` with the actual name from step 2.

```bash theme={null}
snctl context use --pulsar-instance <your-instance-name> --pulsar-cluster <your-cluster-name>
```

Verify the current context:

```bash theme={null}
snctl context current
```

Now, verify connectivity by listing tenants. Since we activated the `snctl-super-admin` service account (in step 0), `snctl` commands will run as that identity by default.

```bash theme={null}
snctl pulsar admin tenants list
```

You should see the default tenants:

```bash theme={null}
public
pulsar
sn
```

This confirms `snctl` can communicate with the Pulsar cluster's admin endpoint using the super-admin credentials via the active context.

## 6. Create Pulsar Resources using `snctl`

Now, let's create the tenant, namespace, and topic for our application `sl-app` using `snctl pulsar admin` commands. These commands will use the active context (`<your-cluster-name>`) and run as the activated super-admin user (`snctl-super-admin`).

First, create a tenant named `sl-app-tenant`.

```bash theme={null}
snctl pulsar admin tenants create sl-app-tenant --allowed-clusters <your-cluster-name>
```

Output:

```bash theme={null}
Tenant "sl-app-tenant" created successfully.
```

Second, create a namespace named `sl-app-ns` under the tenant `sl-app-tenant`.

```bash theme={null}
snctl pulsar admin namespaces create sl-app-tenant/sl-app-ns --clusters <your-cluster-name>
```

Output:

```bash theme={null}
Namespace "sl-app-tenant/sl-app-ns" created successfully.
```

Next, create a partitioned topic named `sl-app-topic` with 4 partitions under the namespace `sl-app-tenant/sl-app-ns`.

```bash theme={null}
snctl pulsar admin topics create sl-app-tenant/sl-app-ns/sl-app-topic 4
```

Output:

```bash theme={null}
Create topic persistent://sl-app-tenant/sl-app-ns/sl-app-topic with 4 partitions successfully
```

Finally, grant the application Service Account `<your-service-account-name>` (created in step 3) the permission to produce and consume messages within the `sl-app-tenant/sl-app-ns` namespace. We are still running as `snctl-super-admin` to grant these permissions. Replace `<your-service-account-name>` with the name from step 3.

```bash theme={null}
snctl pulsar admin namespaces grant-permission --role <your-service-account-name>@<your-org-id>.auth.streamnative.cloud --actions produce,consume sl-app-tenant/sl-app-ns
```

Output:

```bash theme={null}
Grant permissions [produce consume] to the client role <your-service-account-name>@<your-org-id>.auth.streamnative.cloud to access the namespace sl-app-tenant/sl-app-ns successfully
```

## 7. Produce and consume messages using `snctl pulsar client`

Now we'll use `snctl pulsar client` commands to produce and consume messages. Crucially, these actions should be performed *as the application service account* (`<your-service-account-name>`) because we granted *it* the produce/consume permissions, not the super-admin. We use the `--as-service-account` flag for this, leveraging `snctl`'s ability to impersonate the specified service account (assuming the logged-in super-admin has permission to do so, which is typical).

Produce 10 messages to the topic, acting as the application service account. Replace `<your-service-account-name>` with the name from step 3.

```bash theme={null}
snctl pulsar client produce --topic sl-app-tenant/sl-app-ns/sl-app-topic \
  --messages "hello sl-app from snctl" \
  --num-times 10 \
  --as-service-account <your-service-account-name>
```

You should see output indicating successful production, similar to:

```bash theme={null}
Successfully produced 10 message(s) to topic persistent://sl-app-tenant/sl-app-ns/sl-app-topic
```

*(Exact output message may vary)*

Consume the 10 messages from the topic, again acting as the application service account. Replace `<your-service-account-name>` with the name from step 3.

```bash theme={null}
snctl pulsar client consume --topic sl-app-tenant/sl-app-ns/sl-app-topic \
  --subscription-name sl-app-sub \
  --num-messages 10 \
  --initial-position earliest \
  --as-service-account <your-service-account-name>
```

You should see the 10 messages printed to your console, similar to:

```bash theme={null}
sidebarTitle: Tutorial
----- got message -----
# ... message details ... content:hello sl-app from snctl
----- got message -----
# ... message details ... content:hello sl-app from snctl
... (10 messages total) ...
```

Followed by a confirmation like:

```bash theme={null}
Consumed 10 message(s) from topic sl-app-tenant/sl-app-ns/sl-app-topic
```

*(Exact output format may vary)*

This demonstrates using `snctl` for the entire lifecycle using the unified approach.

## sidebarTitle: Tutorial

## Method 2: Traditional Management with `pulsarctl` and `pulsar-client` (Alternative)

This section demonstrates the alternative approach using the separate `pulsarctl` tool for admin tasks and the `pulsar-client` tool for producing/consuming. This method often relies on API Key authentication for simplicity when interacting with StreamNative Cloud clusters via these tools.

## Alt 5. Configure `pulsarctl` Context (Using API Key)

First, get the admin service URL of the cluster by running the following command:

```bash theme={null}
export ADMIN_SERVICE_URL="https://$(snctl get PulsarCluster <your-cluster-name> -o jsonpath='{.spec.serviceEndpoints[0].dnsName}')"
export BROKER_SERVICE_URL="pulsar+ssl://$(snctl get PulsarCluster <your-cluster-name> -o jsonpath='{.spec.serviceEndpoints[1].dnsName}'):6651"
```

Once you get the `ADMIN_SERVICE_URL`, you can use the following command to configure `pulsarctl` to access the cluster we created in the previous steps:

```bash theme={null}
pulsarctl context set -s ${ADMIN_SERVICE_URL} --key-file /path/to/oauth2-credentials-file.json --audience urn:sn:pulsar:<your-org-id>:<your-instance-name> <your-cluster-name>-admin
```

This command will create a new context named `<your-cluster-name>-admin` and update the `pulsarctl` configuration to use the oauth2 credentials of `snctl-super-admin` to authenticate to the cluster.

You should see the following message:

```bash theme={null}
Context "<your-cluster-name>-admin" created.
```

You can verify the `pulsarctl` has been configured properly by running the following command:

```bash theme={null}
pulsarctl context current
```

You should see the following message:

```bash theme={null}
<your-cluster-name>-admin
```

Then you can run `pulsarctl tenants list` to verify if you configured the `pulsarctl` properly.

```bash theme={null}
pulsarctl tenants list -o yaml
```

You should be able to see the tenants in the cluster.

```bash theme={null}
- public
- pulsar
- sn
```

## Alt 6. Create Pulsar Resources using `pulsarctl`

Assume you want to build a sample application `sl-app` that produces messages to a topic `persistent://sl-app-tenant/sl-app-ns/sl-app-topic` and consumes messages from the same topic.

First, create a tenant named `sl-app-tenant`.

```bash theme={null}
pulsarctl tenants create sl-app-tenant --allowed-clusters <your-cluster-name>
```

Output:

```bash theme={null}
Create tenant sl-app-tenant successfully
```

Second, create a namespace named `sl-app-ns` under the tenant `sl-app-tenant`.

```bash theme={null}
pulsarctl namespaces create sl-app-tenant/sl-app-ns --clusters <your-cluster-name>
```

Output:

```bash theme={null}
Created sl-app-tenant/sl-app-ns successfully
```

Next, create a topic named `sl-app-topic` with 4 partitions under the namespace `sl-app-tenant/sl-app-ns`.

```bash theme={null}
pulsarctl topics create sl-app-tenant/sl-app-ns/sl-app-topic 4
```

Output:

```bash theme={null}
Create topic persistent://sl-app-tenant/sl-app-ns/sl-app-topic with 4 partitions successfully
```

Finally, grant the Service Account `<your-service-account-name>` the permission to produce and consume messages from the namespace `sl-app-tenant/sl-app-ns`.

```bash theme={null}
pulsarctl namespaces grant-permission --role <your-service-account-name>@<your-org-id>.auth.streamnative.cloud --actions produce,consume sl-app-tenant/sl-app-ns
```

Output:

```bash theme={null}
Grant permissions [produce consume] to the client role <your-service-account-name>@<your-org-id>.auth.streamnative.cloud to access the namespace sl-app-tenant/sl-app-ns successfully
```

## Alt 7. Produce and consume messages using `pulsar-client`

1. Download the Pulsar distribution from [Pulsar Downloads](https://pulsar.apache.org/download/). Assume you have downloaded the Pulsar distribution and extracted the tarball to `/path/to/pulsar-dist`.

2. Enter the root directory of the Pulsar distribution:

   ```bash theme={null}
   cd /path/to/pulsar-dist
   ```

3. Configure the `conf/client.conf` file:

   * **webServiceUrl**: Set the `webServiceUrl` to the `ADMIN_SERVICE_URL` you obtained in the previous steps.
   * **brokerServiceUrl**: Set the `brokerServiceUrl` to the `BROKER_SERVICE_URL` you obtained in the previous steps.
   * **authPlugin**: Set the `authPlugin` to `org.apache.pulsar.client.impl.auth.AuthenticationToken`.
   * **authParams**: Set the `authParams` to be `token:<your-api-key>`. `<your-api-key>` is the API Key you obtained in the previous steps.

4. Produce 10 messages.

   ```bash theme={null}
   bin/pulsar-client produce -m "hello sl-app" -n 10 sl-app-tenant/sl-app-ns/sl-app-topic
   ```

   You should see a similar message in the output:

   ```bash theme={null}
   10 messages successfully produced
   ```

5. Consume the messages.

   ```bash theme={null}
   bin/pulsar-client consume -n 10 -p Earliest -s sl-app-sub sl-app-tenant/sl-app-ns/sl-app-topic
   ```

   You should see a similar message in the output:

   ```bash theme={null}
   ----- got message -----
   publishTime:[1732951012862], eventTime:[0], key:[null], properties:[], content:hello sl-app
   ----- got message -----
   publishTime:[1732951012952], eventTime:[0], key:[null], properties:[], content:hello sl-app
   ----- got message -----
   publishTime:[1732951013162], eventTime:[0], key:[null], properties:[], content:hello sl-app
   ----- got message -----
   publishTime:[1732951013095], eventTime:[0], key:[null], properties:[], content:hello sl-app
   ----- got message -----
   publishTime:[1732951013299], eventTime:[0], key:[null], properties:[], content:hello sl-app
   ----- got message -----
   publishTime:[1732951013368], eventTime:[0], key:[null], properties:[], content:hello sl-app
   ----- got message -----
   publishTime:[1732951013436], eventTime:[0], key:[null], properties:[], content:hello sl-app
   ----- got message -----
   publishTime:[1732951013510], eventTime:[0], key:[null], properties:[], content:hello sl-app
   ----- got message -----
   publishTime:[1732951013025], eventTime:[0], key:[null], properties:[], content:hello sl-app
   ----- got message -----
   publishTime:[1732951013229], eventTime:[0], key:[null], properties:[], content:hello sl-app
   ```

   You should see a final message in the output:

   ```bash theme={null}
   10 messages successfully consumed
   ```

***

## 8. Next Steps

Once you have verified the application works as expected using either method, you can try out more guided tutorials:

* [Kafka Client Guides](/clients/kafka-clients/kafka-clients-overview)
* [Pulsar Client Guides](/clients/pulsar-clients/pulsar-clients-overview)
* [Run Pulsar I/O Connectors](/cloud/connect/connector-index)
* [Run Kafka Connect Connectors](/cloud/connect/kafka-connect/kafka-connect-overview)
* [Deploy Pulsar Functions](/cloud/process/pulsar-functions/functions-overview)

## 9. Clean up

After you finish the tutorial, you can clean up the resources you created in this tutorial by running the following command:

<Note title="Note">
  Please note that you can't use `snctl delete -f 002-cluster.yaml` to delete the cluster because the cluster name is generated by StreamNative Cloud. So you need to delete the cluster using the `snctl delete PulsarCluster <your-cluster-name>` command.
</Note>

```bash theme={null}
snctl delete -f 004-api-key.yaml
snctl delete -f 003-sa.yaml
snctl delete PulsarCluster <your-cluster-name>
snctl delete -f 001-instance.yaml
```


# Configure Pulsar Terraform Provider
Source: https://docs.streamnative.io/tools/terraform/configure/configure-pulsar-provider



## Install the Provider

You can install the Pulsar Terraform Provider using the following code snippet.

```hcl theme={null}
terraform {
  required_providers {
    pulsar = {
      source  = "streamnative/pulsar"
    }
  }
}
```

## Configure the Provider

Once you have added the provider to your Terraform configuration, you need to configure the provider with your Pulsar Cluster service url and credentials.

Below is an example of configuring the provider to connect to a Pulsar cluster at `<pulsar-cluster-service-url>` using Token authentication. Please make sure the token has the Super Admin permission in order to create resources within the Pulsar cluster.

```hcl theme={null}
provider "pulsar" {
  web_service_url = "<pulsar-cluster-service-url>"
  token = "<pulsar-cluster-token>"
}
```

## Configure the Provider with Different Authentication Methods

### Token Authentication

You can specify the token in the provider configuration. Make sure the token has the Super Admin permission in order to create resources within the Pulsar cluster.

```hcl theme={null}
provider "pulsar" {
  web_service_url = "<pulsar-cluster-service-url>"
  token = "<pulsar-cluster-token>"
}
```

### OAuth2 Authentication

You can specify the OAuth2 key file path in the provider configuration. If your cluster is configured to require audience, you also need to specify the audience in the provider configuration.

```hcl theme={null}
provider "pulsar" {
  web_service_url = "<pulsar-cluster-service-url>"
  key_file_path = "<path-to-your-oauth2-key-file>"
  audience = "<audience>"
}
```

## Validate the Configuration

After you have configured the provider in your Terraform configuration, you can run the following command to download and install the providers defined in the configuration:

```sh theme={null}
terraform init
```

You can then run the following command to ensure the configuration is syntactically valid and internally consistent:

```sh theme={null}
terraform validate
```

Apply the configuration:

```sh theme={null}
terraform plan
```

If you are not able to connect to the Pulsar cluster, you will see an error message similar to the following:

```sh theme={null}
Error: failed to create pulsar oauth2 provider: authentication failed using client credentials: <error-message>
```

Otherwise, you should see a successful output.

## Full Code Example

Below is the full code example of how to configure the Pulsar Terraform Provider.

```hcl theme={null}
terraform {
  required_providers {
    pulsar = {
      source = "streamnative/pulsar"
    }
  }
}

provider "pulsar" {
  web_service_url = "<pulsar-cluster-service-url>"
  token = "<your-api-key>"
}
```


# Configure StreamNative Terraform Provider
Source: https://docs.streamnative.io/tools/terraform/configure/configure-streamnative-provider



## Install the Provider

You can install the StreamNative Terraform Provider using the following code snippet.

```hcl theme={null}
terraform {
  required_providers {
    streamnative = {
      source  = "streamnative/streamnative"
      version = ">= 0.7.0"
    }
  }
}
```

## Configure the Provider

Once you have added the provider to your Terraform configuration, you need to configure the provider with your StreamNative Cloud credentials.

StreamNative Cloud currently only supports OAuth2 authentication. You need to [create a service account with **Super Admin** access](/cloud/security/authentication/service-accounts/manage-service-accounts#create-a-service-account) and [download the OAuth2 credentials file](/cloud/security/authentication/service-accounts/use-oauth/oauth-overview#credentials-file). Assume your service account name is `test-sa` and the OAuth2 key file is stored in `/path/to/your/service/account/key.json`. You can configure the provider as follows:

```hcl theme={null}
provider "streamnative" {
  key_file_path = "/path/to/your/service/account/key.json"
}
```

Alternatively, if you can't access file in your Terraform setup, you can choose to configure the provider with `client_id` and `client_secret`. You can put the actual values of `client_id` and `client_secret` as [system environment variables](https://support.hashicorp.com/hc/en-us/articles/4547786359571-Reading-and-using-environment-variables-in-Terraform-runs) or in the Terraform configuration file.

```hcl theme={null}
provider "streamnative" {
  client_id     = "<client-id>"
  client_secret = "<client-secret>"
}
```

## Validate the Configuration

In order to validate the provider is configured correctly, you can try to access the [Service Account](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/data-sources/service_account) data source of the SA you used to configure the provider. Below is an example of how to do so. You can replace `<organization>` and `<service-account-name>` with your actual values.

```hcl theme={null}
data "streamnative_service_account" "test-sa" {
  organization = "<organization>"
  name         = "<service-account-name>"
}

output "service_account_id" {
  value = data.streamnative_service_account.test-sa
}
```

After you have configured the provider and added the code snippet above to your Terraform configuration, you can run the following command to download and install the providers defined in the configuration:

```sh theme={null}
terraform init
```

You can then run the following command to ensure the configuration is syntactically valid and internally consistent:

```sh theme={null}
terraform validate
```

Apply the configuration:

```sh theme={null}
terraform apply
```

You should be able to see a similar output as follows:

```sh theme={null}
...

service_account_id = {
  "admin" = true
  "id" = "<organization>/<service-account-name>"
  "name" = "<service-account-name>"
  "organization" = "<organization>"
  "private_key_data" = "<private-key-data>"
}
```

## Full Code Example

Below is the full code example of how to configure the StreamNative Terraform Provider.

```hcl theme={null}
terraform {
  required_providers {
    streamnative = {
      source  = "streamnative/streamnative"
    }
  }
}

provider "streamnative" {
  client_id     = "<client-id>"
  client_secret = "<client-secret>"
}

data "streamnative_service_account" "test-sa" {
  organization = "<organization>"
  name         = "<service-account-name>"
}

output "service_account_id" {
  value = data.streamnative_service_account.test-sa
}
```


# StreamNative Terraform Provider Overview
Source: https://docs.streamnative.io/tools/terraform/terraform-provider-overview



## What is Terraform?

[HashiCorp Terraform](https://developer.hashicorp.com/terraform/docs) is an open source **infrastructure-as-code** tool that lets you build, change, and version your cloud data infrastructure in a safe, efficient way. You program Terraform with human-readable configuration files that you can version, reuse, share, and deploy in your CI/CD pipelines.

## Why Terraform and StreamNative?

StreamNative provides two terraform providers for your to deploy and manage StreamNative Cloud infrastructure and Pulsar resources respectively.

* [StreamNative Terraform Provider](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs) to deploy and manage StreamNative Cloud infrastructure
* [Pulsar Terraform Provider](https://registry.terraform.io/providers/streamnative/pulsar/latest/docs) to manage the Pulsar resources in your StreamNative Cloud clusters

By leveraging the two providers together, you can automate the workflow of managing instances, clusters, tenants, namespaces, topics, and other resources in StreamNative Cloud.

These are some of the benefits you get with using the StreamNative & Pulsar Terraform Providers:

* **Human Readable Configuration**: Define your infrastructure in Terraform configuration files that are human readable and can be versioned, reused, shared, and deployed in your CI/CD pipelines.
* **Consistent Infrastructure**: Provision and manage your StreamNative Cloud infrastructure and Pulsar resources safely and efficiently throughout its lifecycle.
* **Cloud Flexibility**: Deploy your StreamNative Cloud infrastructure seamlessly across different cloud providers.
* **Scale Quickly**: Provision complicated and dependent infrastructure and resources quickly and easily.
* **Open Standard**: Enable industry standard GitOps workflows and infrastructure-as-code practices.

## Tutorials

## Configure the providers

## Resources you can manage

### by StreamNative Provider

You can provision the following StreamNative Cloud resources and get data from these data sources in your Terraform configuration files using the StreamNative provider:

**Infrastructure & Networking**

| Resource                                                                                                                       | Data Source                                                                                                                       |
| ------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- |
| [Cloud Connection](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/resources/cloud_connection)   | [Cloud Connection](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/data-sources/cloud_connection)   |
| [Cloud Environment](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/resources/cloud_environment) | [Cloud Environment](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/data-sources/cloud_environment) |
| [Pulsar Gateway](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/resources/pulsar_gateway)       | [Pulsar Gateway](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/data-sources/pulsar_gateway)       |
|                                                                                                                                | [Pool](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/data-sources/pool)                           |
|                                                                                                                                | [Pool Member](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/data-sources/pool_member)             |

**Instance & Cluster**

| Resource                                                                                                                   | Data Source                                                                                                                   |
| -------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| [Pulsar Instance](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/resources/pulsar_instance) | [Pulsar Instance](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/data-sources/pulsar_instance) |
| [Pulsar Cluster](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/resources/pulsar_cluster)   | [Pulsar Cluster](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/data-sources/pulsar_cluster)   |

**Security, access control and identity**

| Resource                                                                                                                                   | Data Source                                                                                                                                   |
| ------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- |
| [Service Account](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/resources/service_account)                 | [Service Account](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/data-sources/service_account)                 |
| [Service Account Binding](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/resources/service_account_binding) | [Service Account Binding](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/data-sources/service_account_binding) |
| [API Key](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/resources/apikey)                                  | [API Key](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs/data-sources/apikey)                                  |

### by Pulsar Provider

You can provision the following Pulsar resources in your Terraform configuration files using the Pulsar provider:

| Resource                                                                                                        | Data Source |
| --------------------------------------------------------------------------------------------------------------- | ----------- |
| [Pulsar Cluster](https://registry.terraform.io/providers/streamnative/pulsar/latest/docs/resources/cluster)     |             |
| [Pulsar Tenant](https://registry.terraform.io/providers/streamnative/pulsar/latest/docs/resources/tenant)       |             |
| [Pulsar Namespace](https://registry.terraform.io/providers/streamnative/pulsar/latest/docs/resources/namespace) |             |
| [Pulsar Topic](https://registry.terraform.io/providers/streamnative/pulsar/latest/docs/resources/topic)         |             |
| [Pulsar Function](https://registry.terraform.io/providers/streamnative/pulsar/latest/docs/resources/function)   |             |
| [Pulsar Source](https://registry.terraform.io/providers/streamnative/pulsar/latest/docs/resources/source)       |             |
| [Pulsar Sink](https://registry.terraform.io/providers/streamnative/pulsar/latest/docs/resources/sink)           |             |

## References

* For the StreamNative provider source code, see [streamnative/terraform-provider-streamnative](https://github.com/streamnative/terraform-provider-streamnative)
* For the Pulsar provider source code, see [streamnative/terraform-provider-pulsar](https://github.com/streamnative/terraform-provider-pulsar)
* For sample configuration files for StreamNative resources, see [streamnative/terraform-provider-streamnative/examples](https://github.com/streamnative/terraform-provider-streamnative/tree/main/examples)
* For sample configuration files for Pulsar resources, see [streamnative/terraform-provider-pulsar/examples](https://github.com/streamnative/terraform-provider-pulsar/tree/master/examples)


# StreamNative Terraform Tutorial
Source: https://docs.streamnative.io/tools/terraform/terraform-provider-tutorial



This tutorial demonstrates how to use [StreamNative Terraform Provider](https://registry.terraform.io/providers/streamnative/streamnative/latest/docs) to deploy a Serverless cluster and use the [Pulsar Terraform Provider](https://registry.terraform.io/providers/streamnative/pulsar/latest/docs) to provision Pulsar resources in the Serverless cluster.

This tutorial provisions the following resources (assuming we name the application as `sl-app`):

1. Provisions a Serverless Instance (i.e., `sl-instance`).
2. Provisions a Serverless Cluster (i.e., `sl-clu`).
3. Provisions a Service Account named `sl-app-sa`.
4. Provisions an API Key for the Service Account `sl-app-apikey`.
5. Provisions a Pulsar Tenant named `sl-app-tenant`
6. Provisions a Pulsar Namespace `sl-app-ns` and grants `produce` and `consume` permissions to the Service Account on the namespace.
7. Provisions a Pulsar Topic with 4 partitions `sl-app-topic`.
8. After provisioning all the resources, we will verify the resources by using the [pulsarctl](/tools/cli/pulsarctl/pulsarctl-overview) command.

The code examples is available in the [examples/terraform/serverless](https://github.com/streamnative/terraform-provider/tree/main/examples/terraform/serverless) folder.

## 0. Prerequisites

Create a new directory anywhere you'd like for this project.

```bash theme={null}
mkdir terraform-getting-started && cd terraform-getting-started
```

## 1. Create a Super-Admin Service Account

First, you need to create a service account called `tf-runner` with **Super Admin** access. Please refer to [Create a Service Account](/cloud/security/authentication/service-accounts/manage-service-accounts#create-a-service-account) for details.

After you have created the service account, download the OAuth2 credentials file and save it as `tf-runner.json` in the `terraform-getting-started` folder that you created earlier.

## 2. Create Terraform Configuration Files to provision StreamNative Cloud resources

### 2.1 Create a Module Folder

Create a module folder `streamnative_cloud`.

```bash theme={null}
mkdir streamnative_cloud && cd streamnative_cloud
```

### 2.2 Create Variables File

Create a `variables.tf` file inside the `streamnative_cloud` folder and add the following code snippet to prepare the variables. Remember to replace `<your-organization-id>` with your StreamNative Cloud organization id.

* **org\_id**: Get your StreamNative Cloud organization id from [here](https://docs.streamnative.io/docs/organizations#cloud-organization-id) and replace `<your-organization-id>` with it.
* **instance\_name**: The name of the Pulsar instance. Default value is `sl-instance`.
* **cluster\_name**: The name of the Pulsar cluster. Default value is `sl-clu`.
* **app\_name**: The name of the application. Default value is `sl-app`.

```hcl theme={null}
variable "org_id" {
  type = string
}

variable "instance_name" {
  type = string
}

variable "cluster_name" {
  type = string
}

variable "app_name" {
  type = string
}

```

### 2.3 Create Terraform Configuration File

Create a `main.tf` file inside the `streamnative_cloud` folder and add the following code snippet to create the resources.

<Note title="Note">
  Use the StreamNative Terraform Provider version `0.7.0` or later.
</Note>

```hcl theme={null}

terraform {
  required_providers {
    streamnative = {
      source  = "streamnative/streamnative"
      version = ">= 0.7.0"
    }
  }
}

provider "streamnative" {
  key_file_path = "./tf-runner.json"
}

data "streamnative_service_account" "tf-runner" {
  organization = var.org_id
  name = "tf-runner"
}

resource "streamnative_pulsar_instance" "serverless-instance" {
  organization = var.org_id
  name = var.instance_name
  availability_mode = "regional"
  pool_name = "shared-gcp"
  pool_namespace = "streamnative"
  type = "serverless"
}

data "streamnative_pulsar_instance" "serverless-instance" {
  depends_on = [streamnative_pulsar_instance.serverless-instance]
  name = streamnative_pulsar_instance.serverless-instance.name
  organization = streamnative_pulsar_instance.serverless-instance.organization
}

resource "streamnative_pulsar_cluster" "serverless-cluster" {
  depends_on = [streamnative_pulsar_instance.serverless-instance]
  organization    = streamnative_pulsar_instance.serverless-instance.organization
  name            = var.cluster_name
  display_name    = "serverless-cluster"
  instance_name   = streamnative_pulsar_instance.serverless-instance.name
  location        = "us-central1"
}

data "streamnative_pulsar_cluster" "serverless-cluster" {
  depends_on   = [streamnative_pulsar_cluster.serverless-cluster]
  organization = streamnative_pulsar_cluster.serverless-cluster.organization
  name         = split("/", streamnative_pulsar_cluster.serverless-cluster.id)[1]
}

resource "streamnative_service_account" "app-sa" {
  organization = var.org_id
  name = "${var.app_name}-sa"
  admin = false
}

data "streamnative_service_account" "app-sa" {
  depends_on = [streamnative_service_account.app-sa]
  organization = streamnative_service_account.app-sa.organization
  name = streamnative_service_account.app-sa.name
}

resource "streamnative_apikey" "app-apikey" {
  depends_on = [streamnative_pulsar_cluster.serverless-cluster, streamnative_service_account.app-sa]
  organization = var.org_id
  name = "${var.app_name}-apikey2"
  instance_name = streamnative_pulsar_instance.serverless-instance.name
  service_account_name = streamnative_service_account.app-sa.name
  description = "This is a test api key for ${var.app_name} in running the terraform tutorial"
  # If you don't want to set expiration time, you can set expiration_time to "0"
  # expiration_time = "2025-01-01T10:00:00Z"
  expiration_time = "0"
}

data "streamnative_apikey" "app-apikey" {
  depends_on = [streamnative_apikey.app-apikey]
  organization = streamnative_apikey.app-apikey.organization
  name = streamnative_apikey.app-apikey.name
  private_key = streamnative_apikey.app-apikey.private_key
}

output "apikey_token" {
  value = data.streamnative_apikey.app-apikey.token
}

output "pulsar_web_service_url" {
  value = data.streamnative_pulsar_cluster.serverless-cluster.http_tls_service_url
}

output "pulsar_cluster_name" {
  value = data.streamnative_pulsar_cluster.serverless-cluster.name
}

output "pulsar_instance_audience" {
  value = data.streamnative_pulsar_instance.serverless-instance.oauth2_audience
}

output "app_service_account_principal" {
  value = "${data.streamnative_service_account.app-sa.name}@${data.streamnative_service_account.app-sa.organization}.auth.streamnative.cloud"
}

output "service_urls" {
  value = {
    pulsar_web_service_url = data.streamnative_pulsar_cluster.serverless-cluster.http_tls_service_url
    pulsar_broker_service_url = data.streamnative_pulsar_cluster.serverless-cluster.pulsar_tls_service_url
    kafka_bootstrap_url = data.streamnative_pulsar_cluster.serverless-cluster.kafka_service_url
  }
}
```

## 3. Create Terraform Configuration Files to provision Pulsar resources

Go back to the root folder `terraform-getting-started`.

```bash theme={null}
cd ..
```

### 3.1 Create Variables File

You can copy the `variables.tf` file from the `streamnative_cloud` module folder.

```bash" { theme={null}
cp streamnative_cloud/variables.tf .
```

### 3.2 Create Terraform Configuration File

Create a `main.tf` file in the root folder and add the following code snippet to create the resources.

```hcl theme={null}
module "streamnative_cloud" {
  source = "./streamnative_cloud"
  org_id = var.org_id
  instance_name = var.instance_name
  cluster_name = var.cluster_name
  app_name = var.app_name
}

terraform {
  required_providers {
    pulsar = {
      source = "streamnative/pulsar"
    }
  }
}

provider "pulsar" {
  web_service_url = module.streamnative_cloud.pulsar_web_service_url
  key_file_path = "./tf-runner.json"
  audience = module.streamnative_cloud.pulsar_instance_audience
}

resource "pulsar_tenant" "app_tenant" {
  depends_on = [module.streamnative_cloud.depends_on]
  tenant = "${var.app_name}-tenant"
  allowed_clusters = [
    module.streamnative_cloud.pulsar_cluster_name,
  ]
}

resource "pulsar_namespace" "app_namespace" {
  depends_on = [pulsar_tenant.app_tenant, module.streamnative_cloud.depends_on]
  tenant    = pulsar_tenant.app_tenant.tenant
  namespace = "${var.app_name}-ns"

  namespace_config {
    replication_clusters = [
      module.streamnative_cloud.pulsar_cluster_name
    ]

  }
  permission_grant {
    role = module.streamnative_cloud.app_service_account_principal
    actions = ["produce", "consume"]
  }
}

resource "pulsar_topic" "app_topic" {
  depends_on = [pulsar_namespace.app_namespace, pulsar_tenant.app_tenant]
  tenant     = pulsar_tenant.app_tenant.tenant
  namespace  = pulsar_namespace.app_namespace.namespace
  topic_type = "persistent"
  topic_name = "${var.app_name}-topic"
  partitions =  4
}

output "apikey" {
  value = module.streamnative_cloud.apikey_token
}

output "pulsar_oauth2_audience" {
  value = module.streamnative_cloud.pulsar_instance_audience
}

output "service_urls" {
  value = module.streamnative_cloud.service_urls
}

output "pulsar_cluster_name" {
  value = module.streamnative_cloud.pulsar_cluster_name
}

output "pulsarctl_command" {
  value = "pulsarctl context set -s ${module.streamnative_cloud.pulsar_web_service_url} --token ${module.streamnative_cloud.apikey_token} ${module.streamnative_cloud.pulsar_cluster_name} && pulsarctl topics get ${var.app_name}-tenant/${var.app_name}-ns/${var.app_name}-topic"
}
```

## 4. Run Terraform Commands

Before running the Terraform commands, you need to expose the following variables:

```bash theme={null}
export TF_VAR_org_id=<your-org-id>
export TF_VAR_instance_name=<your-instance-name>
export TF_VAR_cluster_name=<your-cluster-name>
export TF_VAR_app_name=<your-app-name>
```

Please replace the above placeholders with your actual values:

* `<your-org-id>`: Your StreamNative Cloud organization ID
* `<your-instance-name>`: A unique name for your Pulsar instance (e.g., "sl-instance")
* `<your-cluster-name>`: A unique name for your Pulsar cluster (e.g., "sl-clu")
* `<your-app-name>`: A unique name for your application (e.g., "sl-app")

An example of exposing the variables is as follows:

```bash theme={null}
export TF_VAR_org_id=<your-org-id>
export TF_VAR_instance_name=sl-instance
export TF_VAR_cluster_name=sl-clu
export TF_VAR_app_name=sl-app
```

After exposing the variables, you can run the following Terraform commands to provision the resources.

First, initialize the Terraform working directory.

```bash theme={null}
terraform init
```

Secondly, validate the Terraform configuration files.

```bash theme={null}
terraform validate
```

Since we use two providers in this example (the **StreamNative Provider** and the **Pulsar Provider**), we need to provision the resources in two steps. The Pulsar Provider resources depend on the StreamNative Provider resources being created first.

### 4.1 Provision the Cloud Resources

Run a targeted plan to see the changes and preview the resources that will be created.

```bash theme={null}
terraform plan --target=module.streamnative_cloud.streamnative_apikey.app-apikey
```

After that, run a targeted apply to create the resources.

```bash theme={null}
terraform apply --target=module.streamnative_cloud.streamnative_apikey.app-apikey
```

### 4.2 Provision all the Resources

Run `terraform plan` to see the changes and preview the resources that will be created.

```bash theme={null}
terraform plan
```

After that, run `terraform apply` to create the resources.

```bash theme={null}
terraform apply
```

You should see a similar output as follows:

```bash theme={null}
apikey = "<...>"
pulsarctl_command = "pulsarctl context set -s <pulsar-web-service-url> --token <apikey> sl-clu && pulsarctl topics get sl-app-tenant/sl-app-ns/sl-app-topic"
service_urls = {
  "kafka_bootstrap_url" = "..."
  "pulsar_broker_service_url" = "..."
  "pulsar_web_service_url" = "..."
}
```

## 5. Verify the Resources

Use the [pulsarctl](/tools/cli/pulsarctl/pulsarctl-overview) command to verify all the resources created. Make sure you have installed pulsarctl before running the command.

Copy the `pulsarctl_command` output and run it in your terminal.

```bash theme={null}
pulsarctl context set -s <pulsar-web-service-url> --token <apikey> sl-clu && pulsarctl topics get sl-app-tenant/sl-app-ns/sl-app-topic
```

This command will set the context and get the topic details. It will verify the following resources are created:

* A Pulsar cluster named `sl-clu`
* A Pulsar tenant named `sl-app-tenant`
* A Pulsar namespace named `sl-app-ns`
* A Pulsar topic named `sl-app-topic`
* `produce` and `consume` permissions are granted to `sl-app-sa` on the namespace `sl-app-tenant/sl-app-ns`

You should see the output as follows:

```bash theme={null}
{
  "partitions": 4
}
```


# Create Universal Links in StreamNative Cloud
Source: https://docs.streamnative.io/cloud/universal-linking/unilink-create



StreamNative Cloud provides an intuitive and convenient management interface that allows you to easily create Universal
Links without writing any configuration file code. This document will guide you through the process of creating a
Universal Link in StreamNative Cloud.

Universal Linking offers Data Migration and Schema Migration. Data Migration transfers message data from specified Kafka
topics to the Ursa cluster, while Schema Migration transfers schemas to the KSN Schema Registry. These tools enable
seamless cluster migration.

## Prerequisites

* You have created a Ursa cluster in your organization. Ensure no data or schema is
  produced in the Ursa cluster to avoid potential conflicts.

## Create Data Migration

In the UI, you will create a UniLink Data migration job within your organization.

Ensure no data is produced in replicated topics in the destination cluster during data migration to prevent conflicts.

1. Navigate to the [**Organization Dashboard**](/cloud/get-started/cloud-console#organization-dashboard).
2. Click **Unilink** in the left navigation pane to access the **UniLink** page.
3. Click the **Create** button at the top right corner and select **Data Migration** to initiate the job creation
   process.
4. Enter the Job Name for this UniLink instance. Then click **Source & Destination Details** to proceed.

<img alt="UniLink Job Name" />

### Source & Destination Cluster Configuration

1. On the **Source & Destination Details** page, enter the **Source Cluster** and **Destination Cluster** information.
   Provide the URL for the source cluster and select the destination cluster from the dropdown list.
2. If your source cluster has TLS enabled, toggle the **Cluster with TLS enabled** switch.

<img alt="UniLink Cluster Configuration" />

Create a secret for the source cluster. Click **Create Secret** to generate a secret for the source cluster. The
UniLink job supports SASL authentication for the source cluster. Enter the **username** and **password** for the
source cluster, then click **Create**. You can also select an existing secret from the dropdown list.

<img alt="UniLink Create Secrets" />

Click **Validate & Deploy**. This checks the connection to the source cluster. If successful, click **Continue** for
**Topic Configuration**. If the connection fails, check the source cluster URL and secret.

### Topic Configuration

1. On the **Topic Configuration** page, select the topics you want to migrate from the source cluster to the destination
   cluster. Choose **All topics replications** to replicate all topics or **Selected topics** to replicate specific
   topics.

   If you select **Selected topics**, enter the topic name in the **Include topics** input box. It supports multiple
   regex patterns separated by commas, e.g., `topic1,topic2,topic3-.*`.

2. Enter the **Destination prefix** for the topics in the destination cluster. The prefix will be added to the topic
   name in the destination cluster. For example, if the prefix is `public/test/`, `topic1` will be migrated to
   `public/test/topic1`. This helps migrate topics to a specific namespace. The prefix must be in the format of
   tenant/namespace/ or any string ending with an underscore.

3. UniLink will periodically check the source cluster for new topics and automatically migrate them to the destination
   cluster. Set the **Topic refresh interval** to specify the interval in seconds for checking new topics.

4. Optionally, enter **Excluded topics** to exclude certain topics from migration. It supports multiple regex patterns
   separated by commas, e.g., `topic1,topic2,topic3-.*`.

5. Click **Connect & Deploy**. This builds a topic mapping between the source and destination cluster for review. Click
   **Continue** to proceed to the next step.

<img alt="UniLink Topic Configuration" />

### Consumer Groups Configuration

UniLink will periodically check the source cluster for new consumer groups and automatically migrate them to the
destination cluster. Configure the consumer groups migration in this step.

1. Enable **Enable consumer groups replication** to migrate consumer groups.

2. Choose **All consumer groups** to migrate all consumer groups or **Selected consumer groups** to migrate specific
   ones.

   If you select **Selected consumer groups**, enter the consumer group name in the **Include consumer groups** input
   box. It supports multiple regex patterns separated by commas, e.g., `group1,group2,group3-.*`.

3. Set the **Consumer group refresh interval** to specify the interval in seconds for checking new consumer groups.

4. Optionally, enter **Excluded consumer groups** to exclude certain groups from migration. It supports multiple regex
   patterns separated by commas, e.g., `group1,group2,group3-.*`.

5. Click **Deploy**. This builds a consumer group mapping between the source and destination cluster for review. Click \*
   \*Continue\*\* to deploy the UniLink job.

<img alt="UniLink Consumer Groups Configuration" />

The UniLink job has been successfully created.

## Create Schema Migration

Using UniLink Schema Migration, you can migrate schemas from the source cluster to the destination cluster. UniLink
Schema Migration periodically checks the source cluster for new schemas and automatically migrates them to the
destination cluster.

During schema migration, UniLink will set the destination cluster schema registry to import mode, preventing it from
accepting new schemas from your client. After deleting the UniLink Schema Migration job, the registry will revert to
read-write mode to allow new schema submissions.

1. Navigate to the [**Organization Dashboard**](/cloud/get-started/cloud-console#organization-dashboard).
2. Click **Unilink** in the left navigation pane to access the **UniLink** page.
3. Click the **Create** button at the top right corner and select **Schema Migration** to start the job creation
   process.
4. Enter the Job Name for this UniLink Schema instance. Then click **Source & destination Details** to proceed.

### Source & Destination Cluster Configuration

1. On the **Source & Destination Details** page, enter the **Source Cluster** and **Destination Cluster** information.
   Provide the URL for the source cluster schema registry and select the destination cluster from the dropdown list.
2. Create a secret for the source cluster. Click **Create Secret** to generate a secret for the source cluster. Enter
   the **API Key** and **API Secret** for the source cluster, then click **Create**. You can also select an existing
   secret from the dropdown list.

<img alt="UniLink Schema Cluster" />

Click **Subjects Configuration** to continue.

### Subjects Configuration

1. On the **Subjects Configuration** page, select the subjects you want to migrate from the source cluster to the
   destination cluster. Choose **All subjects replication** to migrate all subjects or **Selected subjects** to migrate
   specific ones.

   If you select **Selected subjects**, enter the subject name in the **Include subjects** input box. It supports
   multiple regex patterns separated by commas, e.g., `subject1,subject2,subject3-.*`.

2. Enter the **Destination prefix** for the subjects in the destination cluster. The prefix will be added to the subject
   name in the destination cluster. For example, if the prefix is `public/test/`, `subject1` will be migrated to
   `public/test/subject1`. This helps migrate subjects to a specific namespace. The prefix must be in the format of
   tenant/namespace/ or any string ending with an underscore.

   Ensure this subject mapping configuration is consistent with the topic mapping configuration in the Data Migration
   job.

<img alt="UniLink Subjects Configuration" />

Click **Deploy**. This builds a subject mapping between the source and destination cluster for review. Click \*
\*Continue\*\* to deploy the UniLink Schema Replication Job.

The UniLink Schema job has been successfully created.


# Manage Universal Links in StreamNative Cloud
Source: https://docs.streamnative.io/cloud/universal-linking/unilink-manage



You can monitor and manage your Universal Linking jobs in the StreamNative Cloud Console.

## View All Universal Linking Jobs

1. Navigate to the [**Organization Dashboard**](/cloud/get-started/cloud-console#organization-dashboard).
2. Click **Unilink** in the left navigation pane to access the **UniLink** page.
3. View all Universal Linking jobs by selecting either the **Data Migration** or **Schema Migration** tab.

<img alt="UniLink List" />

For data replication jobs, you can see the following information:

* **Job Name**: The name of the Universal Linking job, with a unique random suffix.
* **Source Cluster**: The source cluster, also with a unique random suffix.
* **Destination Cluster**: The destination cluster.
* **Throughput**: The total bytes per second and total messages per second of the replication.
* **Message Lag**: The total message lag of the replication, periodically checked by UniLink.
* **Status**: The current status of the Universal Linking job.

<img alt="UniLink Schema List" />

For schema migration jobs, the following details are available:

* **Job Name**: The name of the Schema Replication job, with a unique random suffix.
* **Source Cluster**: A link to the source cluster schema registry.
* **Destination**: A link to the destination cluster schema registry.
* **Status**: The current status of the Schema Replication job.
  s

## View Details of a Universal Linking Job

1. Click the **Job Name** of the Universal Linking job you wish to view.
2. Review the job details, including all configurations. For Data Replication jobs, monitor the progress of all topics,
   including throughput and message lag.

## Edit a Universal Linking Job

1. Click the three dots on the right side of the job you wish to edit.
2. Select **Edit** to modify the job configurations. This process mirrors the steps for creating a new job.
   * For Data Replication jobs, you can edit the source cluster, source cluster secrets, topic replication, and
     consumer group replication. The job name and destination cluster cannot be changed.
   * For Schema Replication jobs, you can edit the source cluster, destination cluster, and schema replication
     settings. The job name and destination cluster cannot be changed.

## Delete a Universal Linking Job

1. Click the three dots on the right side of the job you wish to delete.
2. Select **Delete** to remove the Universal Linking job.


# Manage Universal Linking in StreamNative Cloud
Source: https://docs.streamnative.io/cloud/universal-linking/unilink-overview



<Note title="Note">
  Universal Linking is currently available at no cost. Subject to changes in the future.
</Note>

<Note title="Availability">
  Here is the current status of StreamNative Universal Linking

  * Public Preview on AWS and Google Cloud Platform

  Exercise caution before using preview functionality to provision production environments. If you encounter issues creating a Cloud Environment , please [submit a ticket](https://support.streamnative.io/hc/en-us/requests/new).
</Note>

Universal Linking provides a powerful and cost-effective solution for replicating data across Kafka and Pulsar clusters, whether self-managed or fully-managed. By leveraging S3 object storage, it simplifies networking and storage, significantly reducing operational complexity.

**Key Features:**

* **Offset Preservation**: Ensures that message offsets remain consistent during data replication, maintaining data integrity.
* **No Cross-Zone Traffic**: Enables efficient data transfer without incurring cross-zone charges, optimizing costs.
* **Schema Migration**: Facilitates seamless migration of data schemas, ensuring compatibility and consistency across platforms.
* **Topic and Consumer Group Migration**: Allows easy migration and preview of topics, subjects, and consumer groups, streamlining the transition process.

Universal Linking is ideal for seamless data replication and interoperability, enhancing disaster recovery, data migration, and global data streaming across multi-cloud and hybrid-cloud environments.

**Limitations:**

* StreamNative Universal Linking is currently available in Public Preview on AWS and GCP cloud providers only. It is not available on Azure.
* StreamNative Universal Linking can replicate data on StreamNative Ursa clusters only. It is not compatible with StreamNative’s Classic engine.
* UniLink currently supports data replication only to StreamNative Ursa clusters deployed in a Bring Your Own Cloud (BYOC) environment.
* As StreamNative Ursa is not yet available for Serverless or Dedicated deployment models, this limitation also applies to the use of Universal Linking.
* UniLink currently does not support replication of ACLs
* UniLink only supports Kafka protocol. It does not support the Pulsar protocol.


# Io activemq sink
Source: https://docs.streamnative.io/connect/connectors/activemq-sink/current/io-activemq-sink

ActiveMQ Connector integrates Apache Pulsar with Apache ActiveMQ.

The ActiveMQ sink connector pulls messages from Pulsar topics and persist messages to ActiveMQ.

# Installation

```
git clone https://github.com/streamnative/pulsar-io-activemq.git
cd pulsar-io-activemq/
mvn clean install -DskipTests
cp target/pulsar-io-activemq-0.0.1.nar $PULSAR_HOME/pulsar-io-activemq-0.0.1.nar
```

# Configuration

The configuration of the ActiveMQ sink connector has the following properties.

## ActiveMQ sink connector configuration

| Name                | Type   | Required | Sensitive | Default            | Description                                                              |
| ------------------- | ------ | -------- | --------- | ------------------ | ------------------------------------------------------------------------ |
| `protocol`          | String | true     | false     | "tcp"              | The ActiveMQ protocol.                                                   |
| `host`              | String | true     | false     | " " (empty string) | The ActiveMQ host.                                                       |
| `port`              | int    | true     | false     | 5672               | The ActiveMQ port.                                                       |
| `username`          | String | false    | true      | " " (empty string) | The username used to authenticate to ActiveMQ.                           |
| `password`          | String | false    | true      | " " (empty string) | The password used to authenticate to ActiveMQ.                           |
| `queueName`         | String | false    | false     | " " (empty string) | The ActiveMQ queue name that messages should be read from or written to. |
| `topicName`         | String | false    | false     | " " (empty string) | The ActiveMQ topic name that messages should be read from or written to. |
| `activeMessageType` | String | false    | false     | 0                  | The ActiveMQ message simple class name.                                  |

## Configure ActiveMQ sink connector

Before using the ActiveMQ sink connector, you need to create a configuration file through one of the following methods.

* JSON

  ```json theme={null}
  {
      "tenant": "public",
      "namespace": "default",
      "name": "activemq-sink",
      "inputs": ["user-op-queue-topic"],
      "archive": "connectors/pulsar-io-activemq-2.5.1.nar",
      "parallelism": 1,
      "configs":
      {
          "protocol": "tcp",
          "host": "localhost",
          "port": "61616",
          "username": "admin",
          "password": "admin",
          "queueName": "user-op-queue-pulsar"
      }
  }
  ```

* YAML

  ```yaml theme={null}
  tenant: "public"
  namespace: "default"
  name: "activemq-sink"
  inputs: 
    - "user-op-queue-topic"
  archive: "connectors/pulsar-io-activemq-2.5.1.nar"
  parallelism: 1

  configs:
      protocol: "tcp"
      host: "localhost"
      port: "61616"
      username: "admin"
      password: "admin"
      queueName: "user-op-queue-pulsar"
  ```

# Usage

1. Prepare ActiveMQ service.

   ```
   docker pull rmohr/activemq
   docker run -p 61616:61616 -p 8161:8161 rmohr/activemq
   ```

2. Put the `pulsar-io-activemq-2.5.1.nar` in the pulsar connectors catalog.

   ```
   cp pulsar-io-activemq-2.5.1.nar $PULSAR_HOME/connectors/pulsar-io-activemq-2.5.1.nar
   ```

3. Start Pulsar in standalone mode.

   ```
   $PULSAR_HOME/bin/pulsar standalone
   ```

4. Run ActiveMQ sink locally.

   ```
   $PULSAR_HOME/bin/pulsar-admin sink localrun --sink-config-file activemq-sink-config.yaml
   ```

5. Send Pulsar messages.

   ```
   $PULSAR_HOME/bin/pulsar-client produce public/default/user-op-queue-topic --messages hello -n 10
   ```

6. Consume ActiveMQ messages.

   Use the test method `receiveMessage` of the class `org.apache.pulsar.ecosystem.io.activemq.ActiveMQDemo`
   to consume ActiveMQ messages.

   ```
   @Test
   private void receiveMessage() throws JMSException, InterruptedException {

       ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory("tcp://localhost:61616");

       @Cleanup
       Connection connection = connectionFactory.createConnection();
       connection.start();

       @Cleanup
       Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);

       Destination destination = session.createQueue("user-op-queue-pulsar");

       @Cleanup
       MessageConsumer consumer = session.createConsumer(destination);

       consumer.setMessageListener(new MessageListener() {
           @Override
           public void onMessage(Message message) {
               if (message instanceof ActiveMQTextMessage) {
                   try {
                       System.out.println("get message ----------------- ");
                       System.out.println("receive: " + ((ActiveMQTextMessage) message).getText());
                   } catch (JMSException e) {
                       e.printStackTrace();
                   }
               }
           }
       });
   }
   ```


# Aerospike sink
Source: https://docs.streamnative.io/connect/connectors/aerospike-sink/current/aerospike-sink

The Aerospike sink connector pulls messages from Pulsar topics to Aerospike clusters

The Aerospike sink connector pulls messages from Pulsar topics to Aerospike clusters.

# Configuration

The configuration of the Aerospike sink connector has the following properties.

## Property

| Name                    | Type   | Required | Default          | Description                                                                                                                                                                   |
| ----------------------- | ------ | -------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `seedHosts`             | String | true     | No default value | The comma-separated list of one or more Aerospike cluster hosts.<br /><br />Each host can be specified as a valid IP address or hostname followed by an optional port number. |
| `keyspace`              | String | true     | No default value | The Aerospike namespace.                                                                                                                                                      |
| `columnName`            | String | true     | No default value | The Aerospike column name.                                                                                                                                                    |
| `userName`              | String | false    | NULL             | The Aerospike username.                                                                                                                                                       |
| `password`              | String | false    | NULL             | The Aerospike password.                                                                                                                                                       |
| `keySet`                | String | false    | NULL             | The Aerospike set name.                                                                                                                                                       |
| `maxConcurrentRequests` | int    | false    | 100              | The maximum number of concurrent Aerospike transactions that a sink can open.                                                                                                 |
| `timeoutMs`             | int    | false    | 100              | This property controls `socketTimeout` and `totalTimeout` for Aerospike transactions.                                                                                         |
| `retries`               | int    | false    | 1                | The maximum number of retries before aborting a write transaction to Aerospike.                                                                                               |


# Amqp 1 0 sink
Source: https://docs.streamnative.io/connect/connectors/amqp-1-0-sink/current/amqp-1-0-sink

support sink/source for AMQP version 1.0.0

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

# AMQP 1.0 sink connector

The AMQP 1.0 sink connector pulls messages from Pulsar topics and persists messages to [AMQP 1.0](https://www.amqp.org/).

<img alt="" />

## Quick start

### 1. Start AMQP 1.0 service

Start a service that supports the AMQP 1.0 protocol, such as [Solace](https://docs.solace.com/index.html).

```bash theme={null}
docker run -d -p 8080:8080 -p:8008:8008 -p:1883:1883 -p:8000:8000 -p:5672:5672 -p:9000:9000 -p:2222:2222 --shm-size=2g --env username_admin_globalaccesslevel=admin --env username_admin_password=admin --name=solace solace/solace-pubsub-standard
```

### 2. Create a connector

The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector,
you need to replace `--sink-type amqp1_0` with `--archive /path/to/pulsar-io-amqp1_0.nar`. You can find the button to download the `nar` package at the beginning of the document.

<Note title="For StreamNative Cloud User">
  If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first.
</Note>

```bash theme={null}
pulsarctl sinks create \
  --sink-type amqp1_0 \
  --name amqp1_0-sink \
  --tenant public \
  --namespace default \
  --inputs "Your topic name" \
  --parallelism 1 \
  --sink-config \
  '{
    "connection": {
      "failover": {
        "useFailover": true
      },
      "uris": [
        {
          "protocol": "amqp",
          "host": "localhost",
          "port": 5672,
          "urlOptions": [
            "transport.tcpKeepAlive=true"
          ]
        }
      ]
    },
    "username": "guest",
    "password": "guest",
    "queue": "user-op-queue-pulsar"
  }'
```

The `--sink-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.
If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference.

<Note title="Note">
  You can also choose to use a variety of other tools to create a connector:

  * [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [RestAPI](https://pulsar.apache.org/sink-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document.
</Note>

### 2. Send messages to the topic

<Note title="Note">
  * If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information.
  * The following sample code uses the **Apache qpid** library.
</Note>

```java theme={null}
    public static void main(String[] args) {
        PulsarClient pulsarClient = PulsarClient.builder().serviceUrl("{{Your Pulsar URL}}").build();
        Producer<ByteBuffer> producer = pulsarClient.newProducer(Schema.BYTEBUFFER)
                .topic("{{The topic name that you specified when you created the connector}}")
                .create();

        JmsConnectionFactory jmsConnectionFactory = new JmsConnectionFactory();
        JMSContext jmsContext = jmsConnectionFactory.createContext();

        for (int i = 0; i < 10; i++) {
            JmsTextMessage textMessage = (JmsTextMessage) jmsContext.createTextMessage("text message - " + i);
            ByteBuf byteBuf = (ByteBuf) textMessage.getFacade().encodeMessage();
            producer.send(byteBuf.nioBuffer());
        }
        System.out.println("finish send messages.");
        jmsContext.close();
        pulsarClient.close();
    }
```

### 3. Consume data from AMQP 1.0 service

```java theme={null}
    public static void main(String[] args) {
        ConnectionFactory connectionFactory = new JmsConnectionFactory("guest", "guest", "amqp://localhost:5672");
        Connection connection = connectionFactory.createConnection();
        connection.start();
        Session session = connection.createSession();
        MessageConsumer consumer = session.createConsumer(new JmsQueue("user-op-queue-pulsar"));
        for (int i = 0; i < 10; i++) {
            JmsTextMessage textMessage = (JmsTextMessage) consumer.receive();
            System.out.println("receive msg content: " + textMessage.getText());
            textMessage.acknowledge();
        }
        consumer.close();
        session.close();
        connection.close();
    }
```

## Configuration Properties

Before using the AMQP 1.0 sink connector, you need to configure it.

You can create a configuration file (JSON or YAML) to set the following properties.

| Name                | Type       | Required                                     | Sensitive | Default             | Description                                                                                                                                   |
| ------------------- | ---------- | -------------------------------------------- | --------- | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `protocol`          | String     | required if connection is not used           | false     | "amqp"              | \[deprecated: use connection instead] The AMQP protocol.                                                                                      |
| `host`              | String     | required if connection is not used           | false     | " " (empty string)  | \[deprecated: use connection instead] The AMQP service host.                                                                                  |
| `port`              | int        | required if connection is not used           | false     | 5672                | \[deprecated: use connection instead] The AMQP service port.                                                                                  |
| `connection`        | Connection | required if protocol, host, port is not used | false     | " "  (empty string) | The connection details.                                                                                                                       |
| `username`          | String     | false                                        | true      | " " (empty string)  | The username used to authenticate to ActiveMQ.                                                                                                |
| `password`          | String     | false                                        | true      | " " (empty string)  | The password used to authenticate to ActiveMQ.                                                                                                |
| `queue`             | String     | false                                        | false     | " " (empty string)  | The queue name that messages should be read from or written to.                                                                               |
| `topic`             | String     | false                                        | false     | " " (empty string)  | The topic name that messages should be read from or written to.                                                                               |
| `activeMessageType` | String     | false                                        | false     | 0                   | The ActiveMQ message simple class name.                                                                                                       |
| `onlyTextMessage`   | boolean    | false                                        | false     | false               | If it is set to `true`, the AMQP message type must be set to `TextMessage`. Pulsar consumers can consume the messages with schema ByteBuffer. |

A `Connection` object can be specified as follows:

| Name       | Type                  | Required | Default            | Description                                                                                                                                                       |
| ---------- | --------------------- | -------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `failover` | Failover              | false    | " " (empty string) | The configuration for a failover connection.                                                                                                                      |
| `uris`     | list of ConnectionUri | true     | " " (empty string) | A list of ConnectionUri objects. When useFailover is set to true 1 or more should be provided. Currently only 1 uri is supported when useFailover is set to false |

A `Failover` object can be specified as follows:

| Name                           | Type           | Required                                         | Default            | Description                                                                                                                                                                                                                                                                                                                                                         |
| ------------------------------ | -------------- | ------------------------------------------------ | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `useFailover`                  | boolean        | true                                             | false              | If it is set to true, the connection will be created from the uris provided under uris, using qpid's failover connection factory.                                                                                                                                                                                                                                   |
| `jmsClientId`                  | String         | required if failoverConfigurationOptions is used | " " (empty string) | Identifying name for the jms Client                                                                                                                                                                                                                                                                                                                                 |
| `failoverConfigurationOptions` | List of String | required if jmsClientId is used                  | " " (empty string) | A list of options (e.g. `<key=value>`). The options wil be joined using an '&', prefixed with a the jmsClientId and added to the end of the failoverUri. see also: [https://qpid.apache.org/releases/qpid-jms-2.2.0/docs/index.html#failover-configuration-options](https://qpid.apache.org/releases/qpid-jms-2.2.0/docs/index.html#failover-configuration-options) |

A `ConnectionUri` object can be specified as follows:

| Name         | Type           | Required | Default            | Description                                                                                                                                 |
| ------------ | -------------- | -------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `protocol`   | String         | true     | " " (empty string) | The AMQP protocol.                                                                                                                          |
| `host`       | String         | true     | " " (empty string) | The AMQP service host.                                                                                                                      |
| `port`       | int            | true     | 0                  | The AMQP service port.                                                                                                                      |
| `urlOptions` | List of String | false    | " " (empty string) | A list of url-options (e.g. `<key=value>`). The url options wil be joined using an '&', prefixed with a '?' and added to the end of the uri |


# Aws eventbridge sink
Source: https://docs.streamnative.io/connect/connectors/aws-eventbridge-sink/current/aws-eventbridge-sink

This connector allows you to make sink connections from Pulsar to AWS EventBridge.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

The [Amazon EventBridge](https://aws.amazon.com/eventbridge/) sink connector pulls data from Pulsar topics and persists
data to Amazon EventBridge.

<img alt="" />

## Quick start

### Prerequisites

The prerequisites for connecting an AWS EventBridge sink connector to external systems include:

1. Create EventBridge and EventBus in AWS.
2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `SecretAccessKey`).
3. Assign permissions to AWS User, and ensure they have the `PutEvents` permissions to the AWS EventBus. For details, see [permissions for event buses](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-bus-perms.html)

```json theme={null}
  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "AllowAccountToPutEvents",
        "Effect": "Allow",
        "Principal": {
          "AWS": "<ACCOUNT_ID>"
        },
        "Action": "events:PutEvents",
        "Resource": "{EventBusArn}"
      }
    ]
}
```

* You can set permissions directly for this user. With this method, when you create a connector, you only need to configure `accessKey` and `secretAccessKey`.
* Or you can use [Security Token Service](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html), this [video](https://www.youtube.com/watch?v=dqF4VJCska4) explains how to use STS on AWS.
  With this method, when you create a connector, in addition to configuring `accessKey` and `secretAccessKey`, you also need to configure `role` and `roleSessionName`.

4. Create a [Rule](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html) in EventBridge.

* The data structure sent to Event Bridge is described \[here]\(## Metadata mapping), and you can create **event pattern** based on this structure.
* Set the target according to your needs. If you're testing this connector, you can set the target to [Cloud Watch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html).

### 1. Create a connector

The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector,
you need to replace `--sink-type aws-eventbridge` with `--archive /path/to/pulsar-io-aws-eventbridge.nar`. You can find the button to download the `nar` package at the beginning of the document.

<Note title="For StreamNative Cloud User">
  If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first.
</Note>

```bash theme={null}
pulsarctl sinks create \
  --sink-type aws-eventbridge \
  --name aws-eventbridge-sink \
  --tenant public \
  --namespace default \
  --inputs "Your topic name" \
  --parallelism 1 \
  --sink-config \
  '{
    "accessKeyId": "Your AWS access key", 
    "secretAccessKey": "Your AWS secret access key",
    "region": "Your event bridge region",
    "eventBusName": "Your eventbus name"
  }'
```

The `--sink-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.
If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference.

<Note title="Note">
  You can also choose to use a variety of other tools to create a connector:

  * [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [RestAPI](https://pulsar.apache.org/sink-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document.
</Note>

### 2. Send messages to the topic

<Note title="Note">
  If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information.
</Note>

```java theme={null}
        PulsarClient client = PulsarClient.builder()
            .serviceUrl("{{Your Pulsar URL}}")
            .build();

        Producer<String> producer = client.newProducer(Schema.STRING)
            .topic("{{Your topic name}}")
            .create();

        String message = "{\"msg\": \"msg-data\"}";
        MessageId msgID = producer.send(message);
        System.out.println("Publish " + message + " and message ID " + msgID);
        
        producer.flush();
        producer.close();
        client.close();
```

### 3. Show data on AWS EventBridge

The connector will send the following format of JSON event to EventBridge.

```json theme={null}
{
  "version": "0",
  "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718",
  "detail-type": "{{Your topic name}}",
  "source": "{{Your connector name}}",
  "account": "111122223333",
  "time": "2017-12-22T18:43:48Z",
  "region": "us-west-1",
  "resources": [
    "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0"
  ],
  "detail": {
    "data": {
      "msg": "msg-data"
    },
    "message_id": "124:191:0"
  }
}
```

## Configuration Properties

Before using the AWS EventBridge sink connector, you need to configure it. This table outlines the properties and the
descriptions.

| Name                    | Type   | Required | Sensitive | Default           | Description                                                                                                                                                                                                                                |
| ----------------------- | ------ | -------- | --------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `accessKeyId`           | String | yes      | true      | "" (empty string) | The AWS EventBridge [access key ID.](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html)                                                                                                                     |
| `secretAccessKey`       | String | yes      | true      | "" (empty string) | The AWS EventBridge [secret access key.](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html)                                                                                                                 |
| `region`                | String | yes      | false     | "" (empty string) | The region where AWS EventBridge service is located. [All AWS region](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/Region.html)                                                                                |
| `eventBusName`          | String | yes      | false     | "" (empty string) | The Event Bus name.                                                                                                                                                                                                                        |
| `role`                  | String | false    | false     | "" (empty string) | The AWS STS [roleArn](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html). Example: arn:aws:iam::598203581484:role/test-role                                                                                          |
| `roleSessionName`       | String | false    | false     | "" (empty string) | The AWS role session name, Name it yourself.                                                                                                                                                                                               |
| `stsEndpoint`           | String | false    | false     | "" (empty string) | The AWS STS endpoint. By default, the default STS endpoint: [https://sts.amazonaws.com](https://sts.amazonaws.com) is used. See [Amazon documentation](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) for more details. |
| `stsRegion`             | String | false    | false     | "" (empty string) | The AWS STS region, By default, the 'region' config or env region is used.                                                                                                                                                                 |
| `eventBusResourceName`  | String | no       | false     | "" (empty string) | The Event Bus ARN (AWS Resource Name). Example: `arn:aws:events:ap-northeast-1:598263551484:event-bus/my_eventbus`                                                                                                                         |
| `metaDataField`         | String | no       | false     | "" (empty string) | The metadata fields added to the event. Multiple fields are separated with commas. Optional values: `schema_version`, `partition`,  `event_time`, `publish_time`, `message_id`, `sequence_id`, `producer_name`, `key`, and `properties`.   |
| `batchPendingQueueSize` | int    | no       | false     | 1000              | Pending queue size. This value must be greater than `batchMaxSize`.                                                                                                                                                                        |
| `batchMaxSize`          | int    | no       | false     | 10                | Maximum number of batch messages. The number must be less than or equal to 10 (AWS EventBridge required).                                                                                                                                  |
| `batchMaxBytesSize`     | long   | no       | false     | 640               | Maximum number of batch bytes payload size. This value cannot be greater than 512KB.                                                                                                                                                       |
| `batchMaxTimeMs`        | long   | no       | false     | 5000              | Batch max wait time: milliseconds.                                                                                                                                                                                                         |
| `maxRetryCount`         | long   | no       | false     | 100               | Maximum number of retries to send events, when put events failed.                                                                                                                                                                          |
| `intervalRetryTimeMs`   | long   | no       | false     | 1000              | The interval time(milliseconds) for each retry, when the put events failed.                                                                                                                                                                |

> For details about this connector's advanced features and configurations, see [Advanced features](#advanced-features).

## Advanced features

### Delivery guarantees

The AWS EventBridge sink connector provides two delivery guarantees: **at-most-once** and **at-least-once**.

<Note title="Note">
  Currently, the **effectively-once** delivery guarantee is not supported, because Amazon EventBridge cannot offer the support of the Sink downstream system.
</Note>

### Data convert

In AWS EventBridge, all events
is [JSON format](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html).

Pulsar supports multiple schema types. When receiving the data from Pulsar, the AWS EventBridge sink connectors
recognize it and convert it to a JSON string according to the following table:

| Pulsar Schema  | Convert to JSON | Note                                                                                                                                       |
| -------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| Primitive      | ✔\*             | Just support primitive type is string and data is JSON format.                                                                             |
| Avro           | ✔               | Take advantage of toolkit conversions                                                                                                      |
| Json           | ✔               | Just send it directly                                                                                                                      |
| Protobuf       | X               | The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion. |
| ProtobufNative | ✔               | Take advantage of toolkit conversions                                                                                                      |

In EventBridge, the user data is in the `detail$data` field.

```json theme={null}
{
  "version": "0",
  "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718",
  "detail": {
    "data": {
      "instance-id": " i-1234567890abcdef0",
      "state": "terminated"
    }
  }
}
```

### Metadata mapping

In EventBridge, a complete event contains
many [system fields](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-events.html#eb-custom-event). These
system fields can help you to configure the rule.

An **Event** containing event data:

```json theme={null}
{
  "version": "0",
  "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718",
  "source-type": "test-aws-event-bridge-sink-connector",
  "detail-type": "topic_name_test_1",
  "source": "aws.ec2",
  "account": "111122223333",
  "time": "2017-12-22T18:43:48Z",
  "region": "us-west-1",
  "resources": [
    "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0"
  ],
  "detail": {
    "data": {
      "instance-id": " i-1234567890abcdef0",
      "state": "terminated"
    }
  }
}
```

This connector maps the following fields:

* sourceType: The default value is `${{Connector Name}}`.
* detailType: The default value is `${{Topic Name}}`.

And, this connector supports setting the metadata of Pulsar to every **Event** (set in the **detail** field).

You can select the desired metadata through the following configuration:

```jsx theme={null}
#
optional: schema_version | partition | event_time | publish_time
#
message_id | sequence_id | producer_name | key | properties
metaDataField = event_time, message_id
```

An **Event** containing metadata :

```json theme={null}
{
  "version": "0",
  "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718",
  "source-type": "test-aws-event-bridge-sink-connector",
  "detail-type": "topic_name_test_1",
  "source": "aws.ec2",
  "account": "111122223333",
  "time": "2017-12-22T18:43:48Z",
  "region": "us-west-1",
  "resources": [
    "arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0"
  ],
  "detail": {
    "data": {
      "instance-id": " i-1234567890abcdef0",
      "state": "terminated"
    },
    "event_time": 789894645625,
    "message_id": "1,1,1"
  }
}
```

### Parallelism

You can configure the parallelism of Sink execution by using the scheduling mechanism of the Function, and multiple
sink instances will be scheduled to run on different worker nodes. Multiple sinks will consume messages together
according to the configured subscription mode.

Since EventBus doesn't need to guarantee sequentiality, the connectors support the `shared` subscription model.

To increase the write throughput, you can configure the following:

```jsx theme={null}
parallelism = 4
```

> When `retainOrdering` is set to `false`, the `Shared` subscription mode is used.

### Batch Put

AWS EventBridge connectors support batch put events, which are mainly controlled by the following three parameters:

* **batchSize**: When the buffered message is larger than batchSize, it will trigger flush (put) events. `0` means no
  trigger.
* **maxBatchBytes**: When the buffered message data size is larger than maxBatchBytes, it will trigger flush pending events.
  This value should be less than 256000 and greater than 0, The default value is 640.
* **batchTimeMs**: When the interval from the last flush exceeds `batchTimeMs`, it will trigger flush pending
  events.  `0` means no trigger.

In addition to these three parameters that control flush
behavior, [in AWS EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevent-size.html), batches
larger than 256KB per write are not allowed. So, when the buffered message is larger than 256KB, it will trigger a
flush.

### Retry Put

In AWS Event Bridge, about Handling failures with PutEvents, It suggests retrying each error
message [until it succeeds](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-putevents.html).

This connector will provide two flow configs for the controller's retry strategy:

```jsx theme={null}
maxRetryCount: 100 // Maximum retry send event count, when event send failed.
intervalRetryTimeMs: 1000 //The interval time(milliseconds) for each retry, when event send failed.
```

## More Links

* [GitHub Repo](https://github.com/streamnative/pulsar-io-aws-eventbridge)
* [Announcing the Amazon EventBridge Sink Connector for Apache Pulsar](https://streamnative.io/blog/announcing-the-amazon-eventbridge-sink-connector-for-apache-pulsar)
* [Amazon EventBridge connector is now integrated with StreamNative Cloud](https://streamnative.io/blog/amazon-eventbridge-connector-is-now-integrated-with-streamnative-cloud)


# Aws lambda sink
Source: https://docs.streamnative.io/connect/connectors/aws-lambda-sink/current/aws-lambda-sink

The AWS Lambda sink connector allows you to send messages from Apache Pulsar to AWS Lambda.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

The [AWS Lambda](https://aws.amazon.com/lambda/) sink connector is
a [Pulsar IO connector](http://pulsar.apache.org/docs/en/next/io-overview/) for pulling messages from Pulsar topics to
AWS Lambda to invoke Lambda functions.

<img alt="" />

## Quick start

### Prerequisites

The prerequisites for connecting an AWS Lambda sink connector to external systems include:

1. Create a AWS Lambda function in AWS: [https://docs.aws.amazon.com/lambda/latest/dg/getting-started.html](https://docs.aws.amazon.com/lambda/latest/dg/getting-started.html)
2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(
   Please record `AccessKey` and `SecretAccessKey`).
3. Assign permissions to AWS User, and ensure they have the following permissions to the AWS Lambda. For details,
   see [permissions for AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/lambda-permissions.html)

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "lambda:InvokeFunction",
        "lambda:GetFunction"
      ],
      "Resource": "*"
    }
  ]
}
```

### 1. Create a connector

The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector,
you need to replace `--sink-type aws-lambda` with `--archive /path/to/pulsar-io-aws-lambda.nar`. You can find the button to download the `nar` package at the beginning of the document.

<Note title="For StreamNative Cloud User">
  If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first.
</Note>

```bash theme={null}
pulsarctl sinks create \
  --sink-type aws-lambda \
  --name aws-lambda-sink \
  --tenant public \
  --namespace default \
  --inputs "Your topic name" \
  --parallelism 1 \
  --sink-config \
  '{
    "awsAccessKey": "Your AWS access key", 
    "awsSecretKey": "Your AWS secret key",
    "awsRegion": "Your AWS region",
    "lambdaFunctionName": "Your AWS function name"
    "payloadFormat": "V2"
  }'
```

The `--sink-config` is the minimum necessary and recommended configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.
If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference.

<Note title="Note">
  You can also choose to use a variety of other tools to create a connector:

  * [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [RestAPI](https://pulsar.apache.org/sink-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document.
</Note>

### 2. Send messages to the topic

<Note title="Note">
  If your connector is created on StreamNative Cloud, you need to authenticate your clients.
  See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for
  more information.
</Note>

```java theme={null}
   PulsarClient client = PulsarClient.builder()
            .serviceUrl("{{Your Pulsar URL}}")
            .build();

   Producer<String> producer = client.newProducer(Schema.STRING)
     .topic("{{Your topic name}}")
     .create();

   String message = "Hello, AWS Lambda";
   MessageId msgID = producer.send(message);
   System.out.println("Publish " + message + " and message ID " + msgID);

   producer.flush();
   producer.close();
   client.close();
```

You can also send the message using the command line:

```sh theme={null}
$ bin/pulsar-client produce pulsar-topic-name --messages "Hello, AWS Lambda"
```

### 3. Inspect messages in AWS Lambda

Once you have sent messages to your Pulsar topic, the AWS Lambda sink connector should automatically forward them to the
specified AWS Lambda function. To verify that your messages have been correctly received by AWS Lambda, you can inspect
the logs in the AWS Management Console.

Here are the steps to inspect messages in AWS Lambda:

1. Log in to your AWS Management Console.
2. Navigate to the AWS Lambda service by clicking on "Services" at the top of the page and then typing "Lambda" into the
   search bar.
3. Once you're in the AWS Lambda service, locate and click on the name of the Lambda function you specified when setting
   up your connector.
4. Once you've opened your function, click on the "Monitoring" tab.
5. In the "Monitoring" tab, click on "View logs in CloudWatch". This will redirect you to the AWS CloudWatch service,
   where you can view the log streams for your function.
6. In CloudWatch, select the most recent log stream to view the most recent logs. If your connector is correctly
   forwarding messages, you should see log entries corresponding to the execution of your function with the messages you
   sent.

Remember, the logs may take a few minutes to appear in CloudWatch due to the nature of distributed systems and potential
network latencies.

<Note title="Note">
  If you do not see your messages in the logs, make sure that your AWS Lambda function is correctly logging incoming
  events. You may need to modify your function to explicitly log the event data it receives.
</Note>

By regularly checking the CloudWatch logs for your AWS Lambda function, you can ensure that your Pulsar AWS Lambda sink
connector is correctly forwarding messages and troubleshoot any issues that may arise.

## Configuration Properties

Before using the AWS Lambda sink connector, you need to configure it. This table outlines the properties and the
descriptions.

| Name                       | Type    | Required | Sensitive | Default                                                  | Description                                                                                                                                                                                                                          |
| -------------------------- | ------- | -------- | --------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `awsEndpoint`              | String  | false    | false     | " " (empty string)                                       | The AWS Lambda endpoint URL. It can be found at [AWS Lambda endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/lambda-service.html).                                                                                |
| `awsRegion`                | String  | true     | false     | " " (empty string)                                       | The supported AWS region. For example, `us-west-1`, `us-west-2`.                                                                                                                                                                     |
| `awsAccessKey`             | String  | false    | true      | " " (empty string)                                       | The AWS access key. See here for how to get it: [Managing Access Keys for IAM Users](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).                                                              |
| `awsSecretKey`             | String  | false    | true      | " " (empty string)                                       | The AWS secret key. See here for how to get it: [Managing Access Keys for IAM Users](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).                                                              |
| `lambdaFunctionName`       | String  | true     | false     | " " (empty string)                                       | The Lambda function that should be invoked by the messages.                                                                                                                                                                          |
| `awsCredentialPluginName`  | String  | false    | false     | " " (empty string)                                       | The fully-qualified class name of the `AwsCredentialProviderPlugin` implementation.                                                                                                                                                  |
| `awsCredentialPluginParam` | String  | false    | true      | " " (empty string)                                       | The JSON parameters to initialize `AwsCredentialsProviderPlugin`.                                                                                                                                                                    |
| `synchronousInvocation`    | Boolean | false    | false     | true                                                     | <br />- `true`: invoke a Lambda function synchronously. <br />- `false`: invoke a Lambda function asynchronously.                                                                                                                    |
| `payloadFormat`            | String  | false    | false     | "V1"                                                     | The format of the payload to be sent to the lambda function. Valid values are "V1" and "V2". "V1" is the default value.                                                                                                              |
| `metadataFields`           | String  | false    | false     | "topic,key,partitionIndex,sequence,properties,eventTime" | The metadata fields to be sent to the lambda function. Valid values are `topic,key,partitionIndex,sequence,properties,eventTime,publishTime`, This configuration only takes effect when using the V2 data format (payloadFormat=V2). |
| `batchMaxSize`             | Integer | false    | false     | 10                                                       | The maximum number of records to send to the lambda function in a single batch. This configuration only takes effect when using the V2 data format (payloadFormat=V2).                                                               |
| `batchMaxBytesSize`        | Integer | false    | false     | 262144                                                   | The maximum size of the payload to send to the lambda function in a single batch. This configuration only takes effect when using the V2 data format (payloadFormat=V2).                                                             |
| `batchMaxTimeMs`           | Integer | false    | false     | 5000                                                     | The maximum wait time for batching in milliseconds. This configuration only takes effect when using the V2 data format (payloadFormat=V2).                                                                                           |

## Advanced features

### Payload Format

The payload refers to the actual data that the AWS Lambda sink connector sends to the AWS Lambda function. The AWS
Lambda sink connector supports two payload formats: `V1` and `V2`. It is strongly recommended for you to utilize
the `V2` payload format.

The `V2` payload format provides a more standardized method for managing message data, with added support for schema
conversion and batching.

#### V1 Payload Format

The `V1` payload format is the default payload format. It incorporates three types of data formats, all of which are
represented as JSON objects.

* **Serialization of the Record Object**: Initially, the sink connector attempts to convert the Record object into a
  JSON object.
* **Conversion from the Message Value**: If the conversion of the `Record` object into a JSON object encounters an
  exception, the connector will attempt to convert the message value itself into a JSON object. The format of this data
  entirely depends on how the user has set the message value. It can take any form specified by the user.
* **The JSON Object Containing Metadata and the Value with String Type**: If the message value is not a valid JSON, the
  connector will construct a JSON object that includes the message metadata and the message value. If a metadata field
  does not exist, it will not be included in the JSON object.

Here is an example of the V1 format payload:

```json theme={null}
{
  sourceRecord: {
    ...
    value: 'aGVsbG8=',
    key: {
      empty: true,
      present: false
    },
    ...
  },
  value: 'aGVsbG8=',
  schema: {
    schemaInfo: {
      name: 'Bytes',
      schema: '',
      type: 'BYTES',
      timestamp: 0,
      properties: {},
      schemaDefinition: ''
    },
    nativeSchema: {
      empty: true,
      present: false
    }
  },
  ...
}
```

#### V2 Payload Format

The payload in the `V2` format consists of an array of JSON objects, each representing a message. Each message includes
metadata fields and a value, with the value being either a JSON object or a primitive JSON value.

Here is an example of the V2 payload format:

```json theme={null}
[
  {
    "topic": "my-topic-1",
    "key": "my-key",
    ...
    "value": {
      "my-field": 123
    }
  },
  {
    "topic": "my-topic-2",
    "key": "my-key",
    ...
    "value": "test-value"
  }
]
```

### Schema Support

The AWS Lambda sink connector supports the following schema types: `Primitive Schema`, `Avro Schema`, and `JSON Schema`.

#### Primitive Schema

For the primitive type, the payload format is as follows:

```JSON theme={null}
[
  {
    "topic": "my-topic-1",
    "key": "my-key",
    ...
    "value": 123
  },
  {
    "topic": "my-topic-2",
    "key": "my-key",
    ...
    "value": "test-value"
  },
  {
    "topic": "my-topic-3",
    "key": "my-key",
    ...
    "value": true
  }
]
```

The value types include: Number, Boolean, and String. Here's a table indicating the conversion type for each Primitive
Schema Type:

| Primitive Schema Type                    | JSON Conversion Type                                 | Example                                                   |
| ---------------------------------------- | ---------------------------------------------------- | --------------------------------------------------------- |
| Boolean                                  | Boolean                                              | true                                                      |
| INT8, INT16, INT32, INT64, FLOAT, DOUBLE | Number                                               | 1234                                                      |
| STRING                                   | String                                               | "Hello"                                                   |
| BYTES                                    | Base64-encoded String                                | "SGVsbG8=" (base64-encoded version of the string "Hello") |
| DATE, TIME, TIMESTAMP                    | ISO 8601 String (yyy-MM-dd'T'HH:mm:ss.SSSXXX)        | '2023-10-30T06:13:48.123+08:00'                           |
| LocalDate                                | ISO 8601 String (yyyy-MM-dd)                         | '2023-10-17'                                              |
| LocalTime                                | ISO 8601 String (HH:mm:ss.SSSSSSSSS)                 | '04:30:33.123456789'                                      |
| LocalDateTime                            | ISO 8601 String (yyyy-MM-dd'T'HH:mm:ss.SSSSSSSSS)    | '2023-10-17T04:30:33.123456789'                           |
| Instant                                  | ISO 8601 String (yyyy-MM-dd'T'HH:mm:ss.SSSSSSSSSXXX) | '2023-10-30T06:13:48.123456789+08:00'                     |

#### Struct Schema (Avro Schema and JSON Schema)

For the struct schema types `JSON` and `AVRO`, the value is converted into a JSON object. The conversion rules outlined
in the `Primitive schema section` are applied to all primitive type fields within this value object. Nested objects are
also supported.

Here is an example:

```JSON theme={null}
[
  {
    "topic": "my-topic",
    "key": "my-key",
    ...
    "value": {
      "message": "hello",
      "time": "2023-10-17T08:22:11.263Z"
    }
  }
]
```

Here are the rules for handling the logical type of the Avro based struct schema (`AVRO` and `JSON`):

| Logical Type                 | JSON Conversion Type                             | Example                            |
| ---------------------------- | ------------------------------------------------ | ---------------------------------- |
| `time-millis`, `time-micros` | ISO 8601 String (HH:mm:ss.SSS)                   | '13:48:41.123'                     |
| `timestamp-millis`           | ISO 8601 String (yyy-MM-dd'T'HH:mm:ss.SSSXXX)    | '2023-10-30T06:13:48.123+08:00'    |
| `timestamp-micros`           | ISO 8601 String (yyy-MM-dd'T'HH:mm:ss.SSSSSSXXX) | '2023-10-30T06:13:48.123456+08:00' |
| `local-timestamp-millis`     | ISO 8601 String (yyyy-MM-dd'T'HH:mm:ss.SSS)      | '2023-10-29T22:13:48.123'          |
| `local-timestamp-micros`     | ISO 8601 String (yyyy-MM-dd'T'HH:mm:ss.SSSSSS)   | '2023-10-29T22:13:48.123456'       |

#### Metadata

You can select the metadata fields through the `metaDataField` configuration. The supported metadata fields include:

* `topic`: The source topic name
* `key`: The string type key
* `partitionIndex`: The partition index of the topic
* `sequence`: The sequence ID of the message
* `properties`: The String to String map
* `eventTime`: The event time of the message in the [ISO 8601 format](https://www.w3.org/TR/NOTE-datetime)
* `messageId`: The string representation of a message ID. e.g., '1:1:-1:-1'

#### Batch Support

The AWS Lambda sink connector supports combining multiple messages into a single Lambda invocation for delivery. Each
batch is a V2 format payload that contains multiple messages. The batching support only works when using the V2 format.

You can use the following configurations to control the batch sink policy:

* `batchMaxSize`: The maximum number of records to send to the Lambda function in a single batch.
* `batchMaxBytesSize`: The maximum size of the payload to send to the Lambda function in a single batch.
* `batchMaxTimeMs`: The maximum wait time for batching in milliseconds.

You can simply set `batchMaxSize` to `1` to disable batching.

Please note that AWS Lambda has a payload quota
limit: [AWS Lambda Payload Quotas](https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html). The byte
size of a batch should not exceed this quota limit.


# Aws s3 sink
Source: https://docs.streamnative.io/connect/connectors/aws-s3-sink/current/aws-s3-sink

Cloud Storage Connector integrates Apache Pulsar with cloud storage.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

The [AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) sink connector pulls data from Pulsar topics and persists data to AWS S3 buckets.

<img alt="" />

## Quick start

### Prerequisites

The prerequisites for connecting an AWS S3 sink connector to external systems include:

1. Create S3 buckets in AWS.
2. Create the [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and create `AccessKey`(Please record `AccessKey` and `SecretAccessKey`).
3. Assign permissions to AWS User, and ensure they have the following permissions to the AWS S3.

```json theme={null}
{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "VisualEditor0",
			"Effect": "Allow",
			"Action": [
				"s3:PutObject",
				"s3:AbortMultipartUpload"
			],
			"Resource": "{Your bucket arn}/*"
		}
	]
}
```

### 1. Create a connector

The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector,
you need to replace `--sink-type cloud-storage-s3` with `--archive /path/to/pulsar-io-cloud-storage.nar`. You can find the button to download the `nar` package at the beginning of the document.

<Note title="For StreamNative Cloud User">
  If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first.
</Note>

```bash theme={null}
pulsarctl sinks create \
  --sink-type cloud-storage-s3 \
  --name aws-s3-sink \
  --tenant public \
  --namespace default \
  --inputs "Your topic name" \
  --parallelism 1 \
  --sink-config \
  '{
    "accessKeyId": "Your AWS access key", 
    "secretAccessKey": "Your AWS secret access key",
    "provider": "s3v2",
    "bucket": "Your bucket name",
    "region": "Your AWS S3 region",
    "formatType": "json",
    "partitionerType": "PARTITION",
    "s3StorageClass": "STANDARD"
  }'
```

The `--sink-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.
If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference.

<Note title="Note">
  You can also choose to use a variety of other tools to create a connector:

  * [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [RestAPI](https://pulsar.apache.org/sink-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document.
</Note>

### 2. Send messages to the topic

<Note title="Note">
  If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information.
</Note>

```java theme={null}
    public static void main(String[] args) throws Exception {
        PulsarClient client = PulsarClient.builder()
                             .serviceUrl("{{Your Pulsar URL}}")
                             .build();


        Producer<String> producer = client.newProducer(Schema.STRING)
                .topic("{{Your topic name}}")
                .create();

        for (int i = 0; i < 10; i++) {
            // JSON string containing a single character
            String message = "{\"test-message\": \"test-value\"}";
            producer.send(message);
        }

        producer.close();
        client.close();
    }
```

### 3. Display data on AWS S3 console

You can see the object at public/default/`{{Your topic name}}`-partition-0/xxxx.json on the AWS S3 console. Download and open it, the content is:

```text theme={null}
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
```

## Configuration Properties

Before using the AWS S3 sink connector, you need to configure it. This table outlines the properties and the descriptions.

| Name                            | Type    | Required | Sensitive | Default      | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| ------------------------------- | ------- | -------- | --------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `provider`                      | String  | True     | false     | null         | The AWS S3 client type, such as `aws-s3`,`s3v2`(`s3v2` uses the AWS client but not the JCloud client).                                                                                                                                                                                                                                                                                                                                                                                             |
| `accessKeyId`                   | String  | True     | true      | null         | The AWS access key ID. It requires permission to write objects.                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `secretAccessKey`               | String  | True     | true      | null         | The AWS secret access key.                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `bucket`                        | String  | True     | false     | null         | The AWS S3 bucket.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `formatType`                    | String  | True     | false     | "json"       | The data format type. Available options are `json`, `avro`, `bytes`, or `parquet`. By default, it is set to `json`.                                                                                                                                                                                                                                                                                                                                                                                |
| `partitionerType`               | String  | False    | false     | null         | The partitioning type. It can be configured by topic `PARTITION` or `TIME`. By default, the partition type is configured by topic partitions.                                                                                                                                                                                                                                                                                                                                                      |
| `region`                        | String  | False    | false     | null         | The AWS S3 region. Either the endpoint or region must be set.                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `endpoint`                      | String  | False    | false     | null         | The AWS S3 endpoint. Either the endpoint or region must be set.                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `s3StorageClass`                | String  | False    | false     | "STANDARD"   | The S3 storage class to use when writing objects. Only applies when `provider` is `s3v2`. The value is passed directly to the S3 API and must be a valid [S3 storage class](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html#AmazonS3-PutObject-request-header-StorageClass) string (e.g. `STANDARD`, `STANDARD_IA`, `ONEZONE_IA`, `INTELLIGENT_TIERING`, `GLACIER`, `GLACIER_IR`, `DEEP_ARCHIVE`, `REDUCED_REDUNDANCY`).                                                        |
| `role`                          | String  | False    | false     | null         | The AWS role.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `roleSessionName`               | String  | False    | false     | null         | The AWS role session name.                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `timePartitionPattern`          | String  | False    | false     | "yyyy-MM-dd" | The format pattern of the time-based partitioning. For details, refer to the Java date and time format.                                                                                                                                                                                                                                                                                                                                                                                            |
| `timePartitionDuration`         | String  | False    | false     | "86400000"   | The time interval for time-based partitioning. Support formatted interval string, such as `30d`, `24h`, `30m`, `10s`, and also support number in milliseconds precision, such as `86400000` refers to `24h` or `1d`.                                                                                                                                                                                                                                                                               |
| `pathPrefix`                    | String  | False    | false     | false        | If it is set, the output files are stored in a folder under the given bucket path. The `pathPrefix` must be in the format of `xx/xxx/`.                                                                                                                                                                                                                                                                                                                                                            |
| `partitionerWithTopicName`      | Boolean | False    | false     | true         | Indicates whether to include the topic name in the file path. Default is true. If not included, the path like: `pathPrefix/24.45.0.json`                                                                                                                                                                                                                                                                                                                                                           |
| `partitionerUseIndexAsOffset`   | Boolean | False    | false     | false        | Whether to use the Pulsar's message index as offset or the record sequence. It's recommended if the incoming messages may be batched. The brokers may or not expose the index metadata and, if it's not present on the record, the sequence will be used. See [PIP-70](https://github.com/apache/pulsar/wiki/PIP-70%3A-Introduce-lightweight-broker-entry-metadata) for more details.                                                                                                              |
| `withTopicPartitionNumber`      | Boolean | False    | false     | true         | When it is set to `true`, include the topic partition number to the object path.                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `sliceTopicPartitionPath`       | Boolean | False    | false     | false        | When it is set to `true`, split the partitioned topic name into separate folders in the bucket path.                                                                                                                                                                                                                                                                                                                                                                                               |
| `batchSize`                     | int     | False    | false     | 10           | The number of records submitted in batch.                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `batchTimeMs`                   | long    | False    | false     | 1000         | The interval for batch submission.                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `maxBatchBytes`                 | long    | False    | false     | 10000000     | The maximum number of bytes in a batch.                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `batchModel`                    | Enum    | False    | false     | BLEND        | Determines how records are batched. Options: `BLEND`, `PARTITIONED`. The BLEND which combines all topic records into a single batch, optimizing for throughput, and PARTITIONED which batches records separately for each topic, maintaining topic-level separation. Note: When set to PARTITIONED, the connector will cache data up to the size of the number of subscribed topics multiplied by maxBatchBytes. This means you need to anticipate the connector's memory requirements in advance. |
| `skipFailedMessages`            | Boolean | False    | false     | false        | Configure whether to skip a message which it fails to be processed. If it is set to `true`, the connector will skip the failed messages by `ack` it. Otherwise, the connector will `fail` the message.                                                                                                                                                                                                                                                                                             |
| `withMetadata`                  | Boolean | False    | false     | false        | Save message attributes to metadata.                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `useHumanReadableMessageId`     | Boolean | False    | false     | false        | Use a human-readable format string for messageId in message metadata. The messageId is in a format like `ledgerId:entryId:partitionIndex:batchIndex`. Otherwise, the messageId is a Hex-encoded string.                                                                                                                                                                                                                                                                                            |
| `useHumanReadableSchemaVersion` | Boolean | False    | false     | false        | Use a human-readable format string for the schema version in the message metadata. If it is set to `true`, the schema version is in plain string format. Otherwise, the schema version is in hex-encoded string format.                                                                                                                                                                                                                                                                            |
| `includeTopicToMetadata`        | Boolean | False    | false     | false        | Include the topic name to the metadata.                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `includePublishTimeToMetadata`  | Boolean | False    | false     | false        | Include the message publish time to the metadata as a timestamp.                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `includeMessageKeyToMetadata`   | Boolean | False    | false     | false        | Include the message key to the metadata as a string.                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `avroCodec`                     | String  | False    | false     | snappy       | Compression codec used when formatType=`avro`. Available compression types are: none (no compression), deflate, bzip2, xz, zstandard, snappy.                                                                                                                                                                                                                                                                                                                                                      |
| `parquetCodec`                  | String  | False    | false     | gzip         | Compression codec used when formatType=`parquet`. Available compression types are: none (no compression), snappy, gzip, lzo, brotli, lz4, zstd.                                                                                                                                                                                                                                                                                                                                                    |
| `jsonAllowNaN`                  | Boolean | False    | false     | false        | Recognize 'NaN', 'INF', '-INF' as legal floating number values when formatType=`json`. Since JSON specification does not allow such values this is a non-standard feature and disabled by default.                                                                                                                                                                                                                                                                                                 |
| `bytesFormatTypeSeparator`      | String  | False    | false     | "0x10"       | It is inserted between records for the `formatType` of bytes. By default, it is set to '0x10'. An input record that contains the line separator looks like multiple records in the output object.                                                                                                                                                                                                                                                                                                  |

## Advanced features

### Data format types

AWS S3 Sink Connector provides multiple output format options, including JSON, Avro, Bytes, or Parquet. The default format is JSON.
With current implementation, there are some limitations for different formats:

This table lists the Pulsar Schema types supported by the writers.

| Pulsar Schema  | Writer: Avro | Writer: JSON | Writer: Parquet | Writer: Bytes |
| -------------- | ------------ | ------------ | --------------- | ------------- |
| Primitive      | ✗            | ✔ \*         | ✗               | ✔             |
| Avro           | ✔            | ✔            | ✔               | ✔             |
| Json           | ✔            | ✔            | ✔               | ✔             |
| Protobuf \*\*  | ✔            | ✔            | ✔               | ✔             |
| ProtobufNative | ✔ \*\*\*     | ✗            | ✔               | ✔             |

> \*: The JSON writer will try to convert the data with a `String` or `Bytes` schema to JSON-format data if convertable.
>
> \*\*: The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion.
>
> \*\*\*: The ProtobufNative record holds the Protobuf descriptor and the message. When writing to Avro format, the connector uses [avro-protobuf](https://github.com/apache/avro/tree/master/lang/java/protobuf) to do the conversion.

This table lists the support of `withMetadata` configurations for different writer formats:

| Writer Format | `withMetadata` |
| ------------- | -------------- |
| Avro          | ✔              |
| JSON          | ✔              |
| Parquet       | ✔ \*           |
| Bytes         | ✗              |

> \*: When using `Parquet` with `PROTOBUF_NATIVE` format, the connector will write the messages with `DynamicMessage` format. When `withMetadata` is set to `true`, the connector will add `__message_metadata__` to the messages with `PulsarIOCSCProtobufMessageMetadata` format.
>
> For example, if a message `User` has the following schema:
>
> ```protobuf theme={null}
> syntax = "proto3";
> message User {
>  string name = 1;
>  int32 age = 2;
> }
> ```
>
> When `withMetadata` is set to `true`, the connector will write the message `DynamicMessage` with the following schema:
>
> ```protobuf theme={null}
> syntax = "proto3";
> message PulsarIOCSCProtobufMessageMetadata {
>  map<string, string> properties = 1;
>  string schema_version = 2;
>  string message_id = 3;
> }
> message User {
>  string name = 1;
>  int32 age = 2;
>  PulsarIOCSCProtobufMessageMetadata __message_metadata__ = 3;
> }
> ```

### Dead-letter topics

To use a dead-letter topic, you need to set `skipFailedMessages` to `false`, and set `--max-redeliver-count` and `--dead-letter-topic` when submit the connector with the `pulsar-admin` CLI tool. For more info about dead-letter topics, see the [Pulsar documentation](https://pulsar.apache.org/docs/en/concepts-messaging/#dead-letter-topic).
If a message fails to be sent to the AWS S3 and there is a dead-letter topic, the connector will send the message to the dead-letter topic.

### Sink flushing only after batchTimeMs elapses

There is a scenario where the sink is only flushing whenever the `batchTimeMs` has elapsed, even though there are many messages waiting to be processed.
The reason for this is that the sink will only acknowledge messages after they are flushed to AWS S3 but the broker stops sending messages when it reaches a certain limit of unacknowledged messages.
If this limit is lower or close to `batchSize`, the sink never receives enough messages to trigger a flush based on the amount of messages.
In this case please ensure the `maxUnackedMessagesPerConsumer` set in the broker configuration is sufficiently larger than the `batchSize` setting of the sink.

### Partitioner Type

There are two types of partitioner:

* **PARTITION**: This is the default partitioning method based on Pulsar partitions. In other words, data is
  partitioned according to the pre-existing partitions in Pulsar topics. For instance, a message for the
  topic `public/default/my-topic-partition-0` would be directed to the
  file `public/default/my-topic-partition-0/xxx.json`, where `xxx` signifies the earliest messageId(Format: `ledgerId.entryId.batchIndex`)/offset(Enable config: `partitionerUseIndexAsOffset`) in this file.

* **TIME**: Data is partitioned according to the time it was flushed. Using the previous message as an
  example, if it was received on 2023-12-20, it would be directed
  to `public/default/my-topic-partition-0/2023-12-20/xxx.json`, where `xxx` also denotes the earliest messageId(Format: `ledgerId.entryId.batchIndex`)/offset(Enable config: `partitionerUseIndexAsOffset`) in this file.


# Azure blob storage sink
Source: https://docs.streamnative.io/connect/connectors/azure-blob-storage-sink/current/azure-blob-storage-sink

Cloud Storage Connector integrates Apache Pulsar with cloud storage.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

The [Azure Blob Storage](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-overview) sink connector pulls data from Pulsar topics and persists data to Azure Blob Storage containers.

<img alt="" />

## Quick start

### Prerequisites

The prerequisites for connecting an Azure Blob Storage sink connector to external systems include:

1. Create Blob Storage container in Azure Cloud.
2. Get Storage account `Connection string`.

### 1. Create a connector

The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector,
you need to replace `--sink-type cloud-storage-azure-blob` with `--archive /path/to/pulsar-io-cloud-storage.nar`. You can find the button to download the `nar` package at the beginning of the document.

<Note title="For StreamNative Cloud User">
  If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first.
</Note>

```bash theme={null}
pulsarctl sinks create \
  --sink-type cloud-storage-azure-blob \
  --name azure-blob-sink \
  --tenant public \
  --namespace default \
  --inputs "Your topic name" \
  --parallelism 1 \
  --sink-config \
  '{
    "azureStorageAccountConnectionString": "Your azure blob storage account connection string",
    "provider": "azure-blob-storage",
    "bucket": "Your container name",
    "formatType": "json",
    "partitionerType": "PARTITION"
  }'
```

The `--sink-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.
If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference.

<Note title="Note">
  You can also choose to use a variety of other tools to create a connector:

  * [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [RestAPI](https://pulsar.apache.org/sink-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document.
</Note>

### 2. Send messages to the topic

<Note title="Note">
  If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information.
</Note>

```java theme={null}
    public static void main(String[] args) throws Exception {
        PulsarClient client = PulsarClient.builder()
                             .serviceUrl("{{Your Pulsar URL}}")
                             .build();


        Producer<String> producer = client.newProducer(Schema.STRING)
                .topic("{{Your topic name}}")
                .create();

        for (int i = 0; i < 10; i++) {
            // JSON string containing a single character
            String message = "{\"test-message\": \"test-value\"}";
            producer.send(message);
        }

        producer.close();
        client.close();
    }
```

### 3. Display data on Azure Blob Storage console

You can see the object at public/default/`{{Your topic name}}`-partition-0/xxxx.json on the Azure Blob Storage console. Download and open it, the content is:

```text theme={null}
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
```

## Configuration Properties

Before using the Azure Blob Storage sink connector, you need to configure it. This table outlines the properties and the descriptions.

| Name                                  | Type    | Required | Sensitive | Default      | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| ------------------------------------- | ------- | -------- | --------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `provider`                            | String  | True     | false     | null         | The Cloud Storage type, Azure Blob Storage only supports the `azure-blob-storage` provider.                                                                                                                                                                                                                                                                                                                                                                                                        |
| `bucket`                              | String  | True     | false     | null         | The Azure Blob Storage container name.                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| `formatType`                          | String  | True     | false     | "json"       | The data format type. Available options are `json`, `avro`, `bytes`, or `parquet`. By default, it is set to `json`.                                                                                                                                                                                                                                                                                                                                                                                |
| `partitionerType`                     | String  | False    | false     | null         | The partitioning type. It can be configured by topic `PARTITION` or `TIME`. By default, the partition type is configured by topic partitions.                                                                                                                                                                                                                                                                                                                                                      |
| `azureStorageAccountConnectionString` | String  | False    | true      | ""           | The Azure Blob Storage connection string. Required when authenticating via connection string.                                                                                                                                                                                                                                                                                                                                                                                                      |
| `azureStorageAccountSASToken`         | String  | False    | true      | ""           | The Azure Blob Storage account SAS token. Required when authenticating via SAS token.                                                                                                                                                                                                                                                                                                                                                                                                              |
| `azureStorageAccountName`             | String  | False    | true      | ""           | The Azure Blob Storage account name. Required when authenticating via account name and account key.                                                                                                                                                                                                                                                                                                                                                                                                |
| `azureStorageAccountKey`              | String  | False    | true      | ""           | The Azure Blob Storage account key. Required when authenticating via account name and account key.                                                                                                                                                                                                                                                                                                                                                                                                 |
| `endpoint`                            | String  | False    | false     | null         | The Azure Blob Storage endpoint. Required when authenticating via account name or SAS token.                                                                                                                                                                                                                                                                                                                                                                                                       |
| `timePartitionPattern`                | String  | False    | false     | "yyyy-MM-dd" | The format pattern of the time-based partitioning. For details, refer to the Java date and time format.                                                                                                                                                                                                                                                                                                                                                                                            |
| `timePartitionDuration`               | String  | False    | false     | "86400000"   | The time interval for time-based partitioning. Support formatted interval string, such as `30d`, `24h`, `30m`, `10s`, and also support number in milliseconds precision, such as `86400000` refers to `24h` or `1d`.                                                                                                                                                                                                                                                                               |
| `pathPrefix`                          | String  | False    | false     | false        | If it is set, the output files are stored in a folder under the given bucket path. The `pathPrefix` must be in the format of `xx/xxx/`.                                                                                                                                                                                                                                                                                                                                                            |
| `partitionerWithTopicName`            | Boolean | False    | false     | true         | Indicates whether to include the topic name in the file path. Default is true. If not included, the path like: `pathPrefix/24.45.0.json`                                                                                                                                                                                                                                                                                                                                                           |
| `partitionerUseIndexAsOffset`         | Boolean | False    | false     | false        | Whether to use the Pulsar's message index as offset or the record sequence. It's recommended if the incoming messages may be batched. The brokers may or not expose the index metadata and, if it's not present on the record, the sequence will be used. See [PIP-70](https://github.com/apache/pulsar/wiki/PIP-70%3A-Introduce-lightweight-broker-entry-metadata) for more details.                                                                                                              |
| `withTopicPartitionNumber`            | Boolean | False    | false     | true         | When it is set to `true`, include the topic partition number to the object path.                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `sliceTopicPartitionPath`             | Boolean | False    | false     | false        | When it is set to `true`, split the partitioned topic name into separate folders in the bucket path.                                                                                                                                                                                                                                                                                                                                                                                               |
| `batchSize`                           | int     | False    | false     | 10           | The number of records submitted in batch.                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `batchTimeMs`                         | long    | False    | false     | 1000         | The interval for batch submission.                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `maxBatchBytes`                       | long    | False    | false     | 10000000     | The maximum number of bytes in a batch.                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `batchModel`                          | Enum    | False    | false     | BLEND        | Determines how records are batched. Options: `BLEND`, `PARTITIONED`. The BLEND which combines all topic records into a single batch, optimizing for throughput, and PARTITIONED which batches records separately for each topic, maintaining topic-level separation. Note: When set to PARTITIONED, the connector will cache data up to the size of the number of subscribed topics multiplied by maxBatchBytes. This means you need to anticipate the connector's memory requirements in advance. |
| `skipFailedMessages`                  | Boolean | False    | false     | false        | Configure whether to skip a message which it fails to be processed. If it is set to `true`, the connector will skip the failed messages by `ack` it. Otherwise, the connector will `fail` the message.                                                                                                                                                                                                                                                                                             |
| `withMetadata`                        | Boolean | False    | false     | false        | Save message attributes to metadata.                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `useHumanReadableMessageId`           | Boolean | False    | false     | false        | Use a human-readable format string for messageId in message metadata. The messageId is in a format like `ledgerId:entryId:partitionIndex:batchIndex`. Otherwise, the messageId is a Hex-encoded string.                                                                                                                                                                                                                                                                                            |
| `useHumanReadableSchemaVersion`       | Boolean | False    | false     | false        | Use a human-readable format string for the schema version in the message metadata. If it is set to `true`, the schema version is in plain string format. Otherwise, the schema version is in hex-encoded string format.                                                                                                                                                                                                                                                                            |
| `includeTopicToMetadata`              | Boolean | False    | false     | false        | Include the topic name to the metadata.                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `includePublishTimeToMetadata`        | Boolean | False    | false     | false        | Include the message publish time to the metadata as a timestamp.                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `includeMessageKeyToMetadata`         | Boolean | False    | false     | false        | Include the message key to the metadata as a string.                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `avroCodec`                           | String  | False    | false     | snappy       | Compression codec used when formatType=`avro`. Available compression types are: none (no compression), deflate, bzip2, xz, zstandard, snappy.                                                                                                                                                                                                                                                                                                                                                      |
| `parquetCodec`                        | String  | False    | false     | gzip         | Compression codec used when formatType=`parquet`. Available compression types are: none (no compression), snappy, gzip, lzo, brotli, lz4, zstd.                                                                                                                                                                                                                                                                                                                                                    |
| `jsonAllowNaN`                        | Boolean | False    | false     | false        | Recognize 'NaN', 'INF', '-INF' as legal floating number values when formatType=`json`. Since JSON specification does not allow such values this is a non-standard feature and disabled by default.                                                                                                                                                                                                                                                                                                 |
| `bytesFormatTypeSeparator`            | String  | False    | false     | "0x10"       | It is inserted between records for the `formatType` of bytes. By default, it is set to '0x10'. An input record that contains the line separator looks like multiple records in the output object.                                                                                                                                                                                                                                                                                                  |

There are three methods to authenticate with Azure Blob Storage:

1. `azureStorageAccountConnectionString`: This method involves using the Azure Blob Storage connection string for authentication. It's the simplest method as it only requires the connection string.
2. `azureStorageAccountSASToken`: This method uses a Shared Access Signature (SAS) token for the Azure Blob Storage account. When using this method, you must also set the `endpoint`.
3. `azureStorageAccountName` and `azureStorageAccountKey`: This method uses the Azure Blob Storage account name and account key for authentication. Similar to the SAS token method, you must also set the `endpoint` when using this method.

## Advanced features

### Data format types

Azure Blob Storage Sink Connector provides multiple output format options, including JSON, Avro, Bytes, or Parquet. The default format is JSON.
With current implementation, there are some limitations for different formats:

This table lists the Pulsar Schema types supported by the writers.

| Pulsar Schema  | Writer: Avro | Writer: JSON | Writer: Parquet | Writer: Bytes |
| -------------- | ------------ | ------------ | --------------- | ------------- |
| Primitive      | ✗            | ✔ \*         | ✗               | ✔             |
| Avro           | ✔            | ✔            | ✔               | ✔             |
| Json           | ✔            | ✔            | ✔               | ✔             |
| Protobuf \*\*  | ✔            | ✔            | ✔               | ✔             |
| ProtobufNative | ✔ \*\*\*     | ✗            | ✔               | ✔             |

> \*: The JSON writer will try to convert the data with a `String` or `Bytes` schema to JSON-format data if convertable.
>
> \*\*: The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion.
>
> \*\*\*: The ProtobufNative record holds the Protobuf descriptor and the message. When writing to Avro format, the connector uses [avro-protobuf](https://github.com/apache/avro/tree/master/lang/java/protobuf) to do the conversion.

This table lists the support of `withMetadata` configurations for different writer formats:

| Writer Format | `withMetadata` |
| ------------- | -------------- |
| Avro          | ✔              |
| JSON          | ✔              |
| Parquet       | ✔ \*           |
| Bytes         | ✗              |

> \*: When using `Parquet` with `PROTOBUF_NATIVE` format, the connector will write the messages with `DynamicMessage` format. When `withMetadata` is set to `true`, the connector will add `__message_metadata__` to the messages with `PulsarIOCSCProtobufMessageMetadata` format.
>
> For example, if a message `User` has the following schema:
>
> ```protobuf theme={null}
> syntax = "proto3";
> message User {
>  string name = 1;
>  int32 age = 2;
> }
> ```
>
> When `withMetadata` is set to `true`, the connector will write the message `DynamicMessage` with the following schema:
>
> ```protobuf theme={null}
> syntax = "proto3";
> message PulsarIOCSCProtobufMessageMetadata {
>  map<string, string> properties = 1;
>  string schema_version = 2;
>  string message_id = 3;
> }
> message User {
>  string name = 1;
>  int32 age = 2;
>  PulsarIOCSCProtobufMessageMetadata __message_metadata__ = 3;
> }
> ```

### Dead-letter topics

To use a dead-letter topic, you need to set `skipFailedMessages` to `false`, and set `--max-redeliver-count` and `--dead-letter-topic` when submit the connector with the `pulsar-admin` CLI tool. For more info about dead-letter topics, see the [Pulsar documentation](https://pulsar.apache.org/docs/en/concepts-messaging/#dead-letter-topic).
If a message fails to be sent to the Azure Blob Storage and there is a dead-letter topic, the connector will send the message to the dead-letter topic.

### Sink flushing only after batchTimeMs elapses

There is a scenario where the sink is only flushing whenever the `batchTimeMs` has elapsed, even though there are many messages waiting to be processed.
The reason for this is that the sink will only acknowledge messages after they are flushed to the Azure Blob Storage but the broker stops sending messages when it reaches a certain limit of unacknowledged messages.
If this limit is lower or close to `batchSize`, the sink never receives enough messages to trigger a flush based on the amount of messages.
In this case please ensure the `maxUnackedMessagesPerConsumer` set in the broker configuration is sufficiently larger than the `batchSize` setting of the sink.

### Partitioner Type

There are two types of partitioner:

* **PARTITION**: This is the default partitioning method based on Pulsar partitions. In other words, data is
  partitioned according to the pre-existing partitions in Pulsar topics. For instance, a message for the
  topic `public/default/my-topic-partition-0` would be directed to the
  file `public/default/my-topic-partition-0/xxx.json`, where `xxx` signifies the earliest messageId(Format: `ledgerId.entryId.batchIndex`)/offset(Enable config: `partitionerUseIndexAsOffset`) in this file.

* **TIME**: Data is partitioned according to the time it was flushed. Using the previous message as an
  example, if it was received on 2023-12-20, it would be directed
  to `public/default/my-topic-partition-0/2023-12-20/xxx.json`, where `xxx` also denotes the earliest messageId(Format: `ledgerId.entryId.batchIndex`)/offset(Enable config: `partitionerUseIndexAsOffset`) in this file.


# Canal source
Source: https://docs.streamnative.io/connect/connectors/canal-source/current/canal-source

The Canal source connector pulls messages from MySQL to Pulsar topics.

The Canal source connector pulls messages from MySQL to Pulsar topics.

# Configuration

The configuration of Canal source connector has the following properties.

## Property

| Name             | Required | Sensitive | Default | Description                                                                                                                                                                                                                                                                                                                                                    |
| ---------------- | -------- | --------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `username`       | true     | true      | None    | Canal server account (not MySQL).                                                                                                                                                                                                                                                                                                                              |
| `password`       | true     | true      | None    | Canal server password (not MySQL).                                                                                                                                                                                                                                                                                                                             |
| `destination`    | true     | false     | None    | Source destination that Canal source connector connects to.                                                                                                                                                                                                                                                                                                    |
| `singleHostname` | false    | false     | None    | Canal server address.                                                                                                                                                                                                                                                                                                                                          |
| `singlePort`     | false    | false     | None    | Canal server port.                                                                                                                                                                                                                                                                                                                                             |
| `cluster`        | true     | false     | false   | Whether to enable cluster mode based on Canal server configuration or not.<br /><br /><li>true: **cluster** mode.<br />If set to true, it talks to `zkServers` to figure out the actual database host.<br /><br /></li><li>false: **standalone** mode.<br />If set to false, it connects to the database specified by `singleHostname` and `singlePort`. </li> |
| `zkServers`      | true     | false     | None    | Address and port of the Zookeeper that Canal source connector talks to figure out the actual database host.                                                                                                                                                                                                                                                    |
| `batchSize`      | false    | false     | 1000    | Batch size to fetch from Canal.                                                                                                                                                                                                                                                                                                                                |

## Example

Before using the Canal connector, you can create a configuration file through one of the following methods.

* JSON

  ```json theme={null}
  {
      "zkServers": "127.0.0.1:2181",
      "batchSize": "5120",
      "destination": "example",
      "username": "",
      "password": "",
      "cluster": false,
      "singleHostname": "127.0.0.1",
      "singlePort": "11111",
  }
  ```

* YAML

  You can create a YAML file and copy the [contents](https://github.com/apache/pulsar/blob/master/pulsar-io/canal/src/main/resources/canal-mysql-source-config.yaml) below to your YAML file.

  ```yaml theme={null}
  configs:
      zkServers: "127.0.0.1:2181"
      batchSize: 5120
      destination: "example"
      username: ""
      password: ""
      cluster: false
      singleHostname: "127.0.0.1"
      singlePort: 11111
  ```

# Usage

Here is an example of storing MySQL data using the configuration file as above.

1. Start a MySQL server.

   ```bash theme={null}
   $ docker pull mysql:5.7
   $ docker run -d -it --rm --name pulsar-mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=canal -e MYSQL_USER=mysqluser -e MYSQL_PASSWORD=mysqlpw mysql:5.7
   ```

2. Create a configuration file `mysqld.cnf`.

   ```bash theme={null}
   [mysqld]
   pid-file    = /var/run/mysqld/mysqld.pid
   socket      = /var/run/mysqld/mysqld.sock
   datadir     = /var/lib/mysql
   #log-error  = /var/log/mysql/error.log
   # By default we only accept connections from localhost
   #bind-address   = 127.0.0.1
   # Disabling symbolic-links is recommended to prevent assorted security risks
   symbolic-links=0
   log-bin=mysql-bin
   binlog-format=ROW
   server_id=1
   ```

3. Copy the configuration file `mysqld.cnf` to MySQL server.

   ```bash theme={null}
   $ docker cp mysqld.cnf pulsar-mysql:/etc/mysql/mysql.conf.d/
   ```

4. Restart the MySQL server.

   ```bash theme={null}
   $ docker restart pulsar-mysql
   ```

5. Create a test database in MySQL server.

   ```bash theme={null}
   $ docker exec -it pulsar-mysql /bin/bash
   $ mysql -h 127.0.0.1 -uroot -pcanal -e 'create database test;'
   ```

6. Start a Canal server and connect to MySQL server.

   ```
   $ docker pull canal/canal-server:v1.1.2
   $ docker run -d -it --link pulsar-mysql -e canal.auto.scan=false -e canal.destinations=test -e canal.instance.master.address=pulsar-mysql:3306 -e canal.instance.dbUsername=root -e canal.instance.dbPassword=canal -e canal.instance.connectionCharset=UTF-8 -e canal.instance.tsdb.enable=true -e canal.instance.gtidon=false --name=pulsar-canal-server -p 8000:8000 -p 2222:2222 -p 11111:11111 -p 11112:11112 -m 4096m canal/canal-server:v1.1.2
   ```

7. Start Pulsar standalone.

   ```bash theme={null}
   $ docker pull apachepulsar/pulsar:2.3.0
   $ docker run -d -it --link pulsar-canal-server -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-standalone apachepulsar/pulsar:2.3.0 bin/pulsar standalone
   ```

8. Modify the configuration file `canal-mysql-source-config.yaml`.

   ```yaml theme={null}
   configs:
       zkServers: ""
       batchSize: "5120"
       destination: "test"
       username: ""
       password: ""
       cluster: false
       singleHostname: "pulsar-canal-server"
       singlePort: "11111"
   ```

9. Create a consumer file `pulsar-client.py`.

   ```python theme={null}
   import pulsar

   client = pulsar.Client('pulsar://localhost:6650')
   consumer = client.subscribe('my-topic',
                               subscription_name='my-sub')

   while True:
       msg = consumer.receive()
       print("Received message: '%s'" % msg.data())
       consumer.acknowledge(msg)

   client.close()
   ```

10. Copy the configuration file `canal-mysql-source-config.yaml` and the consumer file  `pulsar-client.py` to Pulsar server.

    ```bash theme={null}
    $ docker cp canal-mysql-source-config.yaml pulsar-standalone:/pulsar/conf/
    $ docker cp pulsar-client.py pulsar-standalone:/pulsar/
    ```

11. Download a Canal connector and start it.

    ```bash theme={null}
    $ docker exec -it pulsar-standalone /bin/bash
    $ wget https://archive.apache.org/dist/pulsar/pulsar-2.3.0/connectors/pulsar-io-canal-2.3.0.nar -P connectors
    $ ./bin/pulsar-admin source localrun \
    --archive ./connectors/pulsar-io-canal-2.3.0.nar \
    --classname org.apache.pulsar.io.canal.CanalStringSource \
    --tenant public \
    --namespace default \
    --name canal \
    --destination-topic-name my-topic \
    --source-config-file /pulsar/conf/canal-mysql-source-config.yaml \
    --parallelism 1
    ```

12. Consume data from MySQL.

    ```bash theme={null}
    $ docker exec -it pulsar-standalone /bin/bash
    $ python pulsar-client.py
    ```

13. Open another window to log in MySQL server.

    ```bash theme={null}
    $ docker exec -it pulsar-mysql /bin/bash
    $ mysql -h 127.0.0.1 -uroot -pcanal
    ```

14. Create a table, and insert, delete, and update data in MySQL server.

    ```bash theme={null}
    mysql> use test;
    mysql> show tables;
    mysql> CREATE TABLE IF NOT EXISTS `test_table`(`test_id` INT UNSIGNED AUTO_INCREMENT,`test_title` VARCHAR(100) NOT NULL,
    `test_author` VARCHAR(40) NOT NULL,
    `test_date` DATE,PRIMARY KEY ( `test_id` ))ENGINE=InnoDB DEFAULT CHARSET=utf8;
    mysql> INSERT INTO test_table (test_title, test_author, test_date) VALUES("a", "b", NOW());
    mysql> UPDATE test_table SET test_title='c' WHERE test_title='a';
    mysql> DELETE FROM test_table WHERE test_title='c';
    ```


# Cassandra sink
Source: https://docs.streamnative.io/connect/connectors/cassandra-sink/current/cassandra-sink

The Cassandra sink connector pulls messages from Pulsar topics to Cassandra clusters

The Cassandra sink connector pulls messages from Pulsar topics to Cassandra clusters.

# Configuration

The configuration of the Cassandra sink connector has the following properties.

## Property

| Name           | Type   | Required | Default            | Description                                                                                                                                                                                                         |
| -------------- | ------ | -------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `roots`        | String | true     | " " (empty string) | A comma-separated list of Cassandra hosts to connect to.                                                                                                                                                            |
| `keyspace`     | String | true     | " " (empty string) | The key space used for writing pulsar messages. <br /><br />**Note: `keyspace` should be created prior to a Cassandra sink.**                                                                                       |
| `keyname`      | String | true     | " " (empty string) | The key name of the Cassandra column family. <br /><br />The column is used for storing Pulsar message keys. <br /><br />If a Pulsar message doesn't have any key associated, the message value is used as the key. |
| `columnFamily` | String | true     | " " (empty string) | The Cassandra column family name.<br /><br />**Note: `columnFamily` should be created prior to a Cassandra sink.**                                                                                                  |
| `columnName`   | String | true     | " " (empty string) | The column name of the Cassandra column family.<br /><br /> The column is used for storing Pulsar message values.                                                                                                   |

## Example

Before using the Cassandra sink connector, you need to create a configuration file through one of the following methods.

* JSON

  ```json theme={null}
  {
      "roots": "localhost:9042",
      "keyspace": "pulsar_test_keyspace",
      "columnFamily": "pulsar_test_table",
      "keyname": "key",
      "columnName": "col"
  }
  ```

* YAML

  ```
  configs:
      roots: "localhost:9042"
      keyspace: "pulsar_test_keyspace"
      columnFamily: "pulsar_test_table"
      keyname: "key"
      columnName: "col"
  ```

# Usage

For more information about **how to connect Pulsar with Cassandra**, see [here](https://pulsar.apache.org/docs/en/next/io-quickstart/#connect-pulsar-to-cassandra).


# Debezium mongodb source
Source: https://docs.streamnative.io/connect/connectors/debezium-mongodb-source/current/debezium-mongodb-source

The Debezium MongoDB source connector pulls messages from MongoDB and persists the messages to Pulsar topics

The Debezium MongoDB source connector pulls messages from MongoDB and persists the messages to Pulsar topics.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

<img alt="" />

## Quick start

### Prerequisites

The prerequisites for connecting a Debezium MongoDB source connector to external systems include:

1. Create a MongoDB service: This connector uses the debezium v3.2, Please refer to this [document](https://debezium.io/releases/3.2/) to see the compatible MongoDB versions.
2. Prepare MongoDB Database: Please refer to this [document](https://debezium.io/documentation/reference/3.2/connectors/mongodb.html#setting-up-mongodb) to complete the prepare steps on MongoDB.
3. Configure topic retention policies: Before running the connector, you must ensure that you have set an infinite retention policy for both the `offset.storage.topic` and `schema.history.internal.pulsar.topic`. Refer to the [Used Topic On Pulsar](#used-topic-on-pulsar) section for more details.

### 1. Prepare MongoDB service

Initialize MongoDB replica set and insert some test data. You can use the following command to start a MongoDB service for the testing purpose.

```sh theme={null}
docker run -d -p 27017:27017 --name mongodb mongo:latest --replSet rs0
```

Shell into the container:

```sh theme={null}
docker exec -it mongodb mongosh
```

Initialize replica sets:

```sh theme={null}
rs.initiate({_id: "rs0", members: [{ _id: 0, host: "localhost:27017" }]})
```

### 2. Create a connector

The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector,
you need to replace `--source-type debezium-mongodb` with `--archive /path/to/pulsar-io-debezium-mongodb.nar`. You can find the button to download the `nar` package at the beginning of the document.

<Note title="For StreamNative Cloud User">
  If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first.
</Note>

```bash theme={null}
pulsarctl sources create \
  --source-type debezium-mongodb \
  --name debezium-mongodb \
  --tenant public \
  --namespace default \
  --parallelism 1 \
  --source-config \
  '{
    "mongodb.connection.string": "rs0/localhost:27017",
    "mongodb.task.id": "1",
    "topic.prefix": "mongodb",
    "connector.class": "io.debezium.connector.mongodb.MongoDbConnector",
    "database.include.list": "inventory"
  }'
```

<Note title="Note">
  1. The `--parallelism` must be set to **1**. Debezium connectors do not support parallel consumption within a single instance. If you need to process tables in parallel, you can deploy multiple connector instances, each configured for different database schemas or tables.
  2. You can set multiple tables for "table.include.list", and the connector will send data from each table to a different topic of pulsar. The topic naming rule is: `{{database.server.name}}.{{table.name}}`. For examples: `public/default/mydbserver.public.io-test`.
</Note>

The `--source-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.

If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference.

<Note title="Note">
  You can also choose to use a variety of other tools to create a connector:

  * [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [RestAPI](https://pulsar.apache.org/sink-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document.
</Note>

### 3. Insert and update a data to the collection

Start the mongosh and run:

```
use inventory;
db.customers.insert([ { _id: NumberLong("1"), first_name: 'Bob', last_name: 'Hopper', email: 'thebob@example.com', unique_id: UUID() }] );
```

### 4. Show data using Pulsar client

<Note title="Note">
  If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information.
</Note>

```sh theme={null}
bin/pulsar-client \
--url "Your Pulsar serviceUrl" \
consume "persistent://public/default/debezium.inventory.customers" -s "test-sub" -n 0 -p Earliest

----- got message -----
key:[eyJpZCI6IjQifQ==], properties:[], content:{"after":"{\"_id\": {\"$numberLong\": \"1\"},\"first_name\": \"Bob\",\"last_name\": \"Hopper\",\"email\": \"thebob@example.com\",\"unique_id\": {\"$binary\": \"xQezJ8i5QTGDG9NXlVFUEw==\",\"$type\": \"04\"}}","patch":null,"filter":null,"updateDescription":null,"source":{"version":"3.2.5.Final","connector":"mongodb","name":"debezium","ts_ms":1701329265000,"snapshot":"false","db":"inventory","sequence":null,"rs":"rs0","collection":"customers","ord":1,"h":null,"tord":null,"stxnid":null,"lsid":null,"txnNumber":null},"op":"c","ts_ms":1701329265295,"transaction":null}
```

## Configuration Properties

The configuration of Debezium Mongodb source connector has the following properties.

| Name                                         | Required | Sensitive | Default | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| -------------------------------------------- | -------- | --------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `mongodb.connection.string`                  | true     | false     | null    | The comma-separated list of hostname and port pairs (in the form 'host' or 'host:port') of the MongoDB servers in the replica set. The list contains a single hostname and a port pair. If mongodb.members.auto.discover is set to false, the host and port pair are prefixed with the replica set name (e.g., rs0/localhost:27017).                                                                                                                                                                                                |
| `mongodb.user`                               | false    | true      | null    | Name of the database user to be used when connecting to MongoDB. This is required only when MongoDB is configured to use authentication.                                                                                                                                                                                                                                                                                                                                                                                            |
| `mongodb.password`                           | false    | true      | null    | Password to be used when connecting to MongoDB. This is required only when MongoDB is configured to use authentication.                                                                                                                                                                                                                                                                                                                                                                                                             |
| `mongodb.task.id`                            | true     | false     | null    | The taskId of the MongoDB connector that attempts to use a separate task for each replica set.                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `topic.prefix`                               | true     | false     | null    | The prefix that is used to name persisted topics.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `connector.class`                            | true     | false     | null    | The Java class for the Debezium MongoDB connector, can only be: `io.debezium.connector.mongodb.MongoDbConnector`.                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `database.include.list`                      | false    | false     | null    | A list of all databases hosted by this server which is monitored by the  connector.<br /><br /> This is optional, and there are other properties for listing databases and tables to include or exclude from monitoring.                                                                                                                                                                                                                                                                                                            |
| `database.exclude.list`                      | false    | false     | null    | A list of all databases hosted by this server which is excluded from being monitored by the  connector.<br /><br /> This is optional, and there are other properties for listing databases and tables to include or exclude from monitoring.                                                                                                                                                                                                                                                                                        |
| `table.include.list`                         | false    | false     | null    | A list of all tables hosted by this server which is monitored by the  connector.<br /><br /> This is optional, and there are other properties for listing tables and tables to include or exclude from monitoring.                                                                                                                                                                                                                                                                                                                  |
| `table.exclude.list`                         | false    | false     | null    | A list of all tables hosted by this server which is excluded from being monitored by the  connector.<br /><br /> This is optional, and there are other properties for listing tables and tables to include or exclude from monitoring.                                                                                                                                                                                                                                                                                              |
| `key.converter`                              | false    | false     | null    | The converter provided by Kafka Connect to convert record key.                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `value.converter`                            | false    | false     | null    | The converter provided by Kafka Connect to convert record value.                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| `schema.history.internal.pulsar.topic`       | false    | false     | null    | The name of the database history topic where the connector writes and recovers DDL statements. <br /><br />**Note: this topic is for internal use only and should not be used by consumers.**                                                                                                                                                                                                                                                                                                                                       |
| `schema.history.internal.pulsar.service.url` | false    | false     | null    | Pulsar cluster service URL for history topic.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `offset.storage.topic`                       | false    | false     | null    | Record the last committed offsets that the connector successfully completes. By default, it's `topicNamespace + "/" + sourceName + "-debezium-offset-topic"`. eg. `persistent://public/default/debezium-mongodb-source-debezium-offset-topic`                                                                                                                                                                                                                                                                                       |
| `json-with-envelope`                         | false    | false     | false   | The`json-with-envelope` config is valid only for the JsonConverter. By default, the value is set to false. When the `json-with-envelope` value is set to false, the consumer uses the schema `Schema.KeyValue(Schema.AUTO_CONSUME(), Schema.AUTO_CONSUME(), KeyValueEncodingType.SEPARATED)`, and the message only consists of the payload. When the `json-with-envelope` value is set to true, the consumer uses the schema `Schema.KeyValue(Schema.BYTES, Schema.BYTES)`, and the message consists of the schema and the payload. |

For more configuration properties, please see [Debezium MongoDB connector configuration properties](https://debezium.io/documentation/reference/3.2/connectors/mongodb.html#mongodb-connector-properties)

## Advanced features

### Converter options

* org.apache.kafka.connect.json.JsonConverter

  The`json-with-envelope` config is valid only for the JsonConverter. By default, the value is set to false. When the `json-with-envelope` value is set to false, the consumer uses the schema `Schema.KeyValue(Schema.AUTO_CONSUME(), Schema.AUTO_CONSUME(), KeyValueEncodingType.SEPARATED)`, and the message only consists of the payload.
  When the `json-with-envelope` value is set to true, the consumer uses the schema `Schema.KeyValue(Schema.BYTES, Schema.BYTES)`, and the message consists of the schema and the payload.

* org.apache.pulsar.kafka.shade.io.confluent.connect.avro.AvroConverter

  If you select the AvroConverter, the consumer uses the schema `Schema.KeyValue(Schema.AUTO_CONSUME(), Schema.AUTO_CONSUME(), KeyValueEncodingType.SEPARATED)`, and the message consists of the payload.

### Used topic on Pulsar

Currently, the destination topic (specified by the `destination-topic-name` option ) is a required configuration but it is not used for the Debezium connector to save data. The Debezium connector saves data on the following 4 types of topics:

* One topic for storing the database metadata messages. It is named with the database server name ( `database.server.name`), like `public/default/database.server.name`.
* One topic (`offset.storage.topic`) for storing the offset metadata messages. The connector saves the last successfully-committed offsets on this topic.
* One topic (`schema.history.internal.pulsar.topic`) for storing the database history information. The connector writes and recovers DDL statements on this topic.
* One per-table topic. You can set multiple tables for "table.include.list", and the connector will send data from each table to a different topic of pulsar. The topic naming rule is: `{{database.server.name}}.{{table.name}}`. For examples: `public/default/mydbserver.public.io-test`.

If automatic topic creation is disabled on the Pulsar broker, you need to manually create these 4 types of topics and the destination topic.

For `offset.storage.topic` and `schema.history.internal.pulsar.topic`, If they are not specified in your connector's configuration, they will be created automatically using the following default naming convention:

* `schema.history.internal.pulsar.topic`: `"{tenant}/{namespace}/{connector-name}-debezium-history-topic"`
* `offset.storage.topic`: `"{tenant}/{namespace}/{connector-name}-offset-storage-topic"`

Here,  and  refer to the tenant and namespace where the connector is running.

Both the history and offset topics require their data to be retained indefinitely to ensure fault-tolerance and prevent data loss. Before running the connector, you must configure an infinite retention policy for both topics. Use the pulsar-admin CLI to set the retention policy:

```shell theme={null}
pulsar-admin topicPolicies set-retention -s -1 -t -1 ${topic_name}
```


# Debezium mssql source
Source: https://docs.streamnative.io/connect/connectors/debezium-mssql-source/current/debezium-mssql-source

The Debezium Microsoft SQL Server source connector pulls messages from SQL Server and persists the messages to Pulsar topics

The MSSQL source connector pulls messages from MSSQL and persists the messages to Pulsar topics by using debezium.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

<img alt="" />

## Quick start

### Prerequisites

The prerequisites for connecting a Debezium MSSQL source connector to external systems include:

1. Create a MSSQL service: This connector uses the debezium v1.9, Please refer to this [document](https://debezium.io/releases/1.9/) to see the compatible SQL Server versions.
2. Prepare SQL Server: Please refer to this [document](https://debezium.io/documentation/reference/1.9/connectors/sqlserver.html) to complete the prepare steps.
3. Enable CDC for SQL Server: Please refer to this [document](https://learn.microsoft.com/en-us/sql/relational-databases/track-changes/enable-and-disable-change-data-capture-sql-server?view=sql-server-ver15)
4. Enable SQL Server Agent: Please refer to this [document](https://learn.microsoft.com/en-us/sql/ssms/agent/start-stop-or-pause-the-sql-server-agent-service?view=sql-server-ver16)
5. Configure topic retention policies: Before running the connector, you must ensure that you have set an infinite retention policy for both the `offset.storage.topic` and `database.history.pulsar.topic`. Refer to the [Used Topic On Pulsar](#used-topic-on-pulsar) section for more details.

### 1. Prepare table and permission on SQL Server

Run the following SQL command on Microsoft SQL Server Management Studio:

```sql theme={null}
CREATE DATABASE mydb;
CREATE TABLE MyTable (id INT PRIMARY KEY, name NVARCHAR(50));
GRANT SELECT ON dbo.MyTable TO {{Your hostname of SQL Server}};
```

Note that we need a primary key for CDC.

### 2. Create a connector

The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector,
you need to replace `--source-type debezium-mssql` with `--archive /path/to/pulsar-io-debezium-mssql.nar`. You can find the button to download the `nar` package at the beginning of the document.

<Note title="For StreamNative Cloud User">
  If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first.
</Note>

```bash theme={null}
pulsarctl sources create \
  --source-type debezium-mssql \
  --name debezium-mssql \
  --tenant public \
  --namespace default \
  --parallelism 1 \
  --source-config \
  '{
    "database.hostname": "Your hostname of SQL Server",
    "database.port": "Your port of SQL Server",
    "database.user": "Your user of SQL Server",
    "database.password": "Your password of SQL Server",
    "database.dbname": "Your dbname of SQL Server",
    "table.whitelist": "public.io-test",
    "database.server.name": "mydbserver"
  }'
```

<Note title="Note">
  1. The `--parallelism` must be set to **1**. Debezium connectors do not support parallel consumption within a single instance. If you need to process tables in parallel, you can deploy multiple connector instances, each configured for different database schemas or tables.
  2. You can set multiple tables for "table.whitelist", and the connector will send data from each table to a different topic of pulsar. The topic naming rule is: `{{database.server.name}}.{{table.name}}`. For examples: `public/default/mydbserver.public.io-test`.
</Note>

The `--source-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.

If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference.

<Note title="Note">
  You can also choose to use a variety of other tools to create a connector:

  * [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [RestAPI](https://pulsar.apache.org/source-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document.
</Note>

### 3. Insert and update a data to table

You can insert and update using the sql:

```sql theme={null}
USE mydb;
INSERT INTO dbo.testtable
           ([id]
           ,[name])
     VALUES (1, 'Zike Yang');
GO
```

### 4. Show data using Pulsar client

<Note title="Note">
  If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information.
</Note>

```
bin/pulsar-client \
--url "Your Pulsar serviceUrl" \
consume "persistent://public/default/mssql-test.dbo.testtable" -s "test-sub" -n 0 -p Earliest

----- got message -----
key:[eyJpZCI6MTB9], properties:[], content:{"before":null,"after":{"id":1,"name":"Zike Yang"},"source":{"version":"1.9.7.Final","connector":"sqlserver","name":"mssql-test","ts_ms":1701424975073,"snapshot":"false","db":"mydb","sequence":null,"schema":"dbo","table":"testtable","change_lsn":"00000027:000005a0:0002","commit_lsn":"00000027:000005a0:0003","event_serial_no":1},"op":"c","ts_ms":1701424977325,"transaction":null}
```

## Configuration Properties

The configuration of Debezium source connector has the following properties.

| Name                                  | Required | Sensitive | Default | Description                                                                                                                                                                                                                                                                |
| ------------------------------------- | -------- | --------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `database.hostname`                   | true     | false     | null    | The address of a database server.                                                                                                                                                                                                                                          |
| `database.port`                       | true     | false     | null    | The port number of a database server.                                                                                                                                                                                                                                      |
| `database.user`                       | true     | true      | null    | The name of a database user that has the required privileges.                                                                                                                                                                                                              |
| `database.password`                   | true     | true      | null    | The password for a database user that has the required privileges.                                                                                                                                                                                                         |
| `database.dbname`                     | true     | false     | null    | The database.dbname parameter in Debezium configuration is used to specify the name of the specific database that the connector should connect to.                                                                                                                         |
| `database.server.name`                | true     | false     | null    | The logical name of a database server/cluster, which forms a namespace and it is used in all the names of Kafka topics to which the connector writes, the Kafka Connect schema names, and the namespaces of the corresponding Avro schema when the Avro Connector is used. |
| `database.server.id`                  | false    | false     | null    | The connector’s identifier that must be unique within a database cluster and similar to the database’s server-id configuration property.                                                                                                                                   |
| `database.whitelist`                  | false    | false     | null    | A list of all databases hosted by this server which is monitored by the  connector.<br /><br /> This is optional, and there are other properties for listing databases and tables to include or exclude from monitoring.                                                   |
| `table.whitelist`                     | false    | false     | null    | A list of all tables hosted by this server which is monitored by the  connector.<br /><br /> This is optional, and there are other properties for listing tables and tables to include or exclude from monitoring.                                                         |
| `key.converter`                       | false    | false     | null    | The converter provided by Kafka Connect to convert record key.                                                                                                                                                                                                             |
| `value.converter`                     | false    | false     | null    | The converter provided by Kafka Connect to convert record value.                                                                                                                                                                                                           |
| `database.history`                    | false    | false     | null    | The name of the database history class.                                                                                                                                                                                                                                    |
| `database.history.pulsar.topic`       | false    | false     | null    | The name of the database history topic where the connector writes and recovers DDL statements. <br /><br />**Note: this topic is for internal use only and should not be used by consumers.**                                                                              |
| `database.history.pulsar.service.url` | false    | false     | null    | Pulsar cluster service URL for history topic.                                                                                                                                                                                                                              |
| `pulsar.service.url`                  | false    | false     | null    | Pulsar cluster service URL.                                                                                                                                                                                                                                                |
| `offset.storage.topic`                | false    | false     | null    | Record the last committed offsets that the connector successfully completes.                                                                                                                                                                                               |

## Advanced features

### Converter options

* org.apache.kafka.connect.json.JsonConverter

  The`json-with-envelope` config is valid only for the JsonConverter. By default, the value is set to false. When the `json-with-envelope` value is set to false, the consumer uses the schema `Schema.KeyValue(Schema.AUTO_CONSUME(), Schema.AUTO_CONSUME(), KeyValueEncodingType.SEPARATED)`, and the message only consists of the payload.
  When the `json-with-envelope` value is set to true, the consumer uses the schema `Schema.KeyValue(Schema.BYTES, Schema.BYTES)`, and the message consists of the schema and the payload.

* org.apache.pulsar.kafka.shade.io.confluent.connect.avro.AvroConverter

  If you select the AvroConverter, the consumer uses the schema `Schema.KeyValue(Schema.AUTO_CONSUME(), Schema.AUTO_CONSUME(), KeyValueEncodingType.SEPARATED)`, and the message consists of the payload.

### Used topic on Pulsar

Currently, the destination topic (specified by the `destination-topic-name` option ) is a required configuration but it is not used for the Debezium connector to save data. The Debezium connector saves data on the following 4 types of topics:

* One topic for storing the database metadata messages. It is named with the database server name ( `database.server.name`), like `public/default/database.server.name`.
* One topic (`offset.storage.topic`) for storing the offset metadata messages. The connector saves the last successfully-committed offsets on this topic.
* One topic (`database.history.pulsar.topic`) for storing the database history information. The connector writes and recovers DDL statements on this topic.
* One per-table topic. You can set multiple tables for "table.whitelist", and the connector will send data from each table to a different topic of pulsar. The topic naming rule is: `{{database.server.name}}.{{table.name}}`. For examples: `public/default/mydbserver.public.io-test`.

If automatic topic creation is disabled on the Pulsar broker, you need to manually create these 4 types of topics and the destination topic.

For `offset.storage.topic` and `database.history.pulsar.topic`, If they are not specified in your connector's configuration, they will be created automatically using the following default naming convention:

* `database.history.pulsar.topic`: `"{tenant}/{namespace}/{connector-name}-debezium-history-topic"`
* `offset.storage.topic`: `"{tenant}/{namespace}/{connector-name}-offset-storage-topic"`

Here,  and  refer to the tenant and namespace where the connector is running.

Both the history and offset topics require their data to be retained indefinitely to ensure fault-tolerance and prevent data loss. Before running the connector, you must configure an infinite retention policy for both topics. Use the pulsar-admin CLI to set the retention policy:

```shell theme={null}
pulsar-admin topicPolicies set-retention -s -1 -t -1 ${topic_name}
```


# Debezium MySQL source
Source: https://docs.streamnative.io/connect/connectors/debezium-mysql-source/current/debezium-MySQL-source

The Debezium source connector pulls messages from MySQL and persists the messages to Pulsar topics.

The MySQL source connector pulls messages from MySQL and persists the messages to Pulsar topics by using debezium.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

<img alt="" />

## Quick start

### Prerequisites

The prerequisites for connecting a Debezium MySQL source connector to external systems include:

1. Create a MySQL service: This connector uses the debezium v3.2, Please refer to this [document](https://debezium.io/releases/3.2/) to see the compatible MySQL versions.
2. Prepare MySQL Database: Please refer to this [document](https://debezium.io/documentation/reference/3.2/connectors/mysql.html#setting-up-mysql) to complete the prepare steps on MySQL.
3. Configure topic retention policies: Before running the connector, you must ensure that you have set an infinite retention policy for both the `offset.storage.topic` and `schema.history.internal.pulsar.topic`. Refer to the [Used Topic On Pulsar](#used-topic-on-pulsar) section for more details.

<Note title="Note">
  If you are using AWS MySQL service, you need to use the [params group](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.html) to set the [binlog\_format](https://debezium.io/documentation/reference/3.2/connectors/mysql.html#enable-mysql-binlog) to `ROW`.
</Note>

### 1. Create a table on MySQL

Run the following SQL command on your MySQL.

```sql theme={null}
CREATE DATABASE io_database;
CREATE TABLE `io_database`.`io-test` (
    `id` INT AUTO_INCREMENT,
    `first_name` TEXT,
    `last_name` TEXT,
    `age` INT,
    PRIMARY KEY (`id`)
);
```

### 2. Create a connector

The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector,
you need to replace `--source-type debezium-mysql` with `--archive /path/to/pulsar-io-debezium-mysql.nar`. You can find the button to download the `nar` package at the beginning of the document.

<Note title="For StreamNative Cloud User">
  If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first.
</Note>

```bash theme={null}
pulsarctl sources create \
  --source-type debezium-mysql \
  --name debezium-mysql-source \
  --tenant public \
  --namespace default \
  --parallelism 1 \
  --source-config \
  '{
    "database.hostname": "Your hostname of MySQL",
    "database.port": "Your port of MySQL",
    "database.user": "Your user of MySQL",
    "database.password": "Your password of MySQL",
    "database.dbname": "Your dbname of MySQL",
    "table.include.list": "public.io-test",
    "topic.prefix": "mysql",
    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "database.server.name": "mydbserver"
  }'
```

<Note title="Note">
  1. The `--parallelism` must be set to **1**. Debezium connectors do not support parallel consumption within a single instance. If you need to process tables in parallel, you can deploy multiple connector instances, each configured for different database schemas or tables.
  2. You can set multiple tables for "table.include.list", and the connector will send data from each table to a different topic of pulsar. The topic naming rule is: `{{database.server.name}}.{{table.name}}`. For examples: `public/default/mydbserver.public.io-test`.
</Note>

The `--source-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.

If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference.

<Note title="Note">
  You can also choose to use a variety of other tools to create a connector:

  * [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [RestAPI](https://pulsar.apache.org/source-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document.
</Note>

### 3. Insert and update a data to table

You can insert and update using the sql:

```sql theme={null}
INSERT INTO `io_database`.`io-test` (`first_name`, `last_name`, `age`)
VALUES ('mysql-io-test', 'streamnative', 4);

UPDATE `io_database`.`io-test`
SET `age` = 5, `last_name` = 'sn'
WHERE `first_name` = 'mysql-io-test' AND `last_name` = 'streamnative';
```

### 4. Show data using Pulsar client

<Note title="Note">
  If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information.
</Note>

```
bin/pulsar-client \
--url "Your Pulsar serviceUrl" \
consume "public/default/mydbserver.io_database.io-test" -s "test-sub" -n 10 -p Earliest

----- got message -----
key:[eyJpZCI6MX0=], properties:[], content:{"before":null,"after":{"id":1,"first_name":"mysql-io-test","last_name":"streamnative","age":4},"source":{"version":"3.2.5.Final","connector":"mysql","name":"mydbserver","ts_ms":1698912778000,"snapshot":"false","db":"io_database","sequence":null,"table":"io-test","server_id":2017181175,"gtid":null,"file":"mysql-bin-changelog.000072","pos":1333,"row":0,"thread":null,"query":null},"op":"c","ts_ms":1698912778909,"transaction":null}
----- got message -----
key:[eyJpZCI6MX0=], properties:[], content:{"before":{"id":1,"first_name":"mysql-io-test","last_name":"streamnative","age":4},"after":{"id":1,"first_name":"mysql-io-test","last_name":"sn","age":5},"source":{"version":"3.2.5.Final","connector":"mysql","name":"mydbserver","ts_ms":1698912782000,"snapshot":"false","db":"io_database","sequence":null,"table":"io-test","server_id":2017181175,"gtid":null,"file":"mysql-bin-changelog.000072","pos":1677,"row":0,"thread":null,"query":null},"op":"u","ts_ms":1698912782158,"transaction":null}
```

## Configuration Properties

The configuration of Debezium source connector has the following properties.

| Name                                         | Required | Sensitive | Default | Description                                                                                                                                                                                                                                                                |
| -------------------------------------------- | -------- | --------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `database.hostname`                          | true     | false     | null    | The address of a database server.                                                                                                                                                                                                                                          |
| `database.port`                              | true     | false     | null    | The port number of a database server.                                                                                                                                                                                                                                      |
| `database.user`                              | true     | true      | null    | The name of a database user that has the required privileges.                                                                                                                                                                                                              |
| `database.password`                          | true     | true      | null    | The password for a database user that has the required privileges.                                                                                                                                                                                                         |
| `database.dbname`                            | true     | false     | null    | The database.dbname parameter in Debezium configuration is used to specify the name of the specific database that the connector should connect to.                                                                                                                         |
| `database.server.name`                       | true     | false     | null    | The logical name of a database server/cluster, which forms a namespace and it is used in all the names of Kafka topics to which the connector writes, the Kafka Connect schema names, and the namespaces of the corresponding Avro schema when the Avro Connector is used. |
| `topic.prefix`                               | true     | false     | null    | The prefix that is used in all the names of topics to which the connector writes.                                                                                                                                                                                          |
| `connector.class`                            | true     | false     | null    | The name of the Debezium MySQL connector class, can only be: 'io.debezium.connector.mysql.MySqlConnector'                                                                                                                                                                  |
| `database.server.id`                         | false    | false     | null    | The connector’s identifier that must be unique within a database cluster and similar to the database’s server-id configuration property.                                                                                                                                   |
| `database.include.list`                      | false    | false     | null    | A list of all databases hosted by this server which is monitored by the  connector.<br /><br /> This is optional, and there are other properties for listing databases and tables to include or exclude from monitoring.                                                   |
| `database.exclude.list`                      | false    | false     | null    | A list of all databases hosted by this server which is not monitored by the  connector.<br /><br /> This is optional, and there are other properties for listing databases and tables to include or exclude from monitoring.                                               |
| `table.include.list`                         | false    | false     | null    | A list of all tables hosted by this server which is monitored by the  connector.<br /><br /> This is optional, and there are other properties for listing tables and tables to include or exclude from monitoring.                                                         |
| `table.exclude.list`                         | false    | false     | null    | A list of all tables hosted by this server which is not monitored by the  connector.<br /><br /> This is optional, and there are other properties for listing tables and tables to include or exclude from monitoring.                                                     |
| `key.converter`                              | false    | false     | null    | The converter provided by Kafka Connect to convert record key.                                                                                                                                                                                                             |
| `value.converter`                            | false    | false     | null    | The converter provided by Kafka Connect to convert record value.                                                                                                                                                                                                           |
| `schema.history.internal.history`            | false    | false     | null    | The name of the database history class.                                                                                                                                                                                                                                    |
| `schema.history.internal.pulsar.topic`       | false    | false     | null    | The name of the database history topic where the connector writes and recovers DDL statements. <br /><br />**Note: this topic is for internal use only and should not be used by consumers.**                                                                              |
| `schema.history.internal.pulsar.service.url` | false    | false     | null    | Pulsar cluster service URL for history topic.                                                                                                                                                                                                                              |
| `pulsar.service.url`                         | false    | false     | null    | Pulsar cluster service URL.                                                                                                                                                                                                                                                |
| `offset.storage.topic`                       | false    | false     | null    | Record the last committed offsets that the connector successfully completes.                                                                                                                                                                                               |

For more configuration properties, please see [Debezium Mysql connector configuration properties](https://debezium.io/documentation/reference/3.2/connectors/mysql.html#mysql-connector-properties)

## Advanced features

### Converter options

* org.apache.kafka.connect.json.JsonConverter

  The`json-with-envelope` config is valid only for the JsonConverter. By default, the value is set to false. When the `json-with-envelope` value is set to false, the consumer uses the schema `Schema.KeyValue(Schema.AUTO_CONSUME(), Schema.AUTO_CONSUME(), KeyValueEncodingType.SEPARATED)`, and the message only consists of the payload.
  When the `json-with-envelope` value is set to true, the consumer uses the schema `Schema.KeyValue(Schema.BYTES, Schema.BYTES)`, and the message consists of the schema and the payload.

* org.apache.pulsar.kafka.shade.io.confluent.connect.avro.AvroConverter

  If you select the AvroConverter, the consumer uses the schema `Schema.KeyValue(Schema.AUTO_CONSUME(), Schema.AUTO_CONSUME(), KeyValueEncodingType.SEPARATED)`, and the message consists of the payload.

### Used topic on Pulsar

Currently, the destination topic (specified by the `destination-topic-name` option ) is a required configuration but it is not used for the Debezium connector to save data. The Debezium connector saves data on the following 4 types of topics:

* One topic for storing the database metadata messages. It is named with the database server name ( `database.server.name`), like `public/default/database.server.name`.
* One topic (`offset.storage.topic`) for storing the offset metadata messages. The connector saves the last successfully-committed offsets on this topic.
* One topic (`schema.history.internal.pulsar.topic`) for storing the database history information. The connector writes and recovers DDL statements on this topic.
* One per-table topic. You can set multiple tables for "table.include.list", and the connector will send data from each table to a different topic of pulsar. The topic naming rule is: `{{database.server.name}}.{{table.name}}`. For examples: `public/default/mydbserver.public.io-test`.

If automatic topic creation is disabled on the Pulsar broker, you need to manually create these 4 types of topics and the destination topic.

For `offset.storage.topic` and `schema.history.internal.pulsar.topic`, If they are not specified in your connector's configuration, they will be created automatically using the following default naming convention:

* `schema.history.internal.pulsar.topic`: `"{tenant}/{namespace}/{connector-name}-debezium-history-topic"`
* `offset.storage.topic`: `"{tenant}/{namespace}/{connector-name}-offset-storage-topic"`

Here,  and  refer to the tenant and namespace where the connector is running.

Both the history and offset topics require their data to be retained indefinitely to ensure fault-tolerance and prevent data loss. Before running the connector, you must configure an infinite retention policy for both topics. Use the pulsar-admin CLI to set the retention policy:

```shell theme={null}
pulsar-admin topicPolicies set-retention -s -1 -t -1 ${topic_name}
```


# Debezium postgres source
Source: https://docs.streamnative.io/connect/connectors/debezium-postgres-source/current/debezium-postgres-source

The Debezium source connector pulls messages from PostgreSQL and persists the messages to Pulsar topics.

The Postgres source connector pulls messages from PostgreSQL and persists the messages to Pulsar topics by using debezium.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

<img alt="" />

## Quick start

### Prerequisites

The prerequisites for connecting a Debezium Postgres source connector to external systems include:

1. Create a Postgres service: This connector uses the debezium v3.2, Please refer to this [document](https://debezium.io/releases/3.2/) to see the compatible PostgreSQL versions.
2. Prepare Postgres Database: Please refer to this [document](https://debezium.io/documentation/reference/3.2/connectors/postgresql.html#setting-up-postgresql) to complete the prepare steps on Postgres.
3. Configure topic retention policies: Before running the connector, you must ensure that you have set an infinite retention policy for both the `offset.storage.topic` and `schema.history.internal.pulsar.topic`. Refer to the [Used Topic On Pulsar](#used-topic-on-pulsar) section for more details.

<Note title="Note">
  The subsequent deployment steps detailed in this document leverage PostgreSQL 11.16 on AWS RDS, which natively supports the `pgoutput` plugin.
</Note>

### 1. Create a table on Postgres

Run the following SQL command on your PostgreSQL. If you don't require the `before` data, you can disregard the configuration of `REPLICA IDENTITY`.

```sql theme={null}
CREATE TABLE "public"."io-test" (
    "id" integer GENERATED ALWAYS AS IDENTITY,
    "first_name" text,
    "last_name" text,
    "age" integer,
    PRIMARY KEY ("id")
);
ALTER TABLE "public"."io-test" REPLICA IDENTITY FULL;
```

### 2. Create a connector

The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector,
you need to replace `--source-type debezium-postgres` with `--archive /path/to/pulsar-io-debezium-postgres.nar`. You can find the button to download the `nar` package at the beginning of the document.

<Note title="For StreamNative Cloud User">
  If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first.
</Note>

```bash theme={null}
pulsarctl sources create \
  --source-type debezium-postgres \
  --name debezium-postgres-source \
  --tenant public \
  --namespace default \
  --parallelism 1 \
  --source-config \
  '{
    "database.hostname": "Your hostname of Postgres",
    "database.port": "Your port of Postgres",
    "database.user": "Your user of Postgres",
    "database.password": "Your password of Postgres",
    "database.dbname": "Your dbname of Postgres",
    "table.include.list": "public.io-test",
    "database.server.name": "mydbserver",
    "topic.prefix": "postgres",
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "plugin.name": "pgoutput"
  }'
```

<Note title="Note">
  1. The `--parallelism` must be set to **1**. Debezium connectors do not support parallel consumption within a single instance. If you need to process tables in parallel, you can deploy multiple connector instances, each configured for different database schemas or tables.
  2. You can set multiple tables for "table.include.list", and the connector will send data from each table to a different topic of pulsar. The topic naming rule is: `{{database.server.name}}.{{table.name}}`. For examples: `public/default/mydbserver.public.io-test`.
</Note>

The `--source-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.

If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference.

<Note title="Note">
  You can also choose to use a variety of other tools to create a connector:

  * [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [RestAPI](https://pulsar.apache.org/source-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document.
</Note>

### 3. Insert and update a data to table

You can insert and update using the sql:

```sql theme={null}
INSERT INTO "public"."io-test" (first_name, last_name, age)
VALUES ('pg-io-test', 'streamnative', 4);

UPDATE "public"."io-test"
SET age = 5, last_name = 'sn'
WHERE first_name = 'pg-io-test' AND last_name = 'streamnative';
```

### 4. Show data using Pulsar client

<Note title="Note">
  If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information.
</Note>

```
bin/pulsar-client \
--url "Your Pulsar serviceUrl" \
consume "public/default/mydbserver.public.io-test" -s "test-sub" -n 10 -p Earliest

----- got message -----
key:[eyJpZCI6Mn0=], properties:[], content:{"before":null,"after":{"id":1,"first_name":"pg-io-test","last_name":"streamnative","age":4},"source":{"version":"3.2.5.Final","connector":"postgresql","name":"mydbserver","ts_ms":1698825100079,"snapshot":"false","db":"postgres","sequence":"[null,\"18052284768\"]","schema":"public","table":"io-test","txId":2245,"lsn":18052284768,"xmin":null},"op":"c","ts_ms":1698825103451,"transaction":null}
----- got message -----
key:[eyJpZCI6M30=], properties:[], content:{"before":{"id":1,"first_name":"pg-io-test","last_name":"streamnative","age":4},"after":{"id":1,"first_name":"pg-io-test","last_name":"sn","age":5},"source":{"version":"3.2.5.Final","connector":"postgresql","name":"mydbserver","ts_ms":1698826703631,"snapshot":"false","db":"postgres","sequence":"[\"18387831504\",\"18387832144\"]","schema":"public","table":"io-test","txId":2284,"lsn":18387832144,"xmin":null},"op":"u","ts_ms":1698826704159,"transaction":null}
```

## Configuration Properties

The configuration of Debezium source connector has the following properties.

| Name                                         | Required | Sensitive | Default | Description                                                                                                                                                                                                                                                                |
| -------------------------------------------- | -------- | --------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `database.hostname`                          | true     | false     | null    | The address of a database server.                                                                                                                                                                                                                                          |
| `database.port`                              | true     | false     | null    | The port number of a database server.                                                                                                                                                                                                                                      |
| `database.user`                              | true     | true      | null    | The name of a database user that has the required privileges.                                                                                                                                                                                                              |
| `database.password`                          | true     | true      | null    | The password for a database user that has the required privileges.                                                                                                                                                                                                         |
| `database.dbname`                            | true     | false     | null    | The database.dbname parameter in Debezium configuration is used to specify the name of the specific database that the connector should connect to.                                                                                                                         |
| `plugin.name`                                | true     | false     | null    | The plugin.name parameter in Debezium configuration is used to specify the logical decoding output plugin installed on the PostgreSQL server that the connector should use: `decoderbufs`, `wal2json`, `pgoutput`                                                          |
| `database.server.name`                       | true     | false     | null    | The logical name of a database server/cluster, which forms a namespace and it is used in all the names of Kafka topics to which the connector writes, the Kafka Connect schema names, and the namespaces of the corresponding Avro schema when the Avro Connector is used. |
| `topic.prefix`                               | true     | false     | null    | The prefix that is used to name the Kafka topics to which the connector writes the change events.                                                                                                                                                                          |
| `connector.class`                            | true     | false     | null    | The name of the Debezium connector class to use. Can only be: `io.debezium.connector.postgresql.PostgresConnector`.                                                                                                                                                        |
| `database.server.id`                         | false    | false     | null    | The connector’s identifier that must be unique within a database cluster and similar to the database’s server-id configuration property.                                                                                                                                   |
| `schema.include.list`                        | false    | false     | null    | A list of all schemas hosted by this server which is monitored by the  connector.<br /><br /> This is optional, and there are other properties for listing schemas and tables to include or exclude from monitoring.                                                       |
| `schema.exclude.list`                        | false    | false     | null    | A list of all schemas hosted by this server which is not monitored by the  connector.<br /><br /> This is optional, and there are other properties for listing schemas and tables to include or exclude from monitoring.                                                   |
| `table.include.list`                         | false    | false     | null    | A list of all tables hosted by this server which is monitored by the  connector.<br /><br /> This is optional, and there are other properties for listing tables and tables to include or exclude from monitoring.                                                         |
| `table.exclude.list`                         | false    | false     | null    | A list of all tables hosted by this server which is not monitored by the  connector.<br /><br /> This is optional, and there are other properties for listing tables and tables to include or exclude from monitoring.                                                     |
| `key.converter`                              | false    | false     | null    | The converter provided by Kafka Connect to convert record key.                                                                                                                                                                                                             |
| `value.converter`                            | false    | false     | null    | The converter provided by Kafka Connect to convert record value.                                                                                                                                                                                                           |
| `schema.history.internal`                    | false    | false     | null    | The name of the database history class.                                                                                                                                                                                                                                    |
| `schema.history.internal.pulsar.topic`       | false    | false     | null    | The name of the database history topic where the connector writes and recovers DDL statements. <br /><br />**Note: this topic is for internal use only and should not be used by consumers.**                                                                              |
| `schema.history.internal.pulsar.service.url` | false    | false     | null    | Pulsar cluster service URL for history topic.                                                                                                                                                                                                                              |
| `pulsar.service.url`                         | false    | false     | null    | Pulsar cluster service URL.                                                                                                                                                                                                                                                |
| `offset.storage.topic`                       | false    | false     | null    | Record the last committed offsets that the connector successfully completes.                                                                                                                                                                                               |

For more configuration properties, please see [Debezium PostgreSQL connector configuration properties](https://debezium.io/documentation/reference/3.2/connectors/postgresql.html#postgresql-connector-properties)

## Advanced features

### Converter options

* org.apache.kafka.connect.json.JsonConverter

  The`json-with-envelope` config is valid only for the JsonConverter. By default, the value is set to false. When the `json-with-envelope` value is set to false, the consumer uses the schema `Schema.KeyValue(Schema.AUTO_CONSUME(), Schema.AUTO_CONSUME(), KeyValueEncodingType.SEPARATED)`, and the message only consists of the payload.
  When the `json-with-envelope` value is set to true, the consumer uses the schema `Schema.KeyValue(Schema.BYTES, Schema.BYTES)`, and the message consists of the schema and the payload.

* org.apache.pulsar.kafka.shade.io.confluent.connect.avro.AvroConverter

  If you select the AvroConverter, the consumer uses the schema `Schema.KeyValue(Schema.AUTO_CONSUME(), Schema.AUTO_CONSUME(), KeyValueEncodingType.SEPARATED)`, and the message consists of the payload.

### Used topic on Pulsar

Currently, the destination topic (specified by the `destination-topic-name` option ) is a required configuration but it is not used for the Debezium connector to save data. The Debezium connector saves data on the following 4 types of topics:

* One topic for storing the database metadata messages. It is named with the database server name ( `database.server.name`), like `public/default/database.server.name`.
* One topic (`offset.storage.topic`) for storing the offset metadata messages. The connector saves the last successfully-committed offsets on this topic.
* One topic (`schema.history.internal.pulsar.topic`) for storing the database history information. The connector writes and recovers DDL statements on this topic.
* One per-table topic. You can set multiple tables for "table.include.list", and the connector will send data from each table to a different topic of pulsar. The topic naming rule is: `{{database.server.name}}.{{table.name}}`. For examples: `public/default/mydbserver.public.io-test`.

If automatic topic creation is disabled on the Pulsar broker, you need to manually create these 4 types of topics and the destination topic.

For `offset.storage.topic` and `schema.history.internal.pulsar.topic`, If they are not specified in your connector's configuration, they will be created automatically using the following default naming convention:

* `schema.history.internal.pulsar.topic`: `"{tenant}/{namespace}/{connector-name}-debezium-history-topic"`
* `offset.storage.topic`: `"{tenant}/{namespace}/{connector-name}-offset-storage-topic"`

Here,  and  refer to the tenant and namespace where the connector is running.

Both the history and offset topics require their data to be retained indefinitely to ensure fault-tolerance and prevent data loss. Before running the connector, you must configure an infinite retention policy for both topics. Use the pulsar-admin CLI to set the retention policy:

```shell theme={null}
pulsar-admin topicPolicies set-retention -s -1 -t -1 ${topic_name}
```


# Dynamodb source
Source: https://docs.streamnative.io/connect/connectors/dynamodb-source/current/dynamodb-source

The Dynamodb source connector pulls messages from Dynamodb to Pulsar topics.

The DynamoDB source connector pulls data from DynamoDB table streams and persists data into Pulsar.

This connector uses the [DynamoDB Streams Kinesis Adapter](https://github.com/awslabs/dynamodb-streams-kinesis-adapter),
which uses the [Kinesis Consumer Library](https://github.com/awslabs/amazon-kinesis-client) (KCL) to do the actual
consuming of messages. The KCL uses DynamoDB to track the state of consumers and requires cloudwatch access to log metrics.

## Quick start

### Prerequisites

The prerequisites for connecting an AWS DynamoDB source connector to external systems include:

1. Enable stream to the DynamoDB in AWS, it will show a `Latest stream ARN` after the stream is turned on.
2. Create an [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and an `AccessKey`(Please record the value of `AccessKey` and its `SecretKey`).
3. Assign the following permissions to the AWS User:

* [AmazonDynamoDBFullAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonDynamoDBFullAccess.html)
* [CloudWatch:PutMetricData](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_PutMetricData.html): it is required because this connector will periodically [send metrics to CloudWatch](https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-cloudwatch.html).

### 1. Create a connector

The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector,
you need to replace `--source-type dynamodb` with `--archive /path/to/pulsar-io-dynamodb.nar`. You can find the button to download the `nar` package at the beginning of the document.

<Note title="For StreamNative Cloud User">
  If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first.
</Note>

```bash theme={null}
pulsarctl sources create \
  --source-type dynamodb \
  --name dynamodb-source \
  --tenant public \
  --namespace default \
  --destination-topic-name "Your topic name" \
  --parallelism 1 \
  --source-config \
  '{
    "awsRegion": "Your aws dynamodb region", 
    "awsDynamodbStreamArn": "the Latest stream ARN of the DynamoDB table",
    "awsCredentialPluginParam": "{\"accessKey\":\"Your AWS access key\",\"secretKey\":\"Your AWS secret access key\"}",
    "applicationName": "Your application name, which will be used as the table name for DynamoDB. E.g.: pulsar-io-dynamodb"
  }'
```

The `--source-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.
If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference.

<Note title="Note">
  You can also choose to use a variety of other tools to create a connector:

  * [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [RestAPI](https://pulsar.apache.org/source-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document.
</Note>

### 2. Insert row to DynamoDB table

```java theme={null}
    public static void main(String[] args) {
        // 1. Create a DynamoDB client
        DynamoDbClient ddb = DynamoDbClient.builder()
                .region(Region.EU_NORTH_1)  // change to your region
                .credentialsProvider(ProfileCredentialsProvider.create())
                .build();

        for (int i = 0; i < 10; ++i) {
            // 2. Define the item to insert
            Map<String, AttributeValue> item = new HashMap<>();
            item.put("streamnative", AttributeValue.fromS("user" + i));   // partition key
            item.put("name", AttributeValue.fromS("Alice"));
            item.put("age", AttributeValue.fromN("30"));
            item.put("email", AttributeValue.fromS("Alive@test.com"));

            // 3. Create the PutItem request
            PutItemRequest request = PutItemRequest.builder()
                    .tableName("pulsar") // change to your table name
                    .item(item)
                    .build();

            // 4. Send the request
            try {
                ddb.putItem(request);
                System.out.println("Item inserted successfully.");
            } catch (DynamoDbException e) {
                System.err.println("Insert failed: " + e.getMessage());
            }
        }

        // 5. Close client
        ddb.close();
    }
```

### 3. Show data using Pulsar client

```
bin/pulsar-client \
--url "Your Pulsar serviceUrl" \
consume "The topic that you specified when you created the connector" -s "test-sub" -n 10 -p Earliest

----- got message -----
publishTime:[1752464541877], eventTime:[0], key:[390cc13abf07d7b9a233daa65231d6c8], properties:[SEQUENCE_NUMBER=18031900001003631107895421, EVENT_NAME=INSERT], content:{"eventID":"390cc13abf07d7b9a233daa65231d6c8","eventName":"INSERT","eventVersion":"1.1","eventSource":"aws:dynamodb","awsRegion":"eu-north-1","dynamodb":{"ApproximateCreationDateTime":1752464541000,"Keys":{"streamnative":{"S":"user2"}},"NewImage":{"name":{"S":"Alice"},"streamnative":{"S":"user2"},"age":{"N":"30"},"email":{"S":"Alive@test.com"}},"SequenceNumber":"18031900001003631107895421","SizeBytes":67,"StreamViewType":"NEW_AND_OLD_IMAGES"}}
----- got message -----
publishTime:[1752464542809], eventTime:[0], key:[cf6f7706eb6e15f7ee7accdde36fd878], properties:[EVENT_NAME=INSERT, SEQUENCE_NUMBER=18035200003649242324186849], content:{"eventID":"cf6f7706eb6e15f7ee7accdde36fd878","eventName":"INSERT","eventVersion":"1.1","eventSource":"aws:dynamodb","awsRegion":"eu-north-1","dynamodb":{"ApproximateCreationDateTime":1752464542000,"Keys":{"streamnative":{"S":"user4"}},"NewImage":{"name":{"S":"Alice"},"streamnative":{"S":"user4"},"age":{"N":"30"},"email":{"S":"Alive@test.com"}},"SequenceNumber":"18035200003649242324186849","SizeBytes":67,"StreamViewType":"NEW_AND_OLD_IMAGES"}}
----- got message -----
publishTime:[1752464542809], eventTime:[0], key:[dd660a4071b9607681be4f56ed7b5b73], properties:[EVENT_NAME=INSERT, SEQUENCE_NUMBER=18032900003894885254372246], content:{"eventID":"dd660a4071b9607681be4f56ed7b5b73","eventName":"INSERT","eventVersion":"1.1","eventSource":"aws:dynamodb","awsRegion":"eu-north-1","dynamodb":{"ApproximateCreationDateTime":1752464541000,"Keys":{"streamnative":{"S":"user1"}},"NewImage":{"name":{"S":"Alice"},"streamnative":{"S":"user1"},"age":{"N":"30"},"email":{"S":"Alive@test.com"}},"SequenceNumber":"18032900003894885254372246","SizeBytes":67,"StreamViewType":"NEW_AND_OLD_IMAGES"}}
----- got message -----
publishTime:[1752464542809], eventTime:[0], key:[526bf489c407d0d7e65af22a9a3461a5], properties:[EVENT_NAME=INSERT, SEQUENCE_NUMBER=18032100001449977882361981], content:{"eventID":"526bf489c407d0d7e65af22a9a3461a5","eventName":"INSERT","eventVersion":"1.1","eventSource":"aws:dynamodb","awsRegion":"eu-north-1","dynamodb":{"ApproximateCreationDateTime":1752464542000,"Keys":{"streamnative":{"S":"user5"}},"NewImage":{"name":{"S":"Alice"},"streamnative":{"S":"user5"},"age":{"N":"30"},"email":{"S":"Alive@test.com"}},"SequenceNumber":"18032100001449977882361981","SizeBytes":67,"StreamViewType":"NEW_AND_OLD_IMAGES"}}
----- got message -----
publishTime:[1752464543205], eventTime:[0], key:[25a0239947e90a1aa7d1fae5cd70cd96], properties:[EVENT_NAME=INSERT, SEQUENCE_NUMBER=18037000000661915555888784], content:{"eventID":"25a0239947e90a1aa7d1fae5cd70cd96","eventName":"INSERT","eventVersion":"1.1","eventSource":"aws:dynamodb","awsRegion":"eu-north-1","dynamodb":{"ApproximateCreationDateTime":1752464541000,"Keys":{"streamnative":{"S":"user0"}},"NewImage":{"name":{"S":"Alice"},"streamnative":{"S":"user0"},"age":{"N":"30"},"email":{"S":"Alive@test.com"}},"SequenceNumber":"18037000000661915555888784","SizeBytes":67,"StreamViewType":"NEW_AND_OLD_IMAGES"}}
----- got message -----
publishTime:[1752464543205], eventTime:[0], key:[a7bf1cbf97f036c198e4f49f86749a3a], properties:[EVENT_NAME=INSERT, SEQUENCE_NUMBER=18037300002951028206567171], content:{"eventID":"a7bf1cbf97f036c198e4f49f86749a3a","eventName":"INSERT","eventVersion":"1.1","eventSource":"aws:dynamodb","awsRegion":"eu-north-1","dynamodb":{"ApproximateCreationDateTime":1752464542000,"Keys":{"streamnative":{"S":"user3"}},"NewImage":{"name":{"S":"Alice"},"streamnative":{"S":"user3"},"age":{"N":"30"},"email":{"S":"Alive@test.com"}},"SequenceNumber":"18037300002951028206567171","SizeBytes":67,"StreamViewType":"NEW_AND_OLD_IMAGES"}}
----- got message -----
publishTime:[1752464543205], eventTime:[0], key:[14e87c4eb4a5b352422ab5632f69d226], properties:[EVENT_NAME=INSERT, SEQUENCE_NUMBER=18037500000367820008016475], content:{"eventID":"14e87c4eb4a5b352422ab5632f69d226","eventName":"INSERT","eventVersion":"1.1","eventSource":"aws:dynamodb","awsRegion":"eu-north-1","dynamodb":{"ApproximateCreationDateTime":1752464543000,"Keys":{"streamnative":{"S":"user6"}},"NewImage":{"name":{"S":"Alice"},"streamnative":{"S":"user6"},"age":{"N":"30"},"email":{"S":"Alive@test.com"}},"SequenceNumber":"18037500000367820008016475","SizeBytes":67,"StreamViewType":"NEW_AND_OLD_IMAGES"}}
----- got message -----
publishTime:[1752464543795], eventTime:[0], key:[6267f254261b4952b4ad6682f88dc821], properties:[SEQUENCE_NUMBER=18035500001731177388073396, EVENT_NAME=INSERT], content:{"eventID":"6267f254261b4952b4ad6682f88dc821","eventName":"INSERT","eventVersion":"1.1","eventSource":"aws:dynamodb","awsRegion":"eu-north-1","dynamodb":{"ApproximateCreationDateTime":1752464543000,"Keys":{"streamnative":{"S":"user7"}},"NewImage":{"name":{"S":"Alice"},"streamnative":{"S":"user7"},"age":{"N":"30"},"email":{"S":"Alive@test.com"}},"SequenceNumber":"18035500001731177388073396","SizeBytes":67,"StreamViewType":"NEW_AND_OLD_IMAGES"}}
----- got message -----
publishTime:[1752464544799], eventTime:[0], key:[eea816f9d21323655124a59e0f1b2aab], properties:[SEQUENCE_NUMBER=18035600004198491318198578, EVENT_NAME=INSERT], content:{"eventID":"eea816f9d21323655124a59e0f1b2aab","eventName":"INSERT","eventVersion":"1.1","eventSource":"aws:dynamodb","awsRegion":"eu-north-1","dynamodb":{"ApproximateCreationDateTime":1752464544000,"Keys":{"streamnative":{"S":"user9"}},"NewImage":{"name":{"S":"Alice"},"streamnative":{"S":"user9"},"age":{"N":"30"},"email":{"S":"Alive@test.com"}},"SequenceNumber":"18035600004198491318198578","SizeBytes":67,"StreamViewType":"NEW_AND_OLD_IMAGES"}}
----- got message -----
publishTime:[1752464545795], eventTime:[0], key:[bdc4688da075ed491848e21009ab2db1], properties:[SEQUENCE_NUMBER=18033200003338420245628766, EVENT_NAME=INSERT], content:{"eventID":"bdc4688da075ed491848e21009ab2db1","eventName":"INSERT","eventVersion":"1.1","eventSource":"aws:dynamodb","awsRegion":"eu-north-1","dynamodb":{"ApproximateCreationDateTime":1752464543000,"Keys":{"streamnative":{"S":"user8"}},"NewImage":{"name":{"S":"Alice"},"streamnative":{"S":"user8"},"age":{"N":"30"},"email":{"S":"Alive@test.com"}},"SequenceNumber":"18033200003338420245628766","SizeBytes":67,"StreamViewType":"NEW_AND_OLD_IMAGES"}}
```

## Configuration Properties

This table outlines the properties of an AWS DynamoDB source connector.

| Name                       | Required | Sensitive | Default           | Description                                                                                                                                                                                                                                                                                                                               |
| -------------------------- | -------- | --------- | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `awsEndpoint`              | false    | false     | None              | Dynamodb streams end-point url.                                                                                                                                                                                                                                                                                                           |
| `awsRegion`                | true     | false     | ''                | Appropriate aws region. E.g. us-west-1, us-west-2.                                                                                                                                                                                                                                                                                        |
| `awsDynamodbStreamArn`     | true     | false     | ''                | the Dynamodb stream arn.                                                                                                                                                                                                                                                                                                                  |
| `awsCredentialPluginName`  | false    | false     | None              | Fully-Qualified class name of implementation of AwsCredentialProviderPlugin..                                                                                                                                                                                                                                                             |
| `awsCredentialPluginParam` | false    | true      | " "               | The JSON parameter to initialize `AwsCredentialsProviderPlugin`.                                                                                                                                                                                                                                                                          |
| `initialPositionInStream`  | false    | false     | LATEST            | Used to specify the position in the stream where the connector should start from, The available options are:<br /> - AT\_TIMESTAMP: Start from the record at or after the specified timestamp<br /> - LATEST: Start after the most recent data record (fetch new data)<br /> - TRIM\_HORIZON: Start from the oldest available data record |
| `startAtTime`              | false    | false     | None              | If the `initalPositionInStream` is set to 'AT\_TIMESTAMP', then this property specifies the point in time to start consumption.                                                                                                                                                                                                           |
| `applicationName`          | false    | false     | "pulsar-dynamodb" | Name of the dynamodb consumer application.                                                                                                                                                                                                                                                                                                |
| `checkpointInterval`       | false    | false     | 60000             | The frequency of the stream checkpointing (in milliseconds).                                                                                                                                                                                                                                                                              |
| `backoffTime`              | false    | false     | 3000              | The amount of time to delay between requests when the connector encounters a Throttling exception from dynamodb (in milliseconds).                                                                                                                                                                                                        |
| `numRetries`               | false    | false     | 3                 | The number of re-attempts to make when the connector encounters an exception while trying to set a checkpoint.                                                                                                                                                                                                                            |
| `receiveQueueSize`         | false    | false     | 1000              | The maximum number of AWS Records that can be buffered inside the connector.                                                                                                                                                                                                                                                              |
| `dynamoEndpoint`           | false    | false     | None              | Dynamo end-point url.                                                                                                                                                                                                                                                                                                                     |
| `cloudwatchEndpoint`       | false    | false     | None              | Cloudwatch end-point url.                                                                                                                                                                                                                                                                                                                 |

### Configure AwsCredentialProviderPlugin

AWS DynamoDB source connector allows you to use three ways to connect to AWS DynamoDB by configuring `awsCredentialPluginName`.

* Leave `awsCredentialPluginName` empty to get the connector authenticated by passing `accessKey` and `secretKey` in `awsCredentialPluginParam`.

  ```json theme={null}
  {"accessKey":"Your access key","secretKey":"Your secret key"}
  ```

* Set `awsCredentialPluginName` to `org.apache.pulsar.io.aws.AwsDefaultProviderChainPlugin` to use the default AWS provider chain. With this option, you don’t need to configure `awsCredentialPluginParam`. For more information, see [AWS documentation](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default).

* Set `awsCredentialPluginName`to `org.apache.pulsar.io.aws.STSAssumeRoleProviderPlugin` to use the [default AWS provider chain](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default), and you need to configure `roleArn` and `roleSessionNmae` in `awsCredentialPluginParam`. For more information, see [AWS documentation](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html)

  ```json theme={null}
  {"roleArn": "arn...", "roleSessionName": "name"}
  ```


# Elasticsearch sink
Source: https://docs.streamnative.io/connect/connectors/elasticsearch-sink/current/elasticsearch-sink

The Elasticsearch sink connector pulls messages from Pulsar topics and persists the messages to indexes

The [Elasticsearch](https://www.elastic.co/elasticsearch/) sink connector pulls messages from Pulsar topics and persists the messages to indexes. For more information about connectors, see [Connector Overview](https://docs.streamnative.io/docs/connector-overview).

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

<img alt="" />

This document introduces how to get started with creating an Elasticsearch sink connector and get it up and running.

## Quick start

### Prerequisites

The prerequisites for connecting an Elasticsearch sink connector to external systems include:

Create a Elasticsearch cluster. You can create a single-node Elasticsearch cluster by executing this command:

```bash theme={null}
docker run -p 9200:9200 -p 9300:9300 \
  -e "discovery.type=single-node" \
       -e "ELASTIC_PASSWORD=pulsar-sink-test" \
  docker.elastic.co/elasticsearch/elasticsearch:7.17.13
```

### 1. Create a connector

The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector,
you need to replace `--sink-type elastic-search` with `--archive /path/to/pulsar-io-elastic-search.nar`. You can find the button to download the `nar` package at the beginning of the document.

<Note title="For StreamNative Cloud User">
  If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first.
</Note>

```bash theme={null}
pulsarctl sinks create \
  --sink-type elastic-search \
  --name es-sink \
  --tenant public \
  --namespace default \
  --inputs "Your topic name" \
  --parallelism 1 \
  --sink-config \
  '{
    "elasticSearchUrl": "http://localhost:90902",
    "indexName": "myindex",
    "typeName": "doc",
    "username": "elastic",
    "password": "pulsar-sink-test"
  }'
```

The `--sink-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.
If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference.

<Note title="Note">
  You can also choose to use a variety of other tools to create a connector:

  * [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [RestAPI](https://pulsar.apache.org/sink-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document.
</Note>

### 2. Send messages to the topic

<Note title="Note">
  If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information.
</Note>

```java theme={null}
   PulsarClient client = PulsarClient.builder()
            .serviceUrl("{{Your Pulsar URL}}")
            .build();

   Producer<String> producer = client.newProducer(Schema.STRING)
     .topic("`{{Your topic name}}`")
     .create();

   String message = "{\"a\":1}";
   MessageId msgID = producer.send(message);
   System.out.println("Publish " + message + " and message ID " + msgID);

   producer.flush();
   producer.close();
   client.close();
```

### 3. Check documents in Elasticsearch

* Refresh the index

```bash theme={null}
curl -s http://localhost:9200/my_index/_refresh
```

* Search documents

```bash theme={null}
curl -s http://localhost:9200/my_index/_search
```

* You can see the record that was published earlier has been successfully written into Elasticsearch.

```json theme={null}
{"took":2,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":1,"relation":"eq"},"max_score":1.0,"hits":[{"_index":"my_index","_type":"_doc","_id":"FSxemm8BLjG_iC0EeTYJ","_score":1.0,"_source":{"a":1}}]}}
```

## Configuration Properties

This table outlines the properties of an Elasticsearch sink connector.

| Name                           | Type                                                  | Required | Sensitive | Default            | Description                                                                                                                                                                                                                                                                                                                                                                    |
| ------------------------------ | ----------------------------------------------------- | -------- | --------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `elasticSearchUrl`             | String                                                | true     | false     | " " (empty string) | The URL of elastic search cluster to which the connector connects.                                                                                                                                                                                                                                                                                                             |
| `indexName`                    | String                                                | false    | false     | " " (empty string) | The index name to which the connector writes messages. The default value is the topic name. It accepts date formats in the name to support event time based index with the pattern `%{+<date-format>}`. For example, suppose the event time of the record is 1645182000000L, the indexName is `logs-%{+yyyy-MM-dd}`, then the formatted index name would be `logs-2022-02-18`. |
| `schemaEnable`                 | Boolean                                               | false    | false     | false              | Turn on the Schema Aware mode.                                                                                                                                                                                                                                                                                                                                                 |
| `createIndexIfNeeded`          | Boolean                                               | false    | false     | false              | Manage index if missing.                                                                                                                                                                                                                                                                                                                                                       |
| `maxRetries`                   | Integer                                               | false    | false     | 1                  | The maximum number of retries for elasticsearch requests. Use -1 to disable it.                                                                                                                                                                                                                                                                                                |
| `retryBackoffInMs`             | Integer                                               | false    | false     | 100                | The base time to wait when retrying an Elasticsearch request (in milliseconds).                                                                                                                                                                                                                                                                                                |
| `maxRetryTimeInSec`            | Integer                                               | false    | false     | 86400              | The maximum retry time interval in seconds for retrying an elasticsearch request.                                                                                                                                                                                                                                                                                              |
| `bulkEnabled`                  | Boolean                                               | false    | false     | false              | Enable the elasticsearch bulk processor to flush write requests based on the number or size of requests, or after a given period.                                                                                                                                                                                                                                              |
| `bulkActions`                  | Integer                                               | false    | false     | 1000               | The maximum number of actions per elasticsearch bulk request. Use -1 to disable it.                                                                                                                                                                                                                                                                                            |
| `bulkSizeInMb`                 | Integer                                               | false    | false     | 5                  | The maximum size in megabytes of elasticsearch bulk requests. Use -1 to disable it.                                                                                                                                                                                                                                                                                            |
| `bulkConcurrentRequests`       | Integer                                               | false    | false     | 0                  | The maximum number of in flight elasticsearch bulk requests. The default 0 allows the execution of a single request. A value of 1 means 1 concurrent request is allowed to be executed while accumulating new bulk requests.                                                                                                                                                   |
| `bulkFlushIntervalInMs`        | Long                                                  | false    | false     | 1000               | The maximum period of time to wait for flushing pending writes when bulk writes are enabled. -1 or zero means the scheduled flushing is disabled.                                                                                                                                                                                                                              |
| `compressionEnabled`           | Boolean                                               | false    | false     | false              | Enable elasticsearch request compression.                                                                                                                                                                                                                                                                                                                                      |
| `connectTimeoutInMs`           | Integer                                               | false    | false     | 5000               | The elasticsearch client connection timeout in milliseconds.                                                                                                                                                                                                                                                                                                                   |
| `connectionRequestTimeoutInMs` | Integer                                               | false    | false     | 1000               | The time in milliseconds for getting a connection from the elasticsearch connection pool.                                                                                                                                                                                                                                                                                      |
| `connectionIdleTimeoutInMs`    | Integer                                               | false    | false     | 5                  | Idle connection timeout to prevent a read timeout.                                                                                                                                                                                                                                                                                                                             |
| `keyIgnore`                    | Boolean                                               | false    | false     | true               | Whether to ignore the record key to build the Elasticsearch document `_id`. If primaryFields is defined, the connector extract the primary fields from the payload to build the document `_id` If no primaryFields are provided, elasticsearch auto generates a random document `_id`.                                                                                         |
| `primaryFields`                | String                                                | false    | false     | "id"               | The comma separated ordered list of field names used to build the Elasticsearch document `_id` from the record value. If this list is a singleton, the field is converted as a string. If this list has 2 or more fields, the generated `_id` is a string representation of a JSON array of the field values.                                                                  |
| `nullValueAction`              | enum (IGNORE,DELETE,FAIL)                             | false    | false     | IGNORE             | How to handle records with null values, possible options are IGNORE, DELETE or FAIL. Default is IGNORE the message.                                                                                                                                                                                                                                                            |
| `malformedDocAction`           | enum (IGNORE,WARN,FAIL)                               | false    | false     | FAIL               | How to handle elasticsearch rejected documents due to some malformation. Possible options are IGNORE, DELETE or FAIL. Default is FAIL the Elasticsearch document.                                                                                                                                                                                                              |
| `stripNulls`                   | Boolean                                               | false    | false     | true               | If stripNulls is false, elasticsearch \_source includes 'null' for empty fields (for example `{"foo": null}`), otherwise null fields are stripped.                                                                                                                                                                                                                             |
| `socketTimeoutInMs`            | Integer                                               | false    | false     | 60000              | The socket timeout in milliseconds waiting to read the elasticsearch response.                                                                                                                                                                                                                                                                                                 |
| `typeName`                     | String                                                | false    | false     | "\_doc"            | The type name to which the connector writes messages to. <br /><br /> The value should be set explicitly to a valid type name other than "\_doc" for Elasticsearch version before 6.2, and left to default otherwise.                                                                                                                                                          |
| `indexNumberOfShards`          | int                                                   | false    | false     | 1                  | The number of shards of the index.                                                                                                                                                                                                                                                                                                                                             |
| `indexNumberOfReplicas`        | int                                                   | false    | false     | 1                  | The number of replicas of the index.                                                                                                                                                                                                                                                                                                                                           |
| `username`                     | String                                                | false    | true      | " " (empty string) | The username used by the connector to connect to the elastic search cluster. <br /><br />If `username` is set, then `password` should also be provided.                                                                                                                                                                                                                        |
| `password`                     | String                                                | false    | true      | " " (empty string) | The password used by the connector to connect to the elastic search cluster. <br /><br />If `username` is set, then `password` should also be provided.                                                                                                                                                                                                                        |
| `ssl`                          | ElasticSearchSslConfig                                | false    | false     |                    | Configuration for TLS encrypted communication                                                                                                                                                                                                                                                                                                                                  |
| `compatibilityMode`            | enum (AUTO,ELASTICSEARCH,ELASTICSEARCH\_7,OPENSEARCH) | false    | false     | AUTO               | Specify compatibility mode with the ElasticSearch cluster. `AUTO` value will try to auto detect the correct compatibility mode to use. Use `ELASTICSEARCH_7` if the target cluster is running ElasticSearch 7 or prior. Use `ELASTICSEARCH` if the target cluster is running ElasticSearch 8 or higher. Use `OPENSEARCH` if the target cluster is running OpenSearch.          |
| `token`                        | String                                                | false    | true      | " " (empty string) | The token used by the connector to connect to the ElasticSearch cluster. Only one between basic/token/apiKey authentication mode must be configured.                                                                                                                                                                                                                           |
| `apiKey`                       | String                                                | false    | true      | " " (empty string) | The apiKey used by the connector to connect to the ElasticSearch cluster. Only one between basic/token/apiKey authentication mode must be configured.                                                                                                                                                                                                                          |
| `canonicalKeyFields`           | Boolean                                               | false    | false     | false              | Whether to sort the key fields for JSON and Avro or not. If it is set to `true` and the record key schema is `JSON` or `AVRO`, the serialized object does not consider the order of properties.                                                                                                                                                                                |
| `stripNonPrintableCharacters`  | Boolean                                               | false    | false     | true               | Whether to remove all non-printable characters from the document or not. If it is set to true, all non-printable characters are removed from the document.                                                                                                                                                                                                                     |
| `idHashingAlgorithm`           | enum(NONE,SHA256,SHA512)                              | false    | false     | NONE               | Hashing algorithm to use for the document id. This is useful in order to be compliant with the ElasticSearch \_id hard limit of 512 bytes.                                                                                                                                                                                                                                     |
| `conditionalIdHashing`         | Boolean                                               | false    | false     | false              | This option only works if idHashingAlgorithm is set. If enabled, the hashing is performed only if the id is greater than 512 bytes otherwise the hashing is performed on each document in any case.                                                                                                                                                                            |
| `copyKeyFields`                | Boolean                                               | false    | false     | false              | If the message key schema is AVRO or JSON, the message key fields are copied into the ElasticSearch document.                                                                                                                                                                                                                                                                  |


# File source
Source: https://docs.streamnative.io/connect/connectors/file-source/current/file-source

The File source connector pulls messages from files in directories and persists the messages to Pulsar topics.

The File source connector pulls messages from files in directories and persists the messages to Pulsar topics.

# Configuration

The configuration of the File source connector has the following properties.

## Property

| Name                | Type    | Required | Default           | Description                                                                                                                                                                                                                                              |
| ------------------- | ------- | -------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `inputDirectory`    | String  | true     | No default value  | The input directory to pull files.                                                                                                                                                                                                                       |
| `recurse`           | Boolean | false    | true              | Whether to pull files from subdirectory or not.                                                                                                                                                                                                          |
| `keepFile`          | Boolean | false    | false             | If set to true, the file is not deleted after it is processed, which means the file can be picked up continually.                                                                                                                                        |
| `fileFilter`        | String  | false    | \[^\\.].\*        | The file whose name matches the given regular expression is picked up.                                                                                                                                                                                   |
| `pathFilter`        | String  | false    | NULL              | If `recurse` is set to true, the subdirectory whose path matches the given regular expression is scanned.                                                                                                                                                |
| `minimumFileAge`    | Integer | false    | 0                 | The minimum age that a file can be processed. <br /><br />Any file younger than `minimumFileAge` (according to the last modification date) is ignored.                                                                                                   |
| `maximumFileAge`    | Long    | false    | Long.MAX\_VALUE   | The maximum age that a file can be processed. <br /><br />Any file older than `maximumFileAge` (according to last modification date) is ignored.                                                                                                         |
| `minimumSize`       | Integer | false    | 1                 | The minimum size (in bytes) that a file can be processed.                                                                                                                                                                                                |
| `maximumSize`       | Double  | false    | Double.MAX\_VALUE | The maximum size (in bytes) that a file can be processed.                                                                                                                                                                                                |
| `ignoreHiddenFiles` | Boolean | false    | true              | Whether the hidden files should be ignored or not.                                                                                                                                                                                                       |
| `pollingInterval`   | Long    | false    | 10000L            | Indicates how long to wait before performing a directory listing.                                                                                                                                                                                        |
| `numWorkers`        | Integer | false    | 1                 | The number of worker threads that process files.<br /><br /> This allows you to process a larger number of files concurrently. <br /><br />However, setting this to a value greater than 1 makes the data from multiple files mixed in the target topic. |

## Example

Before using the File source connector, you need to create a configuration file through one of the following methods.

* JSON

  ```json theme={null}
  {
      "inputDirectory": "/Users/david",
      "recurse": true,
      "keepFile": true,
      "fileFilter": "[^\\.].*",
      "pathFilter": "*",
      "minimumFileAge": 0,
      "maximumFileAge": 9999999999,
      "minimumSize": 1,
      "maximumSize": 5000000,
      "ignoreHiddenFiles": true,
      "pollingInterval": 5000,
      "numWorkers": 1
  }
  ```

* YAML

  ```yaml theme={null}
  configs:
      inputDirectory: "/Users/david"
      recurse: true
      keepFile: true
      fileFilter: "[^\\.].*"
      pathFilter: "*"
      minimumFileAge: 0
      maximumFileAge: 9999999999
      minimumSize: 1
      maximumSize: 5000000
      ignoreHiddenFiles: true
      pollingInterval: 5000
      numWorkers: 1
  ```

# Usage

Here is an example of using the File source connecter.

1. Pull a Pulsar image.

   ```bash theme={null}
   $ docker pull apachepulsar/pulsar:{version}
   ```

2. Start Pulsar standalone.

   ```bash theme={null}
   $ docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-standalone apachepulsar/pulsar:{version} bin/pulsar standalone
   ```

3. Create a configuration file *file-connector.yaml*.

   ```yaml theme={null}
   configs:
       inputDirectory: "/opt"
   ```

4. Copy the configuration file *file-connector.yaml* to the container.

   ```bash theme={null}
   $ docker cp connectors/file-connector.yaml pulsar-standalone:/pulsar/
   ```

5. Download the File source connector.

   ```bash theme={null}
   $ curl -O https://mirrors.tuna.tsinghua.edu.cn/apache/pulsar/pulsar-{version}/connectors/pulsar-io-file-{version}.nar
   ```

6. Start the File source connector.

   ```bash theme={null}
   $ docker exec -it pulsar-standalone /bin/bash

   $ ./bin/pulsar-admin sources localrun \
   --archive /pulsar/pulsar-io-file-{version}.nar \
   --name file-test \
   --destination-topic-name  pulsar-file-test \
   --source-config-file /pulsar/file-connector.yaml
   ```

7. Start a consumer.

   ```bash theme={null}
   ./bin/pulsar-client consume -s file-test -n 0 pulsar-file-test
   ```

8. Write the message to the file *test.txt*.

   ```bash theme={null}
   echo "hello world!" > /opt/test.txt
   ```

   The following information appears on the consumer terminal window.

   ```bash theme={null}
   ----- got message -----
   hello world!
   ```


# Flume sink
Source: https://docs.streamnative.io/connect/connectors/flume-sink/current/flume-sink

The Flume sink connector pulls messages from Pulsar topics to logs

# Configuration

The configuration of the Flume sink connector has the following properties.

## Property

| Name           | Type    | Required | Default           | Description                                         |
| -------------- | ------- | -------- | ----------------- | --------------------------------------------------- |
| `name`         | String  | true     | "" (empty string) | The name of the agent.                              |
| `confFile`     | String  | true     | "" (empty string) | The configuration file.                             |
| `noReloadConf` | Boolean | false    | false             | Whether to reload configuration file if changed.    |
| `zkConnString` | String  | true     | "" (empty string) | The ZooKeeper connection.                           |
| `zkBasePath`   | String  | true     | "" (empty string) | The base path in ZooKeeper for agent configuration. |

## Example

Before using the Flume sink connector, you need to create a configuration file through one of the following methods.

> For more information about the `sink.conf` in the example below, see [here](https://github.com/apache/pulsar/blob/master/pulsar-io/flume/src/main/resources/flume/sink.conf).

* JSON

  ```json theme={null}
  {
      "name": "a1",
      "confFile": "sink.conf",
      "noReloadConf": "false",
      "zkConnString": "",
      "zkBasePath": ""
  }
  ```

* YAML

  ```yaml theme={null}
  configs:
      name: a1
      confFile: sink.conf
      noReloadConf: false
      zkConnString: ""
      zkBasePath: ""
  ```


# Flume source
Source: https://docs.streamnative.io/connect/connectors/flume-source/current/flume-source

The Flume source connector pulls messages from logs to Pulsar topics.

The Flume source connector pulls messages from logs to Pulsar topics.

# Configuration

The configuration of the Flume source connector has the following properties.

## Property

| Name           | Type    | Required | Default           | Description                                         |
| -------------- | ------- | -------- | ----------------- | --------------------------------------------------- |
| `name`         | String  | true     | "" (empty string) | The name of the agent.                              |
| `confFile`     | String  | true     | "" (empty string) | The configuration file.                             |
| `noReloadConf` | Boolean | false    | false             | Whether to reload configuration file if changed.    |
| `zkConnString` | String  | true     | "" (empty string) | The ZooKeeper connection.                           |
| `zkBasePath`   | String  | true     | "" (empty string) | The base path in ZooKeeper for agent configuration. |

## Example

Before using the Flume source connector, you need to create a configuration file through one of the following methods.

> For more information about the `source.conf` in the example below, see [here](https://github.com/apache/pulsar/blob/master/pulsar-io/flume/src/main/resources/flume/source.conf).

* JSON

  ```json theme={null}
  {
      "name": "a1",
      "confFile": "source.conf",
      "noReloadConf": "false",
      "zkConnString": "",
      "zkBasePath": ""
  }
  ```

* YAML

  ```yaml theme={null}
  configs:
      name: a1
      confFile: source.conf
      noReloadConf: false
      zkConnString: ""
      zkBasePath: ""
  ```


# Google bigquery sink
Source: https://docs.streamnative.io/connect/connectors/google-bigquery-sink/current/google-bigquery-sink

BigQuery Connector integrates Apache Pulsar with Google BigQuery.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

The [Google Cloud BigQuery](https://cloud.google.com/bigquery) sink connector pulls data from Pulsar topics and persists data to Google Cloud BigQuery tables.

<img alt="" />

## Quick start

> Data can only be synchronized to [Standard](https://cloud.google.com/bigquery/docs/tables-intro#standard_tables) BigQuery tables.
> [External](https://cloud.google.com/bigquery/docs/tables-intro#external_tables) tables and [View](https://cloud.google.com/bigquery/docs/tables-intro#views) are not supported.

### Prerequisites

The prerequisites for connecting an Google BigQuery sink connector to external systems include:

1. Create GoogleBigQuery, DataSet in Google Cloud.
2. Create the [Gcloud ServiceAccount](https://cloud.google.com/iam/docs/service-accounts-create) and create a public key certificate.
3. Create the [Gcloud Role](https://cloud.google.com/iam/docs/creating-custom-roles), ensure the Google Cloud role have the following permissions to the Google [BigQuery API](https://cloud.google.com/bigquery/docs/access-control):

```text theme={null}
- bigquery.tables.create
- bigquery.tables.get
- bigquery.tables.getData
- bigquery.tables.list
- bigquery.tables.update
- bigquery.tables.updateData
```

4. Grant the service account the above role permissions.

### 1. Create a connector

The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector,
you need to replace `--sink-type bigquery` with `--archive /path/to/pulsar-io-bigquery.nar`. You can find the button to download the `nar` package at the beginning of the document.

<Note title="For StreamNative Cloud User">
  If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first.
</Note>

```bash theme={null}
pulsarctl sinks create \
  --sink-type bigquery \
  --name bigquery-sink \
  --tenant public \
  --namespace default \
  --inputs "Your topic name" \
  --parallelism 1 \
  --sink-config \
  '{
    "projectId": "Your BigQuery project Id", 
    "datasetName": "Your Bigquery DataSet name",
    "tableName": "The name of the table you want to write data to is automatically created by default",
    "credentialJsonString": "Public key certificate you created above"
  }'
```

The `--sink-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.
If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference.

<Note title="Note">
  You can also choose to use a variety of other tools to create a connector:

  * [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [RestAPI](https://pulsar.apache.org/sink-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document.
</Note>

### 2. Send messages to the topic

<Note title="Note">
  If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information.
</Note>

```java theme={null}
@Data
@ToString
public class TestMessage {
    private String testString;
    private int testInt;
 
    public static void main(String[] args) {
        PulsarClient client = PulsarClient.builder()
        .serviceUrl("{{Your Pulsar URL}}")
        .build();

        Producer<TestMessage> producer = client.newProducer(Schema.AVRO(TestMessage.class))
            .topic("{{Your topic name}}")
            .create();

        TestMessage testMessage = new TestMessage();
        testMessage.setTestString("test string");
        testMessage.setTestInt(123);
        MessageId msgID = producer.send(testMessage);
        System.out.println("Publish " + testMessage + " and message ID " + msgID);
        
        producer.flush();
        producer.close();
        client.close();  
    }
}

```

### 3. Show data on Google BigQuery

This connector will automatically create the table structure according to the schema. You can use sql to query the data in the console.

```sql theme={null}
SELECT * FROM `{{Your project id}}.`{{Your dataset name}}`.{{Your table name}}`

+-----------------+-----------------+--------------------------------+----------------------------+-------------+---------+
| __meessage_id__ | __sequence_id__ |         __event_time__         |     __producer_name__      | testString  | testInt |
+-----------------+-----------------+--------------------------------+----------------------------+-------------+---------+
| 9:20:-1         |               0 | 2023-09-14 14:05:29.657000 UTC | test-bigquery-produce-name | test string |     123 |
+-----------------+-----------------+--------------------------------+----------------------------+-------------+---------+

```

## Configuration Properties

Before using the Google Cloud BigQuery sink connector, you need to configure it. This table outlines the properties and the descriptions.

| Name                          | Type    | Required | Sensitive | Default           | Description                                                                                                                                                                                                                                                                                                                                                                                                         |
| ----------------------------- | ------- | -------- | --------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `projectId`                   | String  | Yes      | false     | "" (empty string) | The Google BigQuery project ID.                                                                                                                                                                                                                                                                                                                                                                                     |
| `datasetName`                 | String  | Yes      | false     | "" (empty string) | The Google BigQuery dataset name.                                                                                                                                                                                                                                                                                                                                                                                   |
| `tableName`                   | String  | Yes      | false     | "" (empty string) | The Google BigQuery table name.                                                                                                                                                                                                                                                                                                                                                                                     |
| `credentialJsonString`        | String  | Yes      | true      | "" (empty string) | The authentication JSON key. Set the environment variable `GOOGLE_APPLICATION_CREDENTIALS` to the path of the JSON file that contains your service account key when the `credentialJsonString` is set to an empty string. For details, see the [Google documentation](https://cloud.google.com/bigquery/docs/quickstarts/quickstart-client-libraries#before-you-begin).                                             |
| `visibleModel`                | String  | No       | false     | "Committed"       | The mode that controls when data written to the stream becomes visible in BigQuery for reading. For details, see the [Google documentation](https://cloud.google.com/bigquery/docs/write-api#application-created_streams). Available options are `Committed` and `Pending`.                                                                                                                                         |
| `pendingMaxSize`              | int     | No       | false     | 10000             | The maximum number of messages waiting to be committed in `Pending` mode.                                                                                                                                                                                                                                                                                                                                           |
| `batchMaxSize`                | int     | No       | false     | 20                | The maximum number of batch messages. The actual batch bytes size cannot exceed 10 MB. If it does, the batch will be flushed first. [https://cloud.google.com/bigquery/quotas](https://cloud.google.com/bigquery/quotas)                                                                                                                                                                                            |
| `batchMaxTime`                | long    | No       | false     | 5000              | The maximum batch waiting time (in units of milliseconds).                                                                                                                                                                                                                                                                                                                                                          |
| `batchFlushIntervalTime`      | long    | No       | false     | 2000              | The batch flush interval (in units of milliseconds).                                                                                                                                                                                                                                                                                                                                                                |
| `failedMaxRetryNum`           | int     | No       | false     | 20                | The maximum retries when appending fails. By default, it sets 2 seconds for each retry.                                                                                                                                                                                                                                                                                                                             |
| `autoCreateTable`             | boolean | No       | false     | true              | Automatically create a table if no table is available.                                                                                                                                                                                                                                                                                                                                                              |
| `autoUpdateTable`             | boolean | No       | false     | true              | Automatically update the table schema if the BigQuery table schema is incompatible with the Pulsar schema.                                                                                                                                                                                                                                                                                                          |
| `partitionedTables`           | boolean | No       | false     | true              | Create a partitioned table when the table is automatically created. It will use the `__event_time__` as the partition key.                                                                                                                                                                                                                                                                                          |
| `partitionedTableIntervalDay` | int     | No       | false     | 7                 | The number of days between partitioning of the partitioned table.                                                                                                                                                                                                                                                                                                                                                   |
| `clusteredTables`             | boolean | No       | false     | true              | Create a clustered table when the table is automatically created. It will use the `__message_id__` as the cluster key.                                                                                                                                                                                                                                                                                              |
| `defaultSystemField`          | String  | No       | false     | "" (empty string) | Create the system fields when the table is automatically created. You can use commas to separate multiple fields. The supported system fields are: `__schema_version__` , `__partition__` , `__event_time__`, `__publish_time__` , `__message_id__` , `__sequence_id__` , `__producer_name__` and `__properties__`. The `__properties__` will be a repeat struct on bigquery. key and value will as a string type.. |

## Advanced features

### Delivery guarantees

The Pulsar IO connector framework provides three [delivery guarantees](https://pulsar.apache.org/docs/next/functions-concepts#processing-guarantees-and-subscription-types): `at-most-once`, `at-least-once`, and `effectively-once`.

Currently, the Google Cloud BigQuery sink connector only provides the `at-least-once` delivery guarantee.

### Tables schema

The Google Cloud BigQuery sink connector supports automatically creating and updating a table’s schema based on the Pulsar topic schema. You can configure the following options:

```
autoCreataTables = true
autoUpdateSchema = true
```

If the Pulsar topic schema and BigQuery schema are different, the Google Cloud BigQuery sink connector updates schemas by merging them together.
The Google Cloud BigQuery sink connector supports mapping schema structures to the BigQuery [RECORD TYPE](https://cloud.google.com/bigquery/docs/nested-repeated#example_schema).

In addition, the Google Cloud BigQuery sink connector supports writing some Pulsar-specific fields, as shown below:

```
#
# optional: __schema_version__ , __partition__   , __event_time__    , __publish_time__  
#           __message_id__     , __sequence_id__ , __producer_name__ , __key__ , __properties__ 
#
defaultSystemField = __event_time__,__message_id__
```

<Note title="Note">
  The Google Cloud BigQuery sink connector does not delete any fields. If you change a field name in a Pulsar topic, the Google Cloud BigQuery sink connector will preserve both fields.
</Note>

This table lists the schema types that currently are supported to be converted.

| Schema           | Supported |
| ---------------- | --------- |
| AVRO             | Yes       |
| PRIMITIVE        | Yes       |
| PROTOBUF\_NATIVE | Yes       |
| PROTOBUF         | No        |
| JSON             | No        |
| KEY\_VALUE       | No        |

### Partitioned tables

<Note title="Note">
  This feature is only available when `autoCreateTable` is set to `true`. If you create a table manually, you need to manually specify the partition key.
</Note>

BigQuery supports [partitioned tables](https://cloud.google.com/bigquery/docs/partitioned-tables). Partitioned tables can improve query and control costs by reducing the data read from the table.
The Google Cloud BigQuery sink connector provides an option to create a partitioned table. The partitioned tables use the **event\_time** as the partition key.

```
partitioned-tables = true
```

### Clustered tables

<Note title="Note">
  This feature is only available when `autoCreateTable` is set to `true`. If you create a table manually, you need to manually specify the cluster key.
</Note>

[Clustered tables](https://cloud.google.com/bigquery/docs/clustered-tables) can improve the performance of certain queries, such as queries that use filter clauses and queries that aggregate data. The Google Cloud BigQuery sink connector provides an option to create a clustered table. The clustered tables use the **message\_id** as the cluster key.

```
clustered-tables = true
```

### Multiple tasks

You can leverage the Pulsar Functions scheduling mechanism to configure parallelism of the Google Cloud BigQuery sink connector. You can schedule
multiple sink instances to run on different Function worker nodes. These sink instances consume messages according to the configured subscription mode.

```
parallelism = 4
```

<Note title="Note">
  It is an effective way to increase parallelism when you encounter write bottlenecks. In addition, you need to pay attention to whether the write rate is greater than [BigQuery Rate Limits](https://cloud.google.com/bigquery/quotas#streaming_inserts)
</Note>

### Batch progress

To increase write throughput, the Google Cloud BigQuery sink connector supports configuring the batch size. You can set the batch size and latency using the following options.

```
batchMaxSize = 100
batchMaxTime = 4000
batchFlushIntervalTime = 2000
```


# Google bigquery source
Source: https://docs.streamnative.io/connect/connectors/google-bigquery-source/current/google-bigquery-source

BigQuery Connector integrates Apache Pulsar with Google BigQuery.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

The [Google Cloud BigQuery](https://cloud.google.com/bigquery) Source Connector feeds data from Google Cloud BigQuery tables and writes data to Pulsar topics.

<img alt="" />

## Quick start

### Prerequisites

The prerequisites for connecting an Google BigQuery source connector to external systems include:

1. Create GoogleBigQuery, DataSet and Table in Google Cloud. You can set the schema of the table, and this connector will convert the Avro schema to Pulsar.
2. Create the [Gcloud ServiceAccount](https://cloud.google.com/iam/docs/service-accounts-create) and create a public key certificate.
3. Create the [Gcloud Role](https://cloud.google.com/iam/docs/creating-custom-roles), ensure the Google Cloud role have the following permissions to the Google [BigQuery API](https://cloud.google.com/bigquery/docs/access-control):

```text theme={null}
- bigquery.readsessions.create
- bigquery.readsessions.getData
- bigquery.readsessions.update
- bigquery.jobs.create
- bigquery.tables.get
- bigquery.tables.getData
```

4. Grant the service account the above role permissions.

### 1. Write data to Google Bigquery

You can use SQL to insert some data to a table. For examples:

```sql theme={null}
INSERT INTO `{{Your dataset name}}.{{Your table name}}` (message, info)
VALUES
  ("message-1", "This is a message-1."),
  ("message-2", "This is a message-2."),
  ("message-3", "This is a message-3."),
  ("message-4", "This is a message-4."),
  ("message-5", "This is a message-5."),
  ("message-6", "This is a message-6."),
  ("message-7", "This is a message-7."),
  ("message-8", "This is a message-8."),
  ("message-9", "This is a message-9."),
  ("message-10", "This is a message-10.");
```

<Note title="Note">
  This connector will create a snapshot of BigQueryTable to synchronize data when it starts, so you must make sure that there is data in the table before starting the connector.

  In other words, it will only synchronize the data before the start-up, and once the data synchronization is complete, the current implementation will not discover new data to synchronize.
</Note>

### 2. Create a connector

The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector,
you need to replace `--source-type bigquery` with `--archive /path/to/pulsar-io-bigquery.nar`. You can find the button to download the `nar` package at the beginning of the document.

<Note title="For StreamNative Cloud User">
  If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first.
</Note>

```bash theme={null}
pulsarctl sources create \
  --source-type bigquery \
  --name bigquery-source \
  --tenant public \
  --namespace default \
  --destination-topic-name "Your topic name" \
  --parallelism 1 \
  --batch-source-config '{"discoveryTriggererClassName": "org.apache.pulsar.ecosystem.io.bigquery.source.BigQueryOnceTrigger"}' \
  --source-config \
  '{
    "projectId": "Your BigQuery project Id", 
    "datasetName": "Your Bigquery DataSet name",
    "tableName": "Your Bigquery Table name",
    "credentialJsonString": "Public key certificate you created above"
  }'
```

The `--source-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.
If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference.

<Note title="Note">
  You can also choose to use a variety of other tools to create a connector:

  * [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [RestAPI](https://pulsar.apache.org/source-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document.
</Note>

### 3. Show data by Pulsar Consumer

<Note title="Note">
  If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information.
</Note>

This connector will register the Google BigQuery table schema to pulsar. You can use `AUTO_CONSUMER` to consume the data. For example:

```java theme={null}
    public static void main(String[] args) {
        PulsarClient client = PulsarClient.builder()
            .serviceUrl("{{Your Pulsar URL}}")
            .build();

        Consumer<GenericRecord> consumer = client.newConsumer(Schema.AUTO_CONSUME())
                                                 .topic("{{The topic name that you specified when you created the connector}}")
                                                 .subscriptionName(subscription)
                                                 .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
                                                 .subscribe();

        for (int i = 0; i < 10; i++) {
            Message<GenericRecord> message = consumer.receive(10, TimeUnit.SECONDS);
            GenericRecord value = message.getValue();
            for (Field field : value.getFields()) {
                Object fieldValue = value.getField(field);
                System.out.print(field.getName() + ":" + fieldValue + " ");
            }
            System.out.println();
            consumer.acknowledge(message);
        }
        client.close();  
    }
    // output
    // message:message-1 info:This is a message-1.
    // message:message-2 info:This is a message-2.
    // message:message-3 info:This is a message-3.
    // message:message-4 info:This is a message-4.
    // message:message-5 info:This is a message-5.
    // message:message-6 info:This is a message-6.
    // message:message-7 info:This is a message-7.
    // message:message-8 info:This is a message-8.
    // message:message-9 info:This is a message-9.
    // message:message-10 info:This is a message-10.
```

## Configuration Properties

Before using the Google Cloud BigQuery source connector, you need to configure it. This table outlines the properties and the descriptions.

| Name                        | Type    | Required | Sensitive | Default           | Description                                                                                                                                                                                                                                                                                                                                                                     |   |
| --------------------------- | ------- | -------- | --------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | - |
| `projectId`                 | String  | Yes      | false     | "" (empty string) | The Google BigQuery project ID.                                                                                                                                                                                                                                                                                                                                                 |   |
| `datasetName`               | String  | Yes      | false     | "" (empty string) | The Google BigQuery dataset name.                                                                                                                                                                                                                                                                                                                                               |   |
| `tableName`                 | String  | Yes      | false     | "" (empty string) | The Google BigQuery table name.                                                                                                                                                                                                                                                                                                                                                 |   |
| `credentialJsonString`      | String  | No       | true      | "" (empty string) | The authentication JSON key. Set the environment variable `GOOGLE_APPLICATION_CREDENTIALS` to the path of the JSON file that contains your service account key when the `credentialJsonString` is set to an empty string. For details, see the [Google documentation](https://cloud.google.com/bigquery/docs/quickstarts/quickstart-client-libraries#before-you-begin).         |   |
| `maxParallelism`            | int     | No       | false     | 1                 | The maximum parallelism for reading. In fact, the number may be less if the BigQuery source connector deems the data small enough.                                                                                                                                                                                                                                              |   |
| `forceUpdate`               | Boolean | No       | false     | false             | "if forceUpdate=true,a new session will be created. The connector will transmit the data again.                                                                                                                                                                                                                                                                                 |   |
| `queueSize`                 | int     | No       | false     | 10000             | The buffer queue size of the source. It is used for storing records before they are sent to Pulsar topics. By default, it is set to `10000`.                                                                                                                                                                                                                                    |   |
| `sql`                       | String  | No       | false     | "" (empty string) | The SQL query on BigQuery. The computed result is saved in a temporary table. The temporary table has a configurable expiration time, and the BigQuery source connector automatically deletes the temporary table when the data is transferred completely. The `projectId` and `datasetName` gets values from the configuration file, and the `tableName` is generated by UUID. |   |
| `expirationTimeInMinutes`   | int     | No       | false     | 1440              | The expiration time in minutes until the table is expired and auto-deleted.                                                                                                                                                                                                                                                                                                     |   |
| `selectedFields`            | String  | No       | false     | "" (empty string) | Names of the fields in the table that should be read.                                                                                                                                                                                                                                                                                                                           |   |
| `filters`                   | String  | No       | false     | "" (empty string) | A list of clauses that can filter the result of the table.                                                                                                                                                                                                                                                                                                                      |   |
| `checkpointIntervalSeconds` | int     | No       | false     | 60                | The checkpoint interval (in units of seconds). By default, it is set to 60s.                                                                                                                                                                                                                                                                                                    |   |


# Google cloud storage sink
Source: https://docs.streamnative.io/connect/connectors/google-cloud-storage-sink/current/google-cloud-storage-sink

Cloud Storage Connector integrates Apache Pulsar with cloud storage.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

The [Google Cloud Storage](https://cloud.google.com/storage/docs) sink connector pulls data from Pulsar topics and persists data to Google Cloud Storage buckets.

<img alt="" />

## Quick start

### Prerequisites

The prerequisites for connecting an Google Cloud Storage sink connector to external systems include:

1. Create Cloud Storage buckets in Google Cloud.
2. Create the [Google cloud ServiceAccount](https://cloud.google.com/iam/docs/service-accounts-create) and create a public key certificate.
3. Create the [Google cloud Role](https://cloud.google.com/iam/docs/creating-custom-roles), ensure the Google Cloud role have the following permissions:

```text theme={null}
- storage.buckets.get
- storage.buckets.list
- storage.objects.create
```

4. Grant the `ServiceAccount` the above `Role`.

### 1. Create a connector

The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector,
you need to replace `--sink-type cloud-storage-gcloud` with `--archive /path/to/pulsar-io-cloud-storage.nar`. You can find the button to download the `nar` package at the beginning of the document.

<Note title="For StreamNative Cloud User">
  If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first.
</Note>

```bash theme={null}
pulsarctl sinks create \
  --sink-type cloud-storage-gcloud \
  --name gcloud-storage-sink \
  --tenant public \
  --namespace default \
  --inputs "Your topic name" \
  --parallelism 1 \
  --sink-config \
  '{
    "gcsServiceAccountKeyFileContent": "Public key certificate you created above", 
    "provider": "google-cloud-storage",
    "bucket": "Your bucket name",
    "formatType": "json",
    "partitionerType": "PARTITION"
  }'
```

The `--sink-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.
If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference.

<Note title="Note">
  You can also choose to use a variety of other tools to create a connector:

  * [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [RestAPI](https://pulsar.apache.org/sink-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document.
</Note>

### 2. Send messages to the topic

<Note title="Note">
  If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information.
</Note>

```java theme={null}
    public static void main(String[] args) throws Exception {
        PulsarClient client = PulsarClient.builder()
                             .serviceUrl("{{Your Pulsar URL}}")
                             .build();


        Producer<String> producer = client.newProducer(Schema.STRING)
                .topic("{{Your topic name}}")
                .create();

        for (int i = 0; i < 10; i++) {
            // JSON string containing a single character
            String message = "{\"test-message\": \"test-value\"}";
            producer.send(message);
        }

        producer.close();
        client.close();
    }
```

### 3. Display data on Google Cloud Storage console

You can see the object at public/default/`{{Your topic name}}`-partition-0/xxxx.json on the Google Cloud Storage console. Download and open it, the content is:

```text theme={null}
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
`{"test-message":"test-value"}`
```

## Configuration Properties

Before using the Google Cloud Storage sink connector, you need to configure it. This table outlines the properties and the descriptions.

| Name                              | Type    | Required | Sensitive | Default      | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| --------------------------------- | ------- | -------- | --------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `provider`                        | String  | True     | false     | null         | The Cloud Storage type, google cloud storage only supports the `google-cloud-storage` provider.                                                                                                                                                                                                                                                                                                                                                                                                    |
| `bucket`                          | String  | True     | false     | null         | The Cloud Storage bucket.                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `formatType`                      | String  | True     | false     | "json"       | The data format type. Available options are `json`, `avro`, `bytes`, or `parquet`. By default, it is set to `json`.                                                                                                                                                                                                                                                                                                                                                                                |
| `partitioner`                     | String  | False    | false     | null         | The partitioner for partitioning the resulting files. Available options are `topic`, `time` or `legacy`. By default, it's set to `legacy`. Please see [Partitioner](#partitioner) for more details.                                                                                                                                                                                                                                                                                                |
| `partitionerType`                 | String  | False    | false     | null         | The legacy partitioning type. It can be configured by topic partitions or by time. By default, the partition type is configured by topic partitions. It only works when the partitioner is set to `legacy`.                                                                                                                                                                                                                                                                                        |
| `gcsServiceAccountKeyFileContent` | String  | False    | true      | ""           | The contents of the JSON service key file. If empty, credentials are read from `gcsServiceAccountKeyFilePath` file.                                                                                                                                                                                                                                                                                                                                                                                |
| `gcsServiceAccountKeyFilePath`    | String  | False    | true      | ""           | Path to the GCS credentials file. If empty, the credentials file will be read from the `GOOGLE_APPLICATION_CREDENTIALS` environment variable.                                                                                                                                                                                                                                                                                                                                                      |
| `timePartitionPattern`            | String  | False    | false     | "yyyy-MM-dd" | The format pattern of the time-based partitioning. For details, refer to the Java date and time format.                                                                                                                                                                                                                                                                                                                                                                                            |
| `timePartitionDuration`           | String  | False    | false     | "86400000"   | The time interval for time-based partitioning. Support formatted interval string, such as `30d`, `24h`, `30m`, `10s`, and also support number in milliseconds precision, such as `86400000` refers to `24h` or `1d`.                                                                                                                                                                                                                                                                               |
| `pathPrefix`                      | String  | False    | false     | false        | If it is set, the output files are stored in a folder under the given bucket path. The `pathPrefix` must be in the format of `xx/xxx/`.                                                                                                                                                                                                                                                                                                                                                            |
| `partitionerWithTopicName`        | Boolean | False    | false     | true         | Indicates whether to include the topic name in the file path. Default is true. If not included, the path like: `pathPrefix/24.45.0.json`                                                                                                                                                                                                                                                                                                                                                           |
| `partitionerUseIndexAsOffset`     | Boolean | False    | false     | false        | Whether to use the Pulsar's message index as offset or the record sequence. It's recommended if the incoming messages may be batched. The brokers may or not expose the index metadata and, if it's not present on the record, the sequence will be used. See [PIP-70](https://github.com/apache/pulsar/wiki/PIP-70%3A-Introduce-lightweight-broker-entry-metadata) for more details.                                                                                                              |
| `withTopicPartitionNumber`        | Boolean | False    | false     | true         | When it is set to `true`, include the topic partition number to the object path.                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `sliceTopicPartitionPath`         | Boolean | False    | false     | false        | When it is set to `true`, split the partitioned topic name into separate folders in the bucket path.                                                                                                                                                                                                                                                                                                                                                                                               |
| `batchSize`                       | int     | False    | false     | 10           | The number of records submitted in batch.                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `batchTimeMs`                     | long    | False    | false     | 1000         | The interval for batch submission.                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `maxBatchBytes`                   | long    | False    | false     | 10000000     | The maximum number of bytes in a batch.                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `batchModel`                      | Enum    | False    | false     | BLEND        | Determines how records are batched. Options: `BLEND`, `PARTITIONED`. The BLEND which combines all topic records into a single batch, optimizing for throughput, and PARTITIONED which batches records separately for each topic, maintaining topic-level separation. Note: When set to PARTITIONED, the connector will cache data up to the size of the number of subscribed topics multiplied by maxBatchBytes. This means you need to anticipate the connector's memory requirements in advance. |
| `skipFailedMessages`              | Boolean | False    | false     | false        | Configure whether to skip a message which it fails to be processed. If it is set to `true`, the connector will skip the failed messages by `ack` it. Otherwise, the connector will `fail` the message.                                                                                                                                                                                                                                                                                             |
| `withMetadata`                    | Boolean | False    | false     | false        | Save message attributes to metadata.                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `useHumanReadableMessageId`       | Boolean | False    | false     | false        | Use a human-readable format string for messageId in message metadata. The messageId is in a format like `ledgerId:entryId:partitionIndex:batchIndex`. Otherwise, the messageId is a Hex-encoded string.                                                                                                                                                                                                                                                                                            |
| `useHumanReadableSchemaVersion`   | Boolean | False    | false     | false        | Use a human-readable format string for the schema version in the message metadata. If it is set to `true`, the schema version is in plain string format. Otherwise, the schema version is in hex-encoded string format.                                                                                                                                                                                                                                                                            |
| `includeTopicToMetadata`          | Boolean | False    | false     | false        | Include the topic name to the metadata.                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `includePublishTimeToMetadata`    | Boolean | False    | false     | false        | Include the message publish time to the metadata as a timestamp.                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `includeMessageKeyToMetadata`     | Boolean | False    | false     | false        | Include the message key to the metadata as a string.                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| `avroCodec`                       | String  | False    | false     | snappy       | Compression codec used when formatType=`avro`. Available compression types are: none (no compression), deflate, bzip2, xz, zstandard, snappy.                                                                                                                                                                                                                                                                                                                                                      |
| `parquetCodec`                    | String  | False    | false     | gzip         | Compression codec used when formatType=`parquet`. Available compression types are: none (no compression), snappy, gzip, lzo, brotli, lz4, zstd.                                                                                                                                                                                                                                                                                                                                                    |
| `jsonAllowNaN`                    | Boolean | False    | false     | false        | Recognize 'NaN', 'INF', '-INF' as legal floating number values when formatType=`json`. Since JSON specification does not allow such values this is a non-standard feature and disabled by default.                                                                                                                                                                                                                                                                                                 |
| `bytesFormatTypeSeparator`        | String  | False    | false     | "0x10"       | It is inserted between records for the `formatType` of bytes. By default, it is set to '0x10'. An input record that contains the line separator looks like multiple records in the output object.                                                                                                                                                                                                                                                                                                  |

## Advanced features

### Data format types

Cloud Storage Sink Connector provides multiple output format options, including JSON, Avro, Bytes, or Parquet. The default format is JSON.
With current implementation, there are some limitations for different formats:

This table lists the Pulsar Schema types supported by the writers.

| Pulsar Schema  | Writer: Avro | Writer: JSON | Writer: Parquet | Writer: Bytes |
| -------------- | ------------ | ------------ | --------------- | ------------- |
| Primitive      | ✗            | ✔ \*         | ✗               | ✔             |
| Avro           | ✔            | ✔            | ✔               | ✔             |
| Json           | ✔            | ✔            | ✔               | ✔             |
| Protobuf \*\*  | ✔            | ✔            | ✔               | ✔             |
| ProtobufNative | ✔ \*\*\*     | ✗            | ✔               | ✔             |

> \*: The JSON writer will try to convert the data with a `String` or `Bytes` schema to JSON-format data if convertable.
>
> \*\*: The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion.
>
> \*\*\*: The ProtobufNative record holds the Protobuf descriptor and the message. When writing to Avro format, the connector uses [avro-protobuf](https://github.com/apache/avro/tree/master/lang/java/protobuf) to do the conversion.

This table lists the support of `withMetadata` configurations for different writer formats:

| Writer Format | `withMetadata` |
| ------------- | -------------- |
| Avro          | ✔              |
| JSON          | ✔              |
| Parquet       | ✔ \*           |
| Bytes         | ✗              |

> \*: When using `Parquet` with `PROTOBUF_NATIVE` format, the connector will write the messages with `DynamicMessage` format. When `withMetadata` is set to `true`, the connector will add `__message_metadata__` to the messages with `PulsarIOCSCProtobufMessageMetadata` format.
>
> For example, if a message `User` has the following schema:
>
> ```protobuf theme={null}
> syntax = "proto3";
> message User {
>  string name = 1;
>  int32 age = 2;
> }
> ```
>
> When `withMetadata` is set to `true`, the connector will write the message `DynamicMessage` with the following schema:
>
> ```protobuf theme={null}
> syntax = "proto3";
> message PulsarIOCSCProtobufMessageMetadata {
>  map<string, string> properties = 1;
>  string schema_version = 2;
>  string message_id = 3;
> }
> message User {
>  string name = 1;
>  int32 age = 2;
>  PulsarIOCSCProtobufMessageMetadata __message_metadata__ = 3;
> }
> ```

### Dead-letter topics

To use a dead-letter topic, you need to set `skipFailedMessages` to `false`, and set `--max-redeliver-count` and `--dead-letter-topic` when submit the connector with the `pulsar-admin` CLI tool. For more info about dead-letter topics, see the [Pulsar documentation](https://pulsar.apache.org/docs/en/concepts-messaging/#dead-letter-topic).
If a message fails to be sent to the Cloud Storage and there is a dead-letter topic, the connector will send the message to the dead-letter topic.

### Sink flushing only after batchTimeMs elapses

There is a scenario where the sink is only flushing whenever the `batchTimeMs` has elapsed, even though there are many messages waiting to be processed.
The reason for this is that the sink will only acknowledge messages after they are flushed to cloud storage but the broker stops sending messages when it reaches a certain limit of unacknowledged messages.
If this limit is lower or close to `batchSize`, the sink never receives enough messages to trigger a flush based on the amount of messages.
In this case please ensure the `maxUnackedMessagesPerConsumer` set in the broker configuration is sufficiently larger than the `batchSize` setting of the sink.

### Partitioner Type

There are two types of partitioner:

* **PARTITION**: This is the default partitioning method based on Pulsar partitions. In other words, data is
  partitioned according to the pre-existing partitions in Pulsar topics. For instance, a message for the
  topic `public/default/my-topic-partition-0` would be directed to the
  file `public/default/my-topic-partition-0/xxx.json`, where `xxx` signifies the earliest messageId(Format: `ledgerId.entryId.batchIndex`)/offset(Enable config: `partitionerUseIndexAsOffset`) in this file.

* **TIME**: Data is partitioned according to the time it was flushed. Using the previous message as an
  example, if it was received on 2023-12-20, it would be directed
  to `public/default/my-topic-partition-0/2023-12-20/xxx.json`, where `xxx` also denotes the earliest messageId(Format: `ledgerId.entryId.batchIndex`)/offset(Enable config: `partitionerUseIndexAsOffset`) in this file.


# Google pubsub sink
Source: https://docs.streamnative.io/connect/connectors/google-pubsub-sink/current/google-pubsub-sink

The Google Pub/Sub sink connector is used to write messages from Apache Pulsar topics to Google Cloud Pub/Sub.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

The [Google Cloud PubSub](https://cloud.google.com/pubsub) sink connector pulls data from Pulsar topics and persists data to Google Cloud PubSub tables.

<img alt="" />

## Quick start

### Prerequisites

The prerequisites for connecting an Google PubSub sink connector to external systems include:

1. Create Google PubSub Topic in Google Cloud.
2. Create the [Gcloud ServiceAccount](https://cloud.google.com/iam/docs/service-accounts-create) and create a public key certificate.
3. Create the [Gcloud Role](https://cloud.google.com/iam/docs/creating-custom-roles), ensure the Google Cloud role have the following permissions:

```text theme={null}
- pubsub.topics.create
- pubsub.topics.get
- pubsub.topics.publish
```

4. Grant the service account the above role permissions.

### 1. Create a connector

The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector,
you need to replace `--sink-type google-pubsub` with `--archive /path/to/pulsar-io-google-pubsub.nar`. You can find the button to download the `nar` package at the beginning of the document.

<Note title="For StreamNative Cloud User">
  If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first.
</Note>

```bash theme={null}
pulsarctl sinks create \
  --sink-type google-pubsub \
  --name pubsub-sink \
  --tenant public \
  --namespace default \
  --inputs "Your topic name" \
  --parallelism 1 \
  --sink-config \
  '{
    "pubsubProjectId": "Your google pubsub project Id", 
    "pubsubTopicId": "Your google pubsub Topic name",
    "pubsubCredential": "The escaped and compressed public key certificate you created above"
  }'
```

The `--sink-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.
If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference.

<Note title="Note">
  You can also choose to use a variety of other tools to create a connector:

  * [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [RestAPI](https://pulsar.apache.org/sink-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document.
</Note>

### 2. Send messages to the topic

<Note title="Note">
  If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information.
</Note>

```java theme={null}
public class TestProduce {
 
    public static void main(String[] args) {
        PulsarClient client = PulsarClient.builder()
        .serviceUrl("{{Your Pulsar URL}}")
        .build();

        Producer<byte[]> producer = client.newProducer()
            .topic("{{Your topic name}}")
            .create();

        for (int i = 0; i < 10; i++) {
            String message = "my-message-" + i;
            MessageId msgID = producer.send(message.getBytes());
            System.out.println("Publish " + "my-message-" + i
                    + " and message ID " + msgID);
        }
        
        producer.close();
        client.close();  
    }
}

```

### 3. Show data on Google PubSub

You can create a subscription and pull data from the Google Pub/Sub console.

```text theme={null}
+---------------------------+-----------------+------------------|
| Publish time              | Attribute keys  |  Message body    |   
+---------------------------+-----------------+------------------|
| Feb 19, 2024, 4:17:42 PM  |               - | my-message-0     |
| Feb 19, 2024, 4:17:42 PM  |               - | my-message-1     |
| Feb 19, 2024, 4:17:42 PM  |               - | my-message-2     |
| Feb 19, 2024, 4:17:42 PM  |               - | my-message-3     |
| Feb 19, 2024, 4:17:43 PM  |               - | my-message-4     |
| Feb 19, 2024, 4:17:43 PM  |               - | my-message-5     |
| Feb 19, 2024, 4:17:43 PM  |               - | my-message-6     |
| Feb 19, 2024, 4:17:43 PM  |               - | my-message-7     |
| Feb 19, 2024, 4:17:44 PM  |               - | my-message-8     |
| Feb 19, 2024, 4:17:44 PM  |               - | my-message-9     |
+---------------------------+-----------------+------------------|
```

## Configuration Properties

Before using the Google Cloud PubSub sink connector, you need to configure it. This table outlines the properties and the descriptions.

| Name                     | Type   | Required | Sensitive | Default            | Description                                                                                                                                                                                                    |
| ------------------------ | ------ | -------- | --------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `pubsubCredential`       | String | true     | true      | "" (empty string)  | The credential (JSON string) for accessing the Google Cloud. It needs to be compressed and escaping before use.                                                                                                |
| `pubsubProjectId`        | String | true     | false     | "" (empty string)  | The Google Cloud project ID.                                                                                                                                                                                   |
| `pubsubTopicId`          | String | true     | false     | " " (empty string) | The topic ID. It is used to read messages from or write messages to Google Cloud Pub/Sub topics.                                                                                                               |
| `pubsubSchemaId`         | String | false    | false     | "" (empty string)  | The schema ID. You must set the schema ID when creating a schema for Google Cloud Pub/Sub topics.                                                                                                              |
| `pubsubSchemaType`       | String | false    | false     | "" (empty string)  | The schema type. You must set the schema type when creating a schema for Google Cloud Pub/Sub topics. Currently, only the AVRO format is supported.                                                            |
| `pubsubSchemaEncoding`   | String | false    | false     | "" (empty string)  | The encoding of the schema. You must set the schema encoding when creating a schema for Google Cloud Pub/Sub topics. Currently, only the JSON format is supported.                                             |
| `pubsubSchemaDefinition` | String | false    | false     | "" (empty string)  | The definition of the schema. It is used to create a schema to or parse messages from Google Cloud Pub/Sub topics.                                                                                             |
| `subscriptionName`       | String | false    | false     | "" (empty string)  | The fully-qualified subscription name to read from (e.g., projects/my-sub-project/subscriptions/my-sub). If not set, a subscription is created in the pubsubProjectId with the same name as the pubsubTopicId. |


# Google pubsub source
Source: https://docs.streamnative.io/connect/connectors/google-pubsub-source/current/google-pubsub-source

The Google Pub/Sub source connector allows you to write messages from Google Pub/Sub to Apache Pulsar.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

The [Google Cloud Pub/Sub](https://cloud.google.com/pubsub) source connector feeds data from Google Cloud Pub/Sub topics and writes data to Pulsar topics.

<img alt="" />

## Quick start

### Prerequisites

The prerequisites for connecting an Google PubSub source connector to external systems include:

1. Create Google PubSub Topic in Google Cloud.
2. Create the [Gcloud ServiceAccount](https://cloud.google.com/iam/docs/service-accounts-create) and create a public key certificate.
3. Create the [Gcloud Role](https://cloud.google.com/iam/docs/creating-custom-roles), ensure the Google Cloud role have the following permissions:

```text theme={null}
- pubsub.subscriptions.consume
- pubsub.subscriptions.create
- pubsub.subscriptions.get
- pubsub.subscriptions.update
- pubsub.topics.attachSubscription
```

4. Grant the service account the above role permissions.

### 1. Create a connector

The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector,
you need to replace `--source-type google-pubsub` with `--archive /path/to/pulsar-io-google-pubsub.nar`. You can find the button to download the `nar` package at the beginning of the document.

<Note title="For StreamNative Cloud User">
  If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first.
</Note>

```bash theme={null}
pulsarctl sources create \
  --source-type google-pubsub \
  --name pubsub-source \
  --tenant public \
  --namespace default \
  --destination-topic-name "Your topic name" \
  --parallelism 1 \
  --source-config \
  '{
    "pubsubProjectId": "Your google pubsub project Id", 
    "pubsubTopicId": "Your google pubsub Topic name",
    "pubsubCredential": "The escaped and compressed public key certificate you created above"
  }'
```

The `--source-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.
If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference.

<Note title="Note">
  You can also choose to use a variety of other tools to create a connector:

  * [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [RestAPI](https://pulsar.apache.org/source-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document.
</Note>

### 2. Write data to Google PubSub topic

Send some messages to the Google Cloud PubSub using the [gcloud CLI tool](https://cloud.google.com/sdk/docs/install)

```shell theme={null}
gcloud pubsub topics publish `{{Your PubSub Topic Name}}` --message="my-message-0"
gcloud pubsub topics publish `{{Your PubSub Topic Name}}` --message="my-message-1"
gcloud pubsub topics publish `{{Your PubSub Topic Name}}` --message="my-message-2"
gcloud pubsub topics publish `{{Your PubSub Topic Name}}` --message="my-message-3"
gcloud pubsub topics publish `{{Your PubSub Topic Name}}` --message="my-message-4"
gcloud pubsub topics publish `{{Your PubSub Topic Name}}` --message="my-message-5"
gcloud pubsub topics publish `{{Your PubSub Topic Name}}` --message="my-message-6"
gcloud pubsub topics publish `{{Your PubSub Topic Name}}` --message="my-message-7"
gcloud pubsub topics publish `{{Your PubSub Topic Name}}` --message="my-message-8"
gcloud pubsub topics publish `{{Your PubSub Topic Name}}` --message="my-message-9"
```

### 3. Show data by Pulsar Consumer

<Note title="Note">
  If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information.
</Note>

```java theme={null}
    public static void main(String[] args) {
        PulsarClient client = PulsarClient.builder()
            .serviceUrl("{{Your Pulsar URL}}")
            .build();

        Consumer<GenericRecord> consumer = client.newConsumer(Schema.AUTO_CONSUME())
                                                 .topic("{{The topic name that you specified when you created the connector}}")
                                                 .subscriptionName(subscription)
                                                 .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
                                                 .subscribe();
        Consumer<byte[]> consumer = client.newConsumer()
                .topic("{{The topic name that you specified when you created the connector}}")
                .subscriptionName("test-sub")
                .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
                .subscribe();

        for (int i = 0; i < 10; i++) {
          Message<byte[]> msg = consumer.receive();
          consumer.acknowledge(msg);
          System.out.println("Receive message " + new String(msg.getData()));
        }
        client.close();  
    }
    // output
    // Receive message my-message-0
    // Receive message my-message-1
    // Receive message my-message-2
    // Receive message my-message-3
    // Receive message my-message-4
    // Receive message my-message-5
    // Receive message my-message-6
    // Receive message my-message-7
    // Receive message my-message-8
    // Receive message my-message-9
```

## Configuration Properties

Before using the Google PubSub source connector, you need to configure it. This table outlines the properties and the descriptions.

| Name               | Type   | Required | Sensitive | Default            | Description                                                                                                     |
| ------------------ | ------ | -------- | --------- | ------------------ | --------------------------------------------------------------------------------------------------------------- |
| `pubsubCredential` | String | true     | true      | "" (empty string)  | The credential (JSON string) for accessing the Google Cloud. It needs to be compressed and escaping before use. |
| `pubsubProjectId`  | String | true     | false     | "" (empty string)  | The Google Cloud project ID.                                                                                    |
| `pubsubTopicId`    | String | true     | false     | " " (empty string) | The topic ID. It is used to read messages from or write messages to Google Cloud Pub/Sub topics.                |


# Hbase sink
Source: https://docs.streamnative.io/connect/connectors/hbase-sink/current/hbase-sink

The HBase sink connector pulls the messages from Pulsar topics and persists the messages to HBase tables

The HBase sink connector pulls the messages from Pulsar topics and persists the messages to HBase tables.

# Configuration

The configuration of the HBase sink connector has the following properties.

## Property

| Name                   | Type   | Default | Required | Description                                                                   |
| ---------------------- | ------ | ------- | -------- | ----------------------------------------------------------------------------- |
| `hbaseConfigResources` | String | None    | false    | HBase system configuration `hbase-site.xml` file.                             |
| `zookeeperQuorum`      | String | None    | true     | HBase system configuration about `hbase.zookeeper.quorum` value.              |
| `zookeeperClientPort`  | String | 2181    | false    | HBase system configuration about `hbase.zookeeper.property.clientPort` value. |
| `zookeeperZnodeParent` | String | /hbase  | false    | HBase system configuration about `zookeeper.znode.parent` value.              |
| `tableName`            | None   | String  | true     | HBase table, the value is `namespace:tableName`.                              |
| `rowKeyName`           | String | None    | true     | HBase table rowkey name.                                                      |
| `familyName`           | String | None    | true     | HBase table column family name.                                               |
| `qualifierNames`       | String | None    | true     | HBase table column qualifier names.                                           |
| `batchTimeMs`          | Long   | 1000l   | false    | HBase table operation timeout in milliseconds.                                |
| `batchSize`            | int    | 200     | false    | Batch size of updates made to the HBase table.                                |

## Example

Before using the HBase sink connector, you need to create a configuration file through one of the following methods.

* JSON

  ```json theme={null}
  {
      "hbaseConfigResources": "hbase-site.xml",
      "zookeeperQuorum": "localhost",
      "zookeeperClientPort": "2181",
      "zookeeperZnodeParent": "/hbase",
      "tableName": "pulsar_hbase",
      "rowKeyName": "rowKey",
      "familyName": "info",
      "qualifierNames": [ 'name', 'address', 'age']
  }
  ```

* YAML

  ```yaml theme={null}
  configs:
      hbaseConfigResources: "hbase-site.xml"
      zookeeperQuorum: "localhost"
      zookeeperClientPort: "2181"
      zookeeperZnodeParent: "/hbase"
      tableName: "pulsar_hbase"
      rowKeyName: "rowKey"
      familyName: "info"
      qualifierNames: [ 'name', 'address', 'age']
  ```


# Hdfs3 sink
Source: https://docs.streamnative.io/connect/connectors/hdfs3-sink/current/hdfs3-sink

The HDFS3 sink connector pulls the messages from Pulsar topics and persists the messages to HDFS files.

The HDFS3 sink connector pulls the messages from Pulsar topics and persists the messages to HDFS files.

# Configuration

The configuration of the HDFS3 sink connector has the following properties.

## Property

| Name                    | Type        | Required | Default            | Description                                                                                                                                                                                                                                                                   |
| ----------------------- | ----------- | -------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `hdfsConfigResources`   | String      | true     | None               | A file or a comma-separated list containing the Hadoop file system configuration.<br /><br />**Example**<br />'core-site.xml'<br />'hdfs-site.xml'                                                                                                                            |
| `directory`             | String      | true     | None               | The HDFS directory where files read from or written to.                                                                                                                                                                                                                       |
| `encoding`              | String      | false    | None               | The character encoding for the files.<br /><br />**Example**<br />UTF-8<br />ASCII                                                                                                                                                                                            |
| `compression`           | Compression | false    | None               | The compression code used to compress or de-compress the files on HDFS. <br /><br />Below are the available options:<br /><li>BZIP2</li><li>DEFLATE</li><li>GZIP</li><li>LZ4</li><li>SNAPPY</li>                                                                              |
| `kerberosUserPrincipal` | String      | false    | None               | The principal account of Kerberos user used for authentication.                                                                                                                                                                                                               |
| `keytab`                | String      | false    | None               | The full pathname of the Kerberos keytab file used for authentication.                                                                                                                                                                                                        |
| `filenamePrefix`        | String      | false    | None               | The prefix of the files created inside the HDFS directory.<br /><br />**Example**<br /> The value of topicA result in files named topicA-.                                                                                                                                    |
| `fileExtension`         | String      | false    | None               | The extension added to the files written to HDFS.<br /><br />**Example**<br />'.txt'<br /> '.seq'                                                                                                                                                                             |
| `separator`             | char        | false    | None               | The character used to separate records in a text file. <br /><br />If no value is provided, the contents from all records are concatenated together in one continuous byte array.                                                                                             |
| `syncInterval`          | long        | false    | 0                  | The interval between calls to flush data to HDFS disk in milliseconds.                                                                                                                                                                                                        |
| `maxPendingRecords`     | int         | false    | Integer.MAX\_VALUE | The maximum number of records that hold in memory before acking. <br /><br />Setting this property to 1 makes every record send to disk before the record is acked.<br /><br />Setting this property to a higher value allows buffering records before flushing them to disk. |

## Example

Before using the HDFS3 sink connector, you need to create a configuration file through one of the following methods.

* JSON

  ```json theme={null}
  {
      "hdfsConfigResources": "core-site.xml",
      "directory": "/foo/bar",
      "filenamePrefix": "prefix",
      "compression": "SNAPPY"
  }
  ```

* YAML

  ```yaml theme={null}
  configs:
      hdfsConfigResources: "core-site.xml"
      directory: "/foo/bar"
      filenamePrefix: "prefix"
      compression: "SNAPPY"
  ```


# Influxdb sink
Source: https://docs.streamnative.io/connect/connectors/influxdb-sink/current/influxdb-sink

The InfluxDB sink connector pulls messages from Pulsar topics and persists the messages to InfluxDB.

The InfluxDB sink connector pulls messages from Pulsar topics and persists the messages to InfluxDB.

The InfluxDB sink provides different configurations for InfluxDBv1 and v2 respectively.

# Configuration

The configuration of the InfluxDB sink connector has the following properties.

## Property

### InfluxDBv2

| Name           | Type    | Required | Sensitive | Default            | Description                                                                                                                                           |
| -------------- | ------- | -------- | --------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `influxdbUrl`  | String  | true     | false     | " " (empty string) | The URL of the InfluxDB instance.                                                                                                                     |
| `token`        | String  | true     | true      | " " (empty string) | The authentication token used to authenticate to InfluxDB.                                                                                            |
| `organization` | String  | true     | false     | " " (empty string) | The InfluxDB organization to write to.                                                                                                                |
| `bucket`       | String  | true     | false     | " " (empty string) | The InfluxDB bucket to write to.                                                                                                                      |
| `precision`    | String  | false    | false     | ns                 | The timestamp precision for writing data to InfluxDB. <br /><br />Below are the available options:<li>ns</li><li>us</li><li>ms</li><li>s</li>         |
| `logLevel`     | String  | false    | false     | NONE               | The log level for InfluxDB request and response. <br /><br />Below are the available options:<li>NONE</li><li>BASIC</li><li>HEADERS</li><li>FULL</li> |
| `gzipEnable`   | boolean | false    | false     | false              | Whether to enable gzip or not.                                                                                                                        |
| `batchTimeMs`  | long    | false    | false     | 1000L              | The InfluxDB operation time in milliseconds.                                                                                                          |
| `batchSize`    | int     | false    | false     | 200                | The batch size of writing to InfluxDB.                                                                                                                |

### InfluxDBv1

| Name               | Type    | Required | Sensitive | Default            | Description                                                                                                                                           |
| ------------------ | ------- | -------- | --------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `influxdbUrl`      | String  | true     | false     | " " (empty string) | The URL of the InfluxDB instance.                                                                                                                     |
| `username`         | String  | false    | true      | " " (empty string) | The username used to authenticate to InfluxDB.                                                                                                        |
| `password`         | String  | false    | true      | " " (empty string) | The password used to authenticate to InfluxDB.                                                                                                        |
| `database`         | String  | true     | false     | " " (empty string) | The InfluxDB to which write messages.                                                                                                                 |
| `consistencyLevel` | String  | false    | false     | ONE                | The consistency level for writing data to InfluxDB. <br /><br />Below are the available options:<li>ALL</li><li>ANY</li><li>ONE</li><li>QUORUM</li>   |
| `logLevel`         | String  | false    | false     | NONE               | The log level for InfluxDB request and response. <br /><br />Below are the available options:<li>NONE</li><li>BASIC</li><li>HEADERS</li><li>FULL</li> |
| `retentionPolicy`  | String  | false    | false     | autogen            | The retention policy for InfluxDB.                                                                                                                    |
| `gzipEnable`       | boolean | false    | false     | false              | Whether to enable gzip or not.                                                                                                                        |
| `batchTimeMs`      | long    | false    | false     | 1000L              | The InfluxDB operation time in milliseconds.                                                                                                          |
| `batchSize`        | int     | false    | false     | 200                | The batch size of writing to InfluxDB.                                                                                                                |

## Example

Before using the InfluxDB sink connector, you need to create a configuration file through one of the following methods.

### InfluxDBv2

* JSON
  ```json theme={null}
  {
      "influxdbUrl": "http://localhost:9999",
      "organization": "example-org",
      "bucket": "example-bucket",
      "token": "xxxx",
      "precision": "ns",
      "logLevel": "NONE",
      "gzipEnable": false,
      "batchTimeMs": 1000,
      "batchSize": 100
  }
  ```

* YAML
  ```yaml theme={null}
  {
      influxdbUrl: "http://localhost:9999"
      organization: "example-org"
      bucket: "example-bucket"
      token: "xxxx"
      precision: "ns"
      logLevel: "NONE"
      gzipEnable: false
      batchTimeMs: 1000
      batchSize: 100
  }
  ```

### InfluxDBv1

* JSON

  ```json theme={null}
  {
      "influxdbUrl": "http://localhost:8086",
      "database": "test_db",
      "consistencyLevel": "ONE",
      "logLevel": "NONE",
      "retentionPolicy": "autogen",
      "gzipEnable": false,
      "batchTimeMs": 1000,
      "batchSize": 100
  }
  ```

* YAML

  ```yaml theme={null}
  {
      influxdbUrl: "http://localhost:8086"
      database: "test_db"
      consistencyLevel: "ONE"
      logLevel: "NONE"
      retentionPolicy: "autogen"
      gzipEnable: false
      batchTimeMs: 1000
      batchSize: 100
  }
  ```


# Jdbc sink
Source: https://docs.streamnative.io/connect/connectors/jdbc-clickhouse-sink/current/jdbc-sink

The JDBC sink connector pulls messages from Pulsar topics and persists the messages to MySQL or SQlite.

This document introduces how to get started with creating an JDBC Clickhouse sink connector and get it up and running.

## Quick start

### Prerequisites

The prerequisites for connecting an JDBC Clickhouse sink connector to external systems include:

1. Start a [Clickhouse server](https://clickhouse.com/docs/en/getting-started/quick-start). You can create a single-node Clickhouse cluster by executing this command:

```bash theme={null}
curl https://clickhouse.com/ | sh
./clickhouse server
```

2. Create a table, you can use `./clickhouse client` to open a SQL shell.

```sql theme={null}
CREATE TABLE users (
name String,
age UInt8,
city String
) ENGINE = MergeTree()
ORDER BY (name, age);
```

### 1. Create a connector

The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector,
you need to replace `--sink-type jdbc-clickhouse` with `--archive /path/to/pulsar-io-jdbc-clickhouse.nar`. You can find the button to download the `nar` package at the beginning of the document.

<Note title="For StreamNative Cloud User">
  If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first.
</Note>

```bash theme={null}
pulsarctl sinks create \
  --sink-type jdbc-clickhouse \
  --name jdbc-clickhouse-sink \
  --tenant public \
  --namespace default \
  --inputs "Your topic name" \
  --parallelism 1 \
  --sink-config \
  '{
    "jdbcUrl": "jdbc:clickhouse://127.0.0.1:8123/default",
    "tableName": "users"
  }'
```

The `--sink-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.
If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference.

<Note title="Note">
  You can also choose to use a variety of other tools to create a connector:

  * [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [RestAPI](https://pulsar.apache.org/sink-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document.
</Note>

### 2. Send messages to the topic

<Note title="Note">
  If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information.
</Note>

Note that the current implementation supports structured types of schemas, such as `Avro`, `JSON`, `Protobuf`, `Protobuf_native`, etc.

```java theme={null}
@Data
@AllArgsConstructor
@NoArgsConstructor
public class ProducerTest {
    
    private String name;
    private int age;
    private String city;
    
    
    public static void main(String[] args) throws PulsarClientException {
        PulsarClient client = PulsarClient.builder()
                .serviceUrl("{{Your Pulsar URL}}")
                .build();

        Producer<ProducerTest> producer = client.newProducer(Schema.JSON(ProducerTest.class))
                        .topic("{{Your topic name}}").create();

        MessageId msgID = producer.send(new ProducerTest("John Doe", 30, "New York"));
        System.out.println("Publish message and message ID " + msgID);

        producer.flush();
        producer.close();
        client.close();
        
    }
}
```

### 3. Check data on clickhouse

```text theme={null}
SELECT *
FROM users

Query id: b555a027-a781-47bc-b3dd-c7ffb30dc513

   ┌─name─────┬─age─┬─city─────┐
1. │ John Doe │  30 │ New York │
   └──────────┴─────┴──────────┘

1 row in set. Elapsed: 0.002 sec.
```

## Configuration Properties

The configuration of the JDBC sink connector has the following properties.

| Name                       | Type    | Required | Sensitive | Default            | Description                                                                                                                                                                                                                                                                                                                               |
| -------------------------- | ------- | -------- | --------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `userName`                 | String  | false    | true      | " " (empty string) | The username used to connect to the database specified by `jdbcUrl`.<br /><br />**Note: `userName` is case-sensitive.**                                                                                                                                                                                                                   |
| `password`                 | String  | false    | true      | " " (empty string) | The password used to connect to the database specified by `jdbcUrl`. <br /><br />**Note: `password` is case-sensitive.**                                                                                                                                                                                                                  |
| `jdbcUrl`                  | String  | true     | false     | " " (empty string) | The JDBC URL of the database to which the connector connects.                                                                                                                                                                                                                                                                             |
| `tableName`                | String  | true     | false     | " " (empty string) | The name of the table to which the connector writes.                                                                                                                                                                                                                                                                                      |
| `key`                      | String  | false    | false     | " " (empty string) | A comma-separated list contains the fields used in `where` condition of updating and deleting events.                                                                                                                                                                                                                                     |
| `nonKey`                   | String  | false    | false     | " " (empty string) | A comma-separated list contains the fields used in updating events.                                                                                                                                                                                                                                                                       |
| `insertMode`               | enum    | false    | false     | INSERT             | Option: INSERT, DELETE and UPDATE. If it is configured as UPSERT, the sink will use upsert semantics rather than plain INSERT/UPDATE statements. Upsert semantics refer to atomically adding a new row or updating the existing row if there is a primary key constraint violation, which provides idempotence.                           |
| `nullValueAction`          | enum    | false    | false     | FAIL               | Option: FAIL, DELETE.  How to handle records with null values, possible options are DELETE or FAIL.                                                                                                                                                                                                                                       |
| `useTransactions`          | boolean | false    | false     | false              | Enable transactions of the database.                                                                                                                                                                                                                                                                                                      |
| `excludeNonDeclaredFields` | boolean | false    | false     | false              | All the table fields are discovered automatically. 'excludeNonDeclaredFields' indicates if the table fields not explicitly listed in `nonKey` and `key` must be included in the query. By default all the table fields are included. To leverage of table fields defaults  during insertion, it is suggested to set this value to `true`. |
| `useJdbcBatch`             | boolean | false    | false     | false              | Use the JDBC batch API. This option is suggested to improve write performance.                                                                                                                                                                                                                                                            |
| `timeoutMs`                | int     | false    | false     | 500                | The JDBC operation timeout in milliseconds.                                                                                                                                                                                                                                                                                               |
| `batchSize`                | int     | false    | false     | 200                | The batch size of updates made to the database.                                                                                                                                                                                                                                                                                           |


# Jdbc sink
Source: https://docs.streamnative.io/connect/connectors/jdbc-mariadb-sink/current/jdbc-sink

The JDBC sink connector pulls messages from Pulsar topics and persists the messages to MySQL or SQlite.

The JDBC sink connector pulls messages from Pulsar topics and persists the messages to MySQL or SQlite.

> Currently, INSERT, DELETE and UPDATE operations are supported.

# Configuration

The configuration of the JDBC sink connector has the following properties.

## Property

| Name        | Type   | Required | Sensitive | Default            | Description                                                                                                              |
| ----------- | ------ | -------- | --------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------ |
| `userName`  | String | false    | true      | " " (empty string) | The username used to connect to the database specified by `jdbcUrl`.<br /><br />**Note: `userName` is case-sensitive.**  |
| `password`  | String | false    | true      | " " (empty string) | The password used to connect to the database specified by `jdbcUrl`. <br /><br />**Note: `password` is case-sensitive.** |
| `jdbcUrl`   | String | true     | false     | " " (empty string) | The JDBC URL of the database to which the connector connects.                                                            |
| `tableName` | String | true     | false     | " " (empty string) | The name of the table to which the connector writes.                                                                     |
| `nonKey`    | String | false    | false     | " " (empty string) | A comma-separated list contains the fields used in updating events.                                                      |
| `key`       | String | false    | false     | " " (empty string) | A comma-separated list contains the fields used in `where` condition of updating and deleting events.                    |
| `timeoutMs` | int    | false    | false     | 500                | The JDBC operation timeout in milliseconds.                                                                              |
| `batchSize` | int    | false    | false     | 200                | The batch size of updates made to the database.                                                                          |

## Example

Before using the JDBC sink connector, you need to create a configuration file through one of the following methods.

* JSON

  ```json theme={null}
  {
      "userName": "root",
      "password": "jdbc",
      "jdbcUrl": "jdbc:mysql://127.0.0.1:3306/pulsar_mysql_jdbc_sink",
      "tableName": "pulsar_mysql_jdbc_sink"
  }
  ```

* YAML

  ```yaml theme={null}
  configs:
      userName: "root"
      password: "jdbc"
      jdbcUrl: "jdbc:mysql://127.0.0.1:3306/pulsar_mysql_jdbc_sink"
      tableName: "pulsar_mysql_jdbc_sink"
  ```

# Usage

For more information about **how to use a JDBC sink connector**, see [connect Pulsar to Postgres](https://pulsar.apache.org/docs/io-quickstart/#connect-pulsar-to-postgresql).


# Jdbc sink
Source: https://docs.streamnative.io/connect/connectors/jdbc-postgres-sink/current/jdbc-sink

The JDBC sink connector pulls messages from Pulsar topics and persists the messages to MySQL or SQlite.

The JDBC sink connector pulls messages from Pulsar topics and persists the messages to MySQL or SQlite.

> Currently, INSERT, DELETE and UPDATE operations are supported.

# Configuration

The configuration of the JDBC sink connector has the following properties.

## Property

| Name        | Type   | Required | Sensitive | Default            | Description                                                                                                              |
| ----------- | ------ | -------- | --------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------ |
| `userName`  | String | false    | true      | " " (empty string) | The username used to connect to the database specified by `jdbcUrl`.<br /><br />**Note: `userName` is case-sensitive.**  |
| `password`  | String | false    | true      | " " (empty string) | The password used to connect to the database specified by `jdbcUrl`. <br /><br />**Note: `password` is case-sensitive.** |
| `jdbcUrl`   | String | true     | false     | " " (empty string) | The JDBC URL of the database to which the connector connects.                                                            |
| `tableName` | String | true     | false     | " " (empty string) | The name of the table to which the connector writes.                                                                     |
| `nonKey`    | String | false    | false     | " " (empty string) | A comma-separated list contains the fields used in updating events.                                                      |
| `key`       | String | false    | false     | " " (empty string) | A comma-separated list contains the fields used in `where` condition of updating and deleting events.                    |
| `timeoutMs` | int    | false    | false     | 500                | The JDBC operation timeout in milliseconds.                                                                              |
| `batchSize` | int    | false    | false     | 200                | The batch size of updates made to the database.                                                                          |

## Batch support (PostgreSQL)

When using PostgreSQL with batched writes (for example, setting `batchSize > 1`), add `reWriteBatchedInserts=true` to the `jdbcUrl` so the PostgreSQL JDBC driver rewrites batched inserts for better performance.

* Example: `jdbc:postgresql://<host>:5432/<db>?reWriteBatchedInserts=true`
* The parameter name is case-sensitive: `reWriteBatchedInserts`.

For details, see the PostgreSQL JDBC driver documentation: [https://jdbc.postgresql.org/documentation/use/#:\~:text=reWriteBatchedInserts%20(boolean)%20Default%20false%0AThis%20will](https://jdbc.postgresql.org/documentation/use/#:~:text=reWriteBatchedInserts%20\(boolean\)%20Default%20false%0AThis%20will)

## Example

Before using the JDBC sink connector, you need to create a configuration file through one of the following methods.

* JSON

  ```json theme={null}
  {
      "userName": "root",
      "password": "jdbc",
      "jdbcUrl": "jdbc:mysql://127.0.0.1:3306/pulsar_mysql_jdbc_sink",
      "tableName": "pulsar_mysql_jdbc_sink"
  }
  ```

* YAML

  ```yaml theme={null}
  configs:
      userName: "root"
      password: "jdbc"
      jdbcUrl: "jdbc:mysql://127.0.0.1:3306/pulsar_mysql_jdbc_sink"
      tableName: "pulsar_mysql_jdbc_sink"
  ```

# Usage

For more information about **how to use a JDBC sink connector**, see [connect Pulsar to Postgres](https://pulsar.apache.org/docs/io-quickstart/#connect-pulsar-to-postgresql).


# Jdbc sink
Source: https://docs.streamnative.io/connect/connectors/jdbc-sqlite-sink/current/jdbc-sink

The JDBC sink connector pulls messages from Pulsar topics and persists the messages to MySQL or SQlite.

The JDBC sink connector pulls messages from Pulsar topics and persists the messages to MySQL or SQlite.

> Currently, INSERT, DELETE and UPDATE operations are supported.

# Configuration

The configuration of the JDBC sink connector has the following properties.

## Property

| Name        | Type   | Required | Sensitive | Default            | Description                                                                                                              |
| ----------- | ------ | -------- | --------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------ |
| `userName`  | String | false    | true      | " " (empty string) | The username used to connect to the database specified by `jdbcUrl`.<br /><br />**Note: `userName` is case-sensitive.**  |
| `password`  | String | false    | true      | " " (empty string) | The password used to connect to the database specified by `jdbcUrl`. <br /><br />**Note: `password` is case-sensitive.** |
| `jdbcUrl`   | String | true     | false     | " " (empty string) | The JDBC URL of the database to which the connector connects.                                                            |
| `tableName` | String | true     | false     | " " (empty string) | The name of the table to which the connector writes.                                                                     |
| `nonKey`    | String | false    | false     | " " (empty string) | A comma-separated list contains the fields used in updating events.                                                      |
| `key`       | String | false    | false     | " " (empty string) | A comma-separated list contains the fields used in `where` condition of updating and deleting events.                    |
| `timeoutMs` | int    | false    | false     | 500                | The JDBC operation timeout in milliseconds.                                                                              |
| `batchSize` | int    | false    | false     | 200                | The batch size of updates made to the database.                                                                          |

## Example

Before using the JDBC sink connector, you need to create a configuration file through one of the following methods.

* JSON

  ```json theme={null}
  {
      "userName": "root",
      "password": "jdbc",
      "jdbcUrl": "jdbc:mysql://127.0.0.1:3306/pulsar_mysql_jdbc_sink",
      "tableName": "pulsar_mysql_jdbc_sink"
  }
  ```

* YAML

  ```yaml theme={null}
  configs:
      userName: "root"
      password: "jdbc"
      jdbcUrl: "jdbc:mysql://127.0.0.1:3306/pulsar_mysql_jdbc_sink"
      tableName: "pulsar_mysql_jdbc_sink"
  ```

# Usage

For more information about **how to use a JDBC sink connector**, see [connect Pulsar to Postgres](https://pulsar.apache.org/docs/io-quickstart/#connect-pulsar-to-postgresql).


# Kafka connect bigquery
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-bigquery/current/kafka-connect-bigquery

kafka-connect-bigquery is a Kafka Connect connector for Google BigQuery.

`kafka-connect-bigquery` is a Kafka Connect connector for Google BigQuery. It is available in the StreamNative Cloud.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

### Prerequisites

The following prerequisites are required before setting up the BigQuery connector.

1. A valid Google Cloud account authorized for resource creation.
2. A BigQuery project, which can be set up via the Google Cloud Console.
3. A dataset within the BigQuery project.
4. A service account with access to the BigQuery project that hosts the dataset; this account can be created in the Google Cloud Console.
5. Ensure the service account is granted access to the BigQuery project containing the dataset; create and download a key in JSON format when setting up the service account.
6. According to `GCP specifications <https://cloud.google.com/bigquery/docs/access-control>`, the service account will either need the **BigQueryEditor** primitive IAM role or the **bigquery.dataEditor** predefined IAM role. The minimum permissions are as follows:

```
   bigquery.datasets.get
   bigquery.tables.create
   bigquery.tables.get
   bigquery.tables.getData
   bigquery.tables.list
   bigquery.tables.update
   bigquery.tables.updateData
```

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)

2. Create a GSA(google service account) in Google Cloud, and get the private key of it

3. Create a secret in StreamNative Console, and save the GSA private key's content , please refer to: [doc](https://docs.streamnative.io/docs/kafka-connect-create#create-kafka-connect-with-secret), let's say the secret name is `gcp`, and key is `auth`

4. Create a dataset in Google Cloud

5. Create a json file like below:

   ```json theme={null}
   {
       "name": "test-bq",
       "config": {
           "connector.class": "com.wepay.kafka.connect.bigquery.BigQuerySinkConnector",
           "topics": "${INPUT_TOPIC}",
           "project": "${GCP_PROJECT_NAME}",
           "defaultDataset": "${DATA_SET_NAME}",
           "key.converter": " org.apache.kafka.connect.storage.StringConverter",
           "value.converter": "org.apache.kafka.connect.json.JsonConverter",
           "autoCreateTables": "false",
           "keySource": "JSON",
           "keyfile": "${snsecret:gcp:auth}",
           "value.converter.schemas.enable": false
       }
   }
   ```

6. Run the following command to create the connector:

   ```shell theme={null}
   kcctl apply -f <your-json-file>
   ```

### Configuration

The `kafka-connect-bigquery` connector is configured using the following properties:

| Parameter                            | Description                                                                                                                                                                                                                                                                                                                                                                                                                      | Default                                                           |
| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- |
| project                              | The Google Cloud project ID                                                                                                                                                                                                                                                                                                                                                                                                      |                                                                   |
| defaultDataset                       | The BigQuery dataset name                                                                                                                                                                                                                                                                                                                                                                                                        |                                                                   |
| topics                               | The Kafka topics to sink to BigQuery                                                                                                                                                                                                                                                                                                                                                                                             |                                                                   |
| autoCreateTables                     | Create BigQuery tables if they don’t already exist. This property should only be enabled for Schema Registry-based inputs: Avro, Protobuf, or JSON Schema (JSON\_SR). Table creation is not supported for JSON input.                                                                                                                                                                                                            | true                                                              |
| keyfile                              | keyfile can be either a string representation of the Google credentials file or the path to the Google credentials file itself. The string representation of the Google credentials file is supported in BigQuery sink connector version 1.3 (and later). For StreamNative Cloud, the keyfile will be saved as StreamNative Cloud secret in JSON format                                                                          |                                                                   |
| keySource                            | The source of the keyfile. The keyfile can be provided as a string or a file path. For StreamNative Cloud, only `JSON` supported.                                                                                                                                                                                                                                                                                                | JSON                                                              |
| gcsBucketName                        | The name of the bucket where Google Cloud Storage (GCS) blobs are located. These blobs are used to batch-load to BigQuery. This is applicable only if `enableBatchLoad` is configured.                                                                                                                                                                                                                                           |                                                                   |
| queueSize                            | The maximum size (or -1 for no maximum size) of the worker queue for BigQuery write requests before all topics are paused. This is a soft limit; the size of the queue can go over this before topics are paused. All topics resume once a flush is triggered or the size of the queue drops under half of the maximum size.                                                                                                     | -1                                                                |
| bigQueryRetry                        | The number of retry attempts made for a BigQuery request that fails with a backend error or a quota exceeded error.                                                                                                                                                                                                                                                                                                              | 0                                                                 |
| bigQueryRetryWait                    | The time in milliseconds to wait between retry attempts.                                                                                                                                                                                                                                                                                                                                                                         | 1000                                                              |
| bigQueryMessageTimePartitioning      | The time partitioning configuration for BigQuery tables.                                                                                                                                                                                                                                                                                                                                                                         | false                                                             |
| bigQueryPartitionDecorator           | Whether or not to append partition decorator to BigQuery table name when inserting records. Default is true. Setting this to true appends partition decorator to table name (e.g. table\$yyyyMMdd depending on the configuration set for bigQueryPartitionDecorator). Setting this to false bypasses the logic to append the partition decorator and uses raw table name for inserts.                                            | true                                                              |
| timestampPartitionFieldName          | The field name in the record that contains the timestamp to use for partitioning.                                                                                                                                                                                                                                                                                                                                                | null                                                              |
| clusteringPartitionFieldNames        | The field names in the record that contain the fields to use for clustering.                                                                                                                                                                                                                                                                                                                                                     | null                                                              |
| timePartitioningType                 | The type of time partitioning to use. Support `MONTH, YEAR, HOUR, DAY`                                                                                                                                                                                                                                                                                                                                                           | DAY                                                               |
| sanitizeTopics                       | Whether to sanitize topic names to be compatible with BigQuery table names.                                                                                                                                                                                                                                                                                                                                                      | false                                                             |
| schemaRetriever                      | A class that can be used for automatically creating tables and/or updating schemas. Note that in version 2.0.0, SchemaRetriever API changed to retrieve the schema from each SinkRecord, which will help support multiple schemas per topic. SchemaRegistrySchemaRetriever has been removed as it retrieves schema based on the topic.                                                                                           | com.wepay.kafka.connect.bigquery.retrieve.IdentitySchemaRetriever |
| threadPoolSize                       | The size of the BigQuery write thread pool. This establishes the maximum number of concurrent writes to BigQuery.                                                                                                                                                                                                                                                                                                                | 10                                                                |
| allBQFieldsNullable                  | If true, no fields in any produced BigQuery schema are REQUIRED. All non-nullable Avro fields are translated as NULLABLE (or REPEATED, if arrays).                                                                                                                                                                                                                                                                               | false                                                             |
| avroDataCacheSize                    | The size of the Avro data cache.                                                                                                                                                                                                                                                                                                                                                                                                 | 100                                                               |
| batchLoadIntervalSec                 | The interval, in seconds, in which to attempt to run GCS to BigQuery load jobs. Only relevant if `enableBatchLoad` is configured.                                                                                                                                                                                                                                                                                                | 120                                                               |
| convertDoubleSpecialValues           | Designates whether +Infinity is converted to Double.MAX\_VALUE and whether -Infinity and NaN are converted to Double.MIN\_VALUE to ensure successfull delivery to BigQuery.                                                                                                                                                                                                                                                      | false                                                             |
| enableBatchLoad                      | \[Beta Feature] Use with caution. The sublist of topics to be batch loaded through GCS.                                                                                                                                                                                                                                                                                                                                          | ""                                                                |
| includeKafkaData                     | Whether to include an extra block containing the Kafka source topic, offset, and partition information in the resulting BigQuery rows.                                                                                                                                                                                                                                                                                           | false                                                             |
| upsertEnabled                        | Enable upsert functionality on the connector through the use of record keys, intermediate tables, and periodic merge flushes. Row-matching will be performed based on the contents of record keys. This feature won’t work with SMTs that change the name of the topic and doesn’t support JSON input.                                                                                                                           | false                                                             |
| deleteEnabled                        | Enable delete functionality on the connector through the use of record keys and intermediate tables. Row-matching will be performed based on the contents of record keys. This feature won’t work with SMTs that change the name of the topic and doesn’t support JSON input.                                                                                                                                                    | false                                                             |
| intermediateTableSuffix              | A suffix that will be appended to the names of destination tables to create the names for the corresponding intermediate tables. Multiple intermediate tables may be created for a single destination table, but their names will always start with the name of the destination table, followed by this suffix, and possibly followed by an additional suffix.                                                                   | "tmp"                                                             |
| mergeIntervalMs                      | The interval, in milliseconds, at which to attempt to merge intermediate tables into destination tables.                                                                                                                                                                                                                                                                                                                         | 60000                                                             |
| mergeRecordsThreshold                | The number of records that must be in an intermediate table before it is eligible for a merge operation.                                                                                                                                                                                                                                                                                                                         | -1                                                                |
| autoCreateBucket                     | Whether to create the GCS bucket if it doesn't exist.                                                                                                                                                                                                                                                                                                                                                                            | true                                                              |
| allowNewBigQueryFields               | Whether to allow new fields in BigQuery tables.                                                                                                                                                                                                                                                                                                                                                                                  | false                                                             |
| allowBigQueryRequiredFieldRelaxation | If true, fields in BigQuery Schema can be changed from REQUIRED to NULLABLE. Note that allowNewBigQueryFields and allowBigQueryRequiredFieldRelaxation replaced the autoUpdateSchemas parameter of older versions of this connector.                                                                                                                                                                                             | false                                                             |
| allowSchemaUnionization              | If true, the existing table schema (if one is present) will be unionized with new record schemas during schema updates. If false, the record of the last schema in a batch will be used for any necessary table creation and schema update attempts.                                                                                                                                                                             | false                                                             |
| kafkaDataFieldName                   | The Kafka data field name. The default value is null, which means the Kafka Data field will not be included.                                                                                                                                                                                                                                                                                                                     | null                                                              |
| kafkaKeyFieldName                    | The Kafka key field name. The default value is null, which means the Kafka Key field will not be included.                                                                                                                                                                                                                                                                                                                       | null                                                              |
| topic2TableMap                       | Map of topics to tables (optional). Format: comma-separated tuples, e.g. `<topic-1>:<table-1>,<topic-2>:<table-2>,..` Note that topic name should not be modified using regex SMT while using this option. Also note that SANITIZE\_TOPICS\_CONFIG would be ignored if this config is set. Lastly, if the topic2table map doesn’t contain the topic for a record, a table with the same name as the topic name would be created. | ""                                                                |


# Azure Cosmos DB Kafka Connect Sink
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-cosmosdb-sink/current/kafka-connect-cosmosdb-sink

The Azure Cosmos DB Kafka Connect Sink connector.

The Azure Cosmos DB sink connector reads data from Kafka topics and writes data to Azure Cosmos DB (SQL API). The sink connector fully supports exactly-once semantics.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

### Prerequisites

* An Azure Cosmos DB account with a database and container.
* The Cosmos DB endpoint URI (`connect.cosmos.connection.endpoint`).
* The Cosmos DB primary key (`connect.cosmos.master.key`).

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)

2. Create a JSON file like the following:
   ```json theme={null}
   {
       "name": "cosmosdb-sink",
       "config": {
           "connector.class": "com.azure.cosmos.kafka.connect.sink.CosmosDBSinkConnector",
           "connect.cosmos.connection.endpoint": "https://<cosmosinstance-name>.documents.azure.com:443/",
           "connect.cosmos.master.key": "<cosmosdbprimarykey>",
           "connect.cosmos.databasename": "kafkaconnect",
           "connect.cosmos.containers.topicmap": "hotels#kafka",
           "topics": "hotels",
           "value.converter": "org.apache.kafka.connect.json.JsonConverter",
           "value.converter.schemas.enable": "false",
           "key.converter": "org.apache.kafka.connect.json.JsonConverter",
           "key.converter.schemas.enable": "false",
           "tasks.max": "1"
       }
   }
   ```

3. Run the following command to create the connector:
   ```shell theme={null}
   kcctl apply -f <filename>.json
   ```

### Configuration

Configure the Azure Cosmos DB sink connector with the following properties:

| Property                                    | Required | Default | Description                                                                                                                                                           |
| ------------------------------------------- | -------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `connector.class`                           | true     |         | Classname of the Cosmos DB sink. Should be set to `com.azure.cosmos.kafka.connect.sink.CosmosDBSinkConnector`.                                                        |
| `connect.cosmos.connection.endpoint`        | true     |         | Cosmos endpoint URI string.                                                                                                                                           |
| `connect.cosmos.master.key`                 | true     |         | The Cosmos primary key that the sink connects with.                                                                                                                   |
| `connect.cosmos.databasename`               | true     |         | The name of the Cosmos database the sink writes to.                                                                                                                   |
| `connect.cosmos.containers.topicmap`        | true     |         | Mapping between Kafka Topics and Cosmos Containers, formatted using CSV as shown: `topic#container,topic2#container2`.                                                |
| `topics`                                    | true     |         | A list of Kafka topics to watch.                                                                                                                                      |
| `connect.cosmos.connection.gateway.enabled` | false    | false   | Flag to indicate whether to use gateway mode.                                                                                                                         |
| `connect.cosmos.sink.bulk.enabled`          | false    | true    | Flag to indicate whether bulk mode is enabled.                                                                                                                        |
| `connect.cosmos.sink.maxRetryCount`         | false    | 10      | Max retry attempts on transient write failures. NOTE: This is different from max throttling retry attempts, which are infinite.                                       |
| `connect.cosmos.connection.sharing.enabled` | false    | false   | Flag to enable connection sharing between instances of cosmos clients on the same JVM. NOTE: If gateway mode is enabled, this configure will not make any difference. |
| `key.converter`                             | true     |         | Serialization format for the key data written into Kafka topic.                                                                                                       |
| `value.converter`                           | true     |         | Serialization format for the value data written into the Kafka topic.                                                                                                 |
| `key.converter.schemas.enable`              | false    | true    | Set to `"true"` if the key data has embedded schema.                                                                                                                  |
| `value.converter.schemas.enable`            | false    | true    | Set to `"true"` if the value data has embedded schema.                                                                                                                |
| `tasks.max`                                 | false    | 1       | Maximum number of connector sink tasks.                                                                                                                               |

### Supported Data Formats

The sink connector supports the following data formats:

| Format Name      | Description                                                                                                                                                                |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| JSON (Plain)     | JSON record structure without any attached schema.                                                                                                                         |
| JSON with Schema | JSON record structure with explicit schema information to ensure the data matches the expected format.                                                                     |
| AVRO             | A row-oriented remote procedure call and data serialization framework. It uses JSON for defining data types and protocols, and serializes data in a compact binary format. |

### Supported Data Types

Azure Cosmos DB sink connector converts SinkRecord into JSON Document supporting the following schema types:

| Schema Type | JSON Data Type |
| ----------- | -------------- |
| Array       | Array          |
| Boolean     | Boolean        |
| Float32     | Number         |
| Float64     | Number         |
| Int8        | Number         |
| Int16       | Number         |
| Int32       | Number         |
| Int64       | Number         |
| Map         | Object (JSON)  |
| String      | String         |
| Struct      | Object (JSON)  |

For full details, see the [Azure Cosmos DB sink connector documentation](https://github.com/microsoft/kafka-connect-cosmosdb/blob/main/doc/README_Sink.md)


# Azure Cosmos DB Kafka Connect Source
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-cosmosdb-source/current/kafka-connect-cosmosdb-source

The Azure Cosmos DB Kafka Connect Source connector.

The Azure Cosmos DB source connector reads data from Azure Cosmos DB (SQL API) change feed and writes data to Kafka topics. The source connector supports at-least once with multiple tasks and exactly-once for single tasks.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

### Prerequisites

* An Azure Cosmos DB account with a database and container.
* The Cosmos DB endpoint URI (`connect.cosmos.connection.endpoint`).
* The Cosmos DB primary key (`connect.cosmos.master.key`).

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)

2. Create a JSON file like the following:
   ```json theme={null}
   {
       "name": "cosmosdb-source",
       "config": {
           "connector.class": "com.azure.cosmos.kafka.connect.source.CosmosDBSourceConnector",
           "connect.cosmos.connection.endpoint": "https://<cosmosinstance-name>.documents.azure.com:443/",
           "connect.cosmos.master.key": "<cosmosdbprimarykey>",
           "connect.cosmos.databasename": "kafkaconnect",
           "connect.cosmos.containers.topicmap": "apparels#kafka",
           "connect.cosmos.task.poll.interval": "100",
           "connect.cosmos.offset.useLatest": false,
           "value.converter": "org.apache.kafka.connect.json.JsonConverter",
           "value.converter.schemas.enable": "false",
           "key.converter": "org.apache.kafka.connect.json.JsonConverter",
           "key.converter.schemas.enable": "false",
           "tasks.max": "1"
       }
   }
   ```

3. Run the following command to create the connector:
   ```shell theme={null}
   kcctl apply -f <filename>.json
   ```

### Configuration

Configure the Azure Cosmos DB source connector with the following properties:

| Property                                    | Required | Default | Description                                                                                                                                           |
| ------------------------------------------- | -------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `connector.class`                           | true     |         | Classname of the Cosmos DB source. Should be set to `com.azure.cosmos.kafka.connect.source.CosmosDBSourceConnector`.                                  |
| `connect.cosmos.connection.endpoint`        | true     |         | Cosmos endpoint URI string.                                                                                                                           |
| `connect.cosmos.master.key`                 | true     |         | The Cosmos primary key that the source connects with.                                                                                                 |
| `connect.cosmos.databasename`               | true     |         | The name of the Cosmos database to read from.                                                                                                         |
| `connect.cosmos.containers.topicmap`        | true     |         | Mapping between Kafka Topics and Cosmos Containers, formatted using CSV as shown: `topic#container,topic2#container2`.                                |
| `connect.cosmos.task.poll.interval`         | true     |         | Interval (in milliseconds) to poll the change feed container for changes.                                                                             |
| `connect.cosmos.connection.gateway.enabled` | false    | false   | Flag to indicate whether to use gateway mode.                                                                                                         |
| `connect.cosmos.messagekey.enabled`         | true     | true    | Set if the Kafka message key should be set.                                                                                                           |
| `connect.cosmos.messagekey.field`           | true     | id      | Use the field's value from the document as the message key.                                                                                           |
| `connect.cosmos.offset.useLatest`           | true     | false   | Set to `"true"` to use the latest (most recent) source offset, `"false"` to use the earliest recorded offset.                                         |
| `key.converter`                             | true     |         | Serialization format for the key data written into Kafka topic.                                                                                       |
| `value.converter`                           | true     |         | Serialization format for the value data written into the Kafka topic.                                                                                 |
| `key.converter.schemas.enable`              | false    | true    | Set to `"true"` if the key data has embedded schema.                                                                                                  |
| `value.converter.schemas.enable`            | false    | true    | Set to `"true"` if the value data has embedded schema.                                                                                                |
| `tasks.max`                                 | false    | 1       | Maximum number of connector source tasks. This should be set to equal to or greater than the number of containers specified in the topicmap property. |

### Supported Data Formats

The source connector supports the following data formats:

| Format Name      | Description                                                                                                                                                                |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| JSON (Plain)     | JSON record structure without any attached schema.                                                                                                                         |
| JSON with Schema | JSON record structure with explicit schema information to ensure the data matches the expected format.                                                                     |
| AVRO             | A row-oriented remote procedure call and data serialization framework. It uses JSON for defining data types and protocols, and serializes data in a compact binary format. |

### Supported Data Types

Azure Cosmos DB source connector converts JSON Document to Schema supporting the following data types:

| JSON Data Type | Schema Type                                 |
| -------------- | ------------------------------------------- |
| Array          | Array                                       |
| Boolean        | Boolean                                     |
| Number         | Float32, Float64, Int8, Int16, Int32, Int64 |
| Null           | String                                      |
| Object (JSON)  | Struct                                      |
| String         | String                                      |

For full details, see the [Azure Cosmos DB source connector documentation](https://github.com/microsoft/kafka-connect-cosmosdb/blob/main/doc/README_Source.md)


# Kafka connect datagen source
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-datagen/current/kafka-connect-datagen-source

kafka-connect-datagen is a Kafka Connect connector for generating mock data for testing and is not suitable for production scenarios.

`kafka-connect-datagen` is a Kafka Connect connector for generating mock data for testing and is not suitable for production scenarios. It is available in the StreamNative Cloud.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)
2. Create a json file like below:
   ```
   {
       "name": "datagen",
       "config": {
           "connector.class": "io.confluent.kafka.connect.datagen.DatagenConnector",
           "kafka.topic": "users",
           "quickstart": "users",
           "key.converter": "org.apache.kafka.connect.storage.StringConverter",
           "value.converter": "org.apache.kafka.connect.json.JsonConverter",
           "value.converter.schemas.enable": "false",
           "max.interval": 1000,
           "iterations": 10000000,
           "tasks.max": "1"
       }
   }
   ```
3. Run the following command to create the connector:
   ```
   kcctl apply -f <filename>.json
   ```

### Configuration

The `kafka-connect-datagen` connector is configured using the following properties:

| Parameter         | Description                                                                                                                                                                | Default |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `kafka.topic`     | Topic to write to                                                                                                                                                          |         |
| `max.interval`    | Max interval between messages (ms)                                                                                                                                         | 500     |
| `iterations`      | Number of messages to send from each task, or less than 1 for unlimited                                                                                                    | -1      |
| `schema.string`   | The literal JSON-encoded Avro schema to use. Cannot be set with `schema.filename` or `quickstart`.                                                                         |         |
| `schema.filename` | Filename of schema to use. Cannot be set with `schema.string` or `quickstart`. This config is not enabled on StreamNative Cloud.                                           |         |
| `schema.keyfield` | Name of field to use as the message key                                                                                                                                    |         |
| `quickstart`      | Name of [quickstart](https://github.com/confluentinc/kafka-connect-datagen/tree/v0.6.5/src/main/resources) to use. Cannot be set with `schema.string` or `schema.filename` |         |

For full details, see the [offical kafka-connect-datagen documentation](https://github.com/confluentinc/kafka-connect-datagen/blob/v0.6.5/README.md).

### Using the bundled schema

The `quickstart` property can be set to one of the following values:

| Value                         | Description                                 |
| ----------------------------- | ------------------------------------------- |
| `clickstream_codes`           | Generates clickstream codes data            |
| `clickstream`                 | Generates clickstream data                  |
| `clickstream_users`           | Generates clickstream users data            |
| `orders`                      | Generates order data                        |
| `ratings`                     | Generates ratings data                      |
| `users`                       | Generates user data                         |
| `users_`                      | Generates alternative user data             |
| `pageviews`                   | Generates pageview data                     |
| `stock_trades`                | Generates stock trade data                  |
| `inventory`                   | Generates inventory data                    |
| `product`                     | Generates product data                      |
| `purchases`                   | Generates purchase data                     |
| `transactions`                | Generates transaction data                  |
| `stores`                      | Generates store data                        |
| `creadit_cards`               | Generates credit card data                  |
| `campaign_finance`            | Generates campaign finance data             |
| `fleet_mgmt_description`      | Generates fleet management description data |
| `fleet_mgmt_location`         | Generates fleet management location data    |
| `fleet_mgmt_sensors`          | Generates fleet management sensor data      |
| `pizza_orders`                | Generates pizza order data                  |
| `pizza_orders_completed`      | Generates completed pizza order data        |
| `pizza_orders_cancelled`      | Generates cancelled pizza order data        |
| `insurance_offers`            | Generates insurance offer data              |
| `insurance_customers`         | Generates insurance customer data           |
| `insurance_customer_activity` | Generates insurance customer activity data  |
| `gaming_games`                | Generates gaming game data                  |
| `gaming_players`              | Generates gaming player data                |
| `gaming_player_activity`      | Generates gaming player activity data       |
| `payroll_employee`            | Generates payroll employee data             |
| `payroll_empolyee_location`   | Generates payroll employee location data    |
| `payroll_bonus`               | Generates payroll bonus data                |
| `syslog_logs`                 | Generates syslog log data                   |
| `device_information`          | Generates device information data           |
| `siem_logs`                   | Generates SIEM log data                     |
| `shoes`                       | Generates shoe data                         |
| `shoe_customers`              | Generates shoe customer data                |
| `shoe_orders`                 | Generates shoe order data                   |
| `shoe_clickstream`            | Generates shoe clickstream data             |


# Kafka connect debezium jdbc sink
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-debezium-jdbc-sink/current/kafka-connect-debezium-jdbc-sink

Debezium JDBC Sink connector

The Debezium JDBC connector is a Kafka Connect sink connector implementation that can consume events from multiple source topics, and then write those events to a relational database by using a JDBC driver. This connector supports a wide variety of database dialects, including Db2, MySQL, Oracle, PostgreSQL, and SQL Server.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)
2. Create a JSON file like the following:

   ```json theme={null}
   {
       "name": "debezium-jdbc-sink",
       "config": {
           "connector.class": "io.debezium.connector.jdbc.JdbcSinkConnector",
           "tasks.max": "1",
           "connection.url": "jdbc:postgresql://hostname:port/db",
           "connection.user": "user",
           "connection.password": "password",
           "insert.mode": "upsert",
           "delete.enabled": "true",
           "primary.key.mode": "record_key",
           "schema.evolution": "basic",
           "use.time.zone": "UTC",
           "topics": "orders"
       }
   }
   ```
3. Run the following command to create the connector:

   ```bash theme={null}
   kcctl apply -f <filename>.json
   ```

### Configuration

The Debezium JDBC Sink connector is configured using the following properties:

| Property                           | Required | Default                                                            | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| ---------------------------------- | -------- | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| name                               | true     | No default                                                         | Unique name for the connector. A failure results if you attempt to reuse this name when registering a connector. This property is required by all Kafka Connect connectors.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| connector.class                    | true     | No default                                                         | The name of the Java class for the connector. For the Debezium JDBC connector, specify the value io.debezium.connector.jdbc.JdbcSinkConnector.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| tasks.max                          | true     | 1                                                                  | Maximum number of tasks to use for this connector.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| topics                             | false    | No default                                                         | List of topics to consume, separated by commas. Do not use this property in combination with the topics.regex property.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| topics.regex                       | false    | No default                                                         | A regular expression that specifies the topics to consume. Internally, the regular expression is compiled to a java.util.regex.Pattern. Do not use this property in combination with the topics property.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| connection.provider                | false    | org.hibernate.c3p0.internal.C3P0ConnectionProvider                 | The connection provider implementation to use.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| connection.url                     | true     | No default                                                         | The JDBC connection URL used to connect to the database.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| connection.username                | true     | No default                                                         | The name of the database user account that the connector uses to connect to the database.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| connection.password                | true     | No default                                                         | The password that the connector uses to connect to the database.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| connection.pool.min\_size          | false    | 5                                                                  | Specifies the minimum number of connections in the pool.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| connection.pool.max\_size          | false    | 32                                                                 | Specifies the maximum number of concurrent connections that the pool maintains.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| connection.pool.acquire\_increment | false    | 32                                                                 | Specifies the number of connections that the connector attempts to acquire if the connection pool exceeds its maximum size.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| connection.pool.timeout            | false    | 1800                                                               | Specifies the number of seconds that an unused connection is kept before it is discarded.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| connection.restart.on.errors       | false    | false                                                              | Specifies whether the connector retries after a transient JDBC connection error.<br /><br />When enabled (true), the connector treats connection issues (such as socket closures or timeouts) as retriable, allowing it to retry processing instead of failing the task. This reduces downtime and improves resilience against temporary disruptions.<br /><br />Setting this option to true can reduce downtime. However, in master-replica environments with asynchronous replication, it may lead to data loss if retries occur before all changes are fully replicated.<br /><br />Use with caution where strong data consistency is required.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| use.time.zone                      | false    | UTC                                                                | Specifies the timezone used when inserting JDBC temporal values.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| delete.enabled                     | false    | false                                                              | Specifies whether the connector processes DELETE or tombstone events and removes the corresponding row from the database. Use of this option requires that you set the primary.key.mode to record.key.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| truncate.enabled                   | false    | false                                                              | Specifies whether the connector processes TRUNCATE events and truncates the corresponding tables from the database.<br /><br />Although support for TRUNCATE statements has been available in Db2 since version 9.7, currently, the JDBC connector is unable to process standard TRUNCATE events that the Db2 connector emits.<br /><br />To ensure that the JDBC connector can process TRUNCATE events received from Db2, perform the truncation by using an alternative to the standard TRUNCATE TABLE statement. For example:<br /><br />ALTER TABLE \[table\_name] ACTIVATE NOT LOGGED INITIALLY WITH EMPTY TABLE<br /><br />The user account that submits the preceding query requires ALTER privileges on the table to be truncated.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| insert.mode                        | false    | insert                                                             | Specifies the strategy used to insert events into the database. The following options are available:<br /><br />insert<br /><br />Specifies that all events should construct INSERT-based SQL statements. Use this option only when no primary key is used, or when you can be certain that no updates can occur to rows with existing primary key values.<br /><br />update<br /><br />Specifies that all events should construct UPDATE-based SQL statements. Use this option only when you can be certain that the connector receives only events that apply to existing rows.<br /><br />upsert<br /><br />Specifies that the connector adds events to the table using upsert semantics. That is, if the primary key does not exist, the connector performs an INSERT operation, and if the key does exist, the connector performs an UPDATE operation. When idempotent writes are required, the connector should be configured to use this option.                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| primary.key.mode                   | false    | none                                                               | Specifies how the connector resolves the primary key columns from the event.<br /><br />none<br /><br />Specifies that no primary key columns are created.<br /><br />kafka<br /><br />Specifies that the connector uses Kafka coordinates as the primary key columns. The key coordinates are defined from the topic name, partition, and offset of the event, and are mapped to columns with the following names:<br /><br />\_\_connect\_topic<br /><br />\_\_connect\_partition<br /><br />\_\_connect\_offset<br /><br />record\_key<br /><br />Specifies that the primary key columns are sourced from the event’s record key. If the record key is a primitive type, the primary.key.fields property is required to specify the name of the primary key column. If the record key is a struct type, the primary.key.fields property is optional, and can be used to specify a subset of columns from the event’s key as the table’s primary key.<br /><br />record\_value<br /><br />Specifies that the primary key columns is sourced from the event’s value. You can set the primary.key.fields property to define the primary key as a subset of fields from the event’s value; otherwise all fields are used by default.                                                                                                                                                                |
| primary.key.fields                 | false    | No default                                                         | Either the name of the primary key column or a comma-separated list of fields to derive the primary key from.<br /><br />When primary.key.mode is set to record\_key and the event’s key is a primitive type, it is expected that this property specifies the column name to be used for the key.<br /><br />When the primary.key.mode is set to record\_key with a non-primitive key, or record\_value, it is expected that this property specifies a comma-separated list of field names from either the key or value. If the primary.key.mode is set to record\_key with a non-primitive key, or record\_value, and this property is not specified, the connector derives the primary key from all fields of either the record key or record value, depending on the specified mode.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| quote.identifiers                  | false    | false                                                              | Specifies whether generated SQL statements use quotation marks to delimit table and column names. See the JDBC quoting case-sensitivity section for more details.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| schema.evolution                   | false    | none                                                               | Specifies how the connector evolves the destination table schemas. For more information, see Schema evolution. The following options are available:<br /><br />none<br /><br />Specifies that the connector does not evolve the destination schema.<br /><br />basic<br /><br />Specifies that basic evolution occurs. The connector adds missing columns to the table by comparing the incoming event’s record schema to the database table structure.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| collection.name.format             | false    | `${topic}`                                                         | Specifies a string pattern that the connector uses to construct the names of destination tables.<br />When the property is set to its default value, `${topic}`, after the connector reads an event from Kafka, it writes the event record to a destination table with a name that matches the name of the source topic.<br /><br />You can also configure this property to extract values from specific fields in incoming event records and then use those values to dynamically generate the names of target tables. This ability to generate table names from values in the message source would otherwise require the use of a custom Kafka Connect single message transformation (SMT).<br /><br />To configure the property to dynamically generate the names of destination tables, set its value to a pattern such as `${source._field_}`. When you specify this type of pattern, the connector extracts values from the source block of the Debezium change event, and then uses those values to construct the table name. For example, you might set the value of the property to the pattern `${source.schema}_${source.table}`. Based on this pattern, if the connector reads an event in which the schema field in the source block contains the value, user, and the table field contains the value, tab, the connector writes the event record to a table with the name user\_tab. |
| dialect.postgres.postgis.schema    | false    | public                                                             | Specifies the schema name where the PostgreSQL PostGIS extension is installed. The default is public; however, if the PostGIS extension was installed in another schema, this property should be used to specify the alternate schema name.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| dialect.sqlserver.identity.insert  | false    | false                                                              | Specifies whether the connector automatically sets an IDENTITY\_INSERT before an INSERT or UPSERT operation into the identity column of SQL Server tables, and then unsets it immediately after the operation. When the default setting (false) is in effect, an INSERT or UPSERT operation into the IDENTITY column of a table results in a SQL exception.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| batch.size                         | false    | 500                                                                | Specifies how many records to attempt to batch together into the destination table.<br /><br />Note that if you set consumer.max.poll.records in the Connect worker properties to a value lower than batch.size, batch processing will be caped by consumer.max.poll.records and the desired batch.size won’t be reached. You can also configure the connector’s underlying consumer’s max.poll.records using consumer.override.max.poll.records in the connector configuration.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| use.reduction.buffer               | false    | false                                                              | Specifies whether to enable the Debezium JDBC connector’s reduction buffer.<br /><br />Choose one of the following settings:<br /><br />false<br /><br />(default) The connector writes each change event that it consumes from Kafka as a separate logical SQL change.<br /><br />true<br /><br />The connector uses the reduction buffer to reduce change events before it writes them to the sink database. That is, if multiple events refer to the same primary key, the connector consolidates the SQL queries and writes only a single logical SQL change, based on the row state that is reported in the most recent offset record.<br />Choose this option to reduce the SQL load on the target database.<br /><br />To optimize query processing in a PostgreSQL sink database when the reduction buffer is enabled, you must also enable the database to execute the batched queries by adding the reWriteBatchedInserts parameter to the JDBC connection URL.                                                                                                                                                                                                                                                                                                                                                                                                                          |
| field.include.list                 | false    | empty string                                                       | An optional, comma-separated list of field names that match the fully-qualified names of fields to include from the change event value. Fully-qualified names for fields are of the form fieldName or topicName:*fieldName*.<br /><br />If you include this property in the configuration, do not set the field.exclude.list property.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| field.exclude.list                 | false    | empty string                                                       | An optional, comma-separated list of field names that match the fully-qualified names of fields to exclude from the change event value. Fully-qualified names for fields are of the form fieldName or topicName:*fieldName*.<br /><br />If you include this property in the configuration, do not set the field.include.list property.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| flush.max.retries                  | false    | 5                                                                  | Specifies the maximum number of retries that the connector performs after an attempt to flush changes to the target database results in certain database errors. If the number of retries exceeds the retry value, the sink connector enters a FAILED state.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| flush.retry.delay.ms               | false    | 1000                                                               | Specifies the number of milliseconds that the connector waits to retry a flush operation that failed.<br /><br />When you set both the flush.retry.delay.ms and flush.max.retries properties, it can affect the behavior of the Kafka max.poll.interval.ms property. To prevent the connector from rebalancing, set the total retry time (flush.retry.delay.ms \* flush.max.retries) to a value that is less than the value of max.poll.interval.ms (default is 5 minutes).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| column.naming.strategy             | false    | io.debezium.connector.jdbc.naming.DefaultColumnNamingStrategy      | Specifies the fully-qualified class name of a ColumnNamingStrategy implementation that the connector uses to resolve column names from incoming event field names.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| collection.naming.strategy         | false    | io.debezium.connector.jdbc.nnaming.DefaultCollectionNamingStrategy | Specifies the fully-qualified class name of a CollectionNamingStrategy implementation that the connector uses to resolve table names from incoming event topic names.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |

For more information about the configuration properties, see the [Official Debezium JDBC Sink Connector documentation](https://debezium.io/documentation/reference/stable/connectors/jdbc.html#jdbc-connector-configuration).


# Kafka connect debezium mongodb
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-debezium-mongodb/current/kafka-connect-debezium-mongodb

Kafka Connect Debezium MongoDB Source connector

The Debezium MongoDB Source connector is a Kafka Connect connector that captures document-level changes in a MongoDB database and streams them to Kafka topics.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

### Prerequisites

* A running MongoDB replica set or sharded cluster

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)
2. Create a JSON file like the following:

   ```json theme={null}
   {
       "name": "debezium-mongodb-source",
       "config": {
           "connector.class": "io.debezium.connector.mongodb.MongoDbConnector",
           "tasks.max": "1",
           "mongodb.connection.string": "mongodb://{host}:27017/?replicaSet=rs0",
           "mongodb.user": "{username}",
           "mongodb.password": "{password}",
           "database.include.list": "db1,db2",
           "topic.prefix": "my_prefix"
       }
   }
   ```
3. Run the following command to create the connector:

   ```bash theme={null}
   kcctl apply -f <filename>.json
   ```

### Configuration

The Debezium MongoDB Source connector is configured using the following properties:

| Property                                                   | Required | Default                                              | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| ---------------------------------------------------------- | -------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| name                                                       | true     | No default                                           | Unique name for the connector. Attempting to register again with the same name will fail. (This property is required by all Kafka Connect connectors.)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| connector.class                                            | true     | No default                                           | The name of the Java class for the connector. Always use a value of io.debezium.connector.mongodb.MongoDbConnector for the MongoDB connector.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| mongodb.connection.string                                  | true     | No default                                           | Specifies a connection string that the connector uses to connect to a MongoDB replica set. This property replaces the mongodb.hosts property that was available in previous versions of the MongoDB connector.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| topic.prefix                                               | true     | No default                                           | A unique name that identifies the connector and/or MongoDB replica set or sharded cluster that this connector monitors. Each server should be monitored by at most one Debezium connector, since this server name prefixes all persisted Kafka topics emanating from the MongoDB replica set or cluster. Use only alphanumeric characters, hyphens, dots and underscores to form the name. The logical name should be unique across all other connectors, because the name is used as the prefix in naming the Kafka topics that receive records from this connector.<br /><br /><br /><br />Do not change the value of this property. If you change the name value, after a restart, instead of continuing to emit events to the original topics, the connector emits subsequent events to topics whose names are based on the new value.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| internal.mongodb.allow\.offset.invalidation                | false    | false                                                | Set this property to true to enable the connector to invalidate and consolidate shard-specific offsets that were recorded by earlier connector versions.<br /><br />This property permits you to modify the current default behavior. The property is subject to removal in a future release if the default behavior changes to permit the connector to automatically invalidate and consolidate offsets that are recorded by earlier connector versions.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| mongodb.authentication.class                               | false    | DefaultMongoDbAuthProvider                           | A full Java class name that is an implementation of the io.debezium.connector.mongodb.connection.MongoDbAuthProvider interface. This class handles setting the credentials on the MongoDB connection (called on each app boot). Default behavior uses the mongodb.user, mongodb.password, and mongodb.authsource properties according to each of their documentation, but other implementations may use them differently or ignore them altogether. Note that any setting in mongodb.connection.string will override settings set by this class                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| mongodb.user                                               | false    | No default                                           | When using default mongodb.authentication.class: Name of the database user to be used when connecting to MongoDB. This is required only when MongoDB is configured to use authentication.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| mongodb.password                                           | false    | No default                                           | When using default mongodb.authentication.class: Password to be used when connecting to MongoDB. This is required only when MongoDB is configured to use authentication.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| mongodb.authsource                                         | false    | admin                                                | When using default mongodb.authentication.class: Database (authentication source) containing MongoDB credentials. This is required only when MongoDB is configured to use authentication with another authentication database than admin.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| mongodb.ssl.enabled                                        | false    | false                                                | Connector will use SSL to connect to MongoDB instances.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| mongodb.ssl.invalid.hostname.allowed                       | false    | false                                                | When SSL is enabled this setting controls whether strict hostname checking is disabled during connection phase. If true the connection will not prevent man-in-the-middle attacks.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| filters.match.mode                                         | false    | regex                                                | The mode used to match events based on included/excluded database and collection names. Set the property to one of the following values:<br /><br />**regex**: Database and collection includes/excludes are evaluated as comma-separated list of regular expressions.<br /><br />**literal**: Database and collection includes/excludes are evaluated as comma-separated list of string literals. Whitespace characters surrounding these literals are stripped.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| database.include.list                                      | false    | empty string                                         | An optional comma-separated list of regular expressions or literals that match database names to be monitored. By default, all databases are monitored.<br />When database.include.list is set, the connector monitors only the databases that the property specifies. Other databases are excluded from monitoring.<br /><br />To match the name of a database, Debezium performs one of the following actions based on the value of filters.match.mode property<br /><br />applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire name string of the database; it does not match substrings that might be present in a database name.<br /><br />compares the literals that you specify with the entire name string of the database<br /><br /><br />If you include this property in the configuration, do not also set the database.exclude.list property.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| database.exclude.list                                      | false    | empty string                                         | An optional comma-separated list of regular expressions or literals that match database names to be excluded from monitoring. When database.exclude.list is set, the connector monitors every database except the ones that the property specifies.<br /><br />To match the name of a database, Debezium performs one of the following actions based on the value of filters.match.mode property<br /><br />applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire name string of the database; it does not match substrings that might be present in a database name.<br /><br />compares the literals that you specify with the entire name string of the database<br /><br /><br />If you include this property in the configuration, do not set the database.include.list property.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| collection.include.list                                    | false    | empty string                                         | An optional comma-separated list of regular expressions or literals that match fully-qualified namespaces for MongoDB collections to be monitored. By default, the connector monitors all collections except those in the local and admin databases. When collection.include.list is set, the connector monitors only the collections that the property specifies. Other collections are excluded from monitoring. Collection identifiers are of the form databaseName.collectionName.<br /><br />To match the name of a namespace, Debezium performs one of the following actions based on the value of filters.match.mode property<br /><br />applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire name string of the namespace; it does not match substrings in the name.<br /><br />compares the literals that you specify with the entire name string of the namespace<br /><br /><br />If you include this property in the configuration, do not also set the collection.exclude.list property.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| collection.exclude.list                                    | false    | empty string                                         | An optional comma-separated list of regular expressions or literals that match fully-qualified namespaces for MongoDB collections to be excluded from monitoring. When collection.exclude.list is set, the connector monitors every collection except the ones that the property specifies. Collection identifiers are of the form databaseName.collectionName.<br /><br /><br />To match the name of a namespace, Debezium performs one of the following actions based on the value of filters.match.mode property<br /><br />applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire name string of the namespace; it does not match substrings that might be present in a database name.<br /><br />compares the literals that you specify with the entire name string of the namespace<br /><br /><br />If you include this property in the configuration, do not set the collection.include.list property.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| capture.mode                                               | false    | change\_streams\_update\_full                        | Specifies the method that the connector uses to capture update event changes from a MongoDB server. Set this property to one of the following values:<br /><br />**change\_streams**: update event messages do not include the full document. Messages do not include a field that represents the state of the document before the change.<br /><br />**change\_streams\_update\_full**: update event messages include the full document. Messages do not include a before field that represents the state of the document before the update. The event message returns the full state of the document in the after field. Set capture.mode.full.update.type to specify how the connector fetches full documents from the database.<br /><br />In some situations, when capture.mode is configured to return full documents, the updateDescription and after fields of the update event message might report inconsistent values. Such discrepancies can result after multiple updates are applied to a document in rapid succession. The connector requests the full document from the MongoDB database only after it receives the update described in the event’s updateDescription field. If a later update modifies the source document before the connector can retrieve it from the database, the connector receives the document that is modified by this later update.<br /><br />**change\_streams\_update\_full\_with\_pre\_image**: update event event messages include the full document, and include a field that represents the state of the document before the change. Set capture.mode.full.update.type to specify how the connector fetches full documents from the database.<br /><br />**change\_streams\_with\_pre\_image**: update events do not include the full document, but include a field that represents the state of the document before the change.                                                                                                                                                                                                                                                         |
| capture.scope                                              | false    | deployment                                           | Specifies the scope of the change streams that the connector opens. Set this property to one of the following values:<br /><br />**deployment**: Opens a change stream cursor for a deployment (either a replica set or a sharded cluster) to watch for changes to all non-system collections across all databases, except for admin, local, and config.<br /><br />**database**: Opens a change stream cursor for a single database to watch for changes to all of its non-system collections.<br /><br />To support Debezium signaling, if you set capture.scope to database, the signaling data collection must reside in a database that is specified by the capture.target property.<br /><br />**collection**: Opens a change stream cursor for a single collection to watch for changes to that collection.<br /><br />This feature is currently in an incubating state. The exact semantics, configuration options, and so forth are subject to change, based on the feedback that we receive.<br /><br />Setting the value of the capture.scope property to collection prevents the connector from using the default source signaling channel. Because the source channel must be enabled to permit connectors to process incremental snapshot signals — even for signals are sent over the Kafka, JMX, or File channels — the connector cannot perform incremental snapshots when capture-scope is set to collection.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| capture.target                                             | false    |                                                      | Specifies the database that the connector monitors for changes. This property applies only if the capture.scope is set to database.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| field.exclude.list                                         | false    | empty string                                         | An optional comma-separated list of the fully-qualified names of fields that should be excluded from change event message values. Fully-qualified names for fields are of the form databaseName.collectionName.fieldName.nestedFieldName, where databaseName and collectionName may contain the wildcard (\*) which matches any characters.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| field.renames                                              | false    | empty string                                         | An optional comma-separated list of the fully-qualified replacements of fields that should be used to rename fields in change event message values. Fully-qualified replacements for fields are of the form databaseName.collectionName.fieldName.nestedFieldName:newNestedFieldName, where databaseName and collectionName may contain the wildcard (\*) which matches any characters, the colon character (:) is used to determine rename mapping of field. The next field replacement is applied to the result of the previous field replacement in the list, so keep this in mind when renaming multiple fields that are in the same path.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| tombstones.on.delete                                       | false    | true                                                 | Controls whether a delete event is followed by a tombstone event.<br /><br />true - a delete operation is represented by a delete event and a subsequent tombstone event.<br /><br />false - only a delete event is emitted.<br /><br />After a source record is deleted, emitting a tombstone event (the default behavior) allows Kafka to completely delete all events that pertain to the key of the deleted row in case log compaction is enabled for the topic.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| schema.name.adjustment.mode                                | false    | none                                                 | Specifies how schema names should be adjusted for compatibility with the message converter used by the connector. Possible settings:<br /><br /><br />none does not apply any adjustment.<br /><br /><br />avro replaces the characters that cannot be used in the Avro type name with underscore.<br /><br /><br />avro\_unicode replaces the underscore or characters that cannot be used in the Avro type name with corresponding unicode like \_uxxxx. Note: \_ is an escape sequence like backslash in Java                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| field.name.adjustment.mode                                 | false    | none                                                 | Specifies how field names should be adjusted for compatibility with the message converter used by the connector. Possible settings:<br /><br /><br />none does not apply any adjustment.<br /><br /><br />avro replaces the characters that cannot be used in the Avro type name with underscore.<br /><br /><br />avro\_unicode replaces the underscore or characters that cannot be used in the Avro type name with corresponding unicode like \_uxxxx. Note: \_ is an escape sequence like backslash in Java<br /><br /><br />See Avro naming for more details.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| capture.mode.full.update.type                              | false    | lookup                                               | Specifies how the connector looks up the full value of an updated document when the capture.mode is set retrieve full documents. The connector retrieves full documents when its capture.mode is set to one of the following options:<br /><br />change\_streams\_update\_full<br /><br />change\_streams\_update\_full\_with\_pre-image<br /><br />To use this option with a MongoDB change streams collection, you must configure the collection to return document pre- and post-images. Pre- and post-images for an operation are available only if the required configuration is in place before the operation occurs.<br /><br />Set this property to one of the following values:<br /><br />**lookup**: The connector uses a separate lookup to fetch the updated full MongoDB document.<br /><br />If the lookup process fails to retrieve a document, it cannot populate the full document to the after state in the event payload. In such a situation, the connector emits an event message that contains a null value in the after field.<br /><br />Failed lookups can occur because a delete operation removed the document immediately after it was created, or because a change to the sharding key results in the document being moved to a different location. Sharding key changes can result when you modify any of the properties that make up the key.<br /><br />**post\_image**: The connector uses MongoDB post images to populate events with the full MongoDB document. The database must be running MongoDB 6.0 or later to use this option.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| max.batch.size                                             | false    | 2048                                                 | Positive integer value that specifies the maximum size of each batch of events that should be processed during each iteration of this connector. Defaults to 2048.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| max.queue.size                                             | false    | 8192                                                 | Positive integer value that specifies the maximum number of records that the blocking queue can hold. When Debezium reads events streamed from the database, it places the events in the blocking queue before it writes them to Kafka. The blocking queue can provide backpressure for reading change events from the database in cases where the connector ingests messages faster than it can write them to Kafka, or when Kafka becomes unavailable. Events that are held in the queue are disregarded when the connector periodically records offsets. Always set the value of max.queue.size to be larger than the value of max.batch.size.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| max.queue.size.in.bytes                                    | false    | 0                                                    | A long integer value that specifies the maximum volume of the blocking queue in bytes. By default, volume limits are not specified for the blocking queue. To specify the number of bytes that the queue can consume, set this property to a positive long value.<br />If max.queue.size is also set, writing to the queue is blocked when the size of the queue reaches the limit specified by either property. For example, if you set max.queue.size=1000, and max.queue.size.in.bytes=5000, writing to the queue is blocked after the queue contains 1000 records, or after the volume of the records in the queue reaches 5000 bytes.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| connect.max.attempts                                       | false    | 16                                                   | Positive integer value that specifies the maximum number of failed connection attempts to a replica set primary before an exception occurs and task is aborted. Defaults to 16, which with the defaults for connect.backoff.initial.delay.ms and connect.backoff.max.delay.ms results in just over 20 minutes of attempts before failing.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| mongodb.ssl.keystore                                       | false    | No Default                                           | An optional setting that specifies the location of the key store file. A key store file can be used for two-way authentication between the client and the MongoDB server.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| mongodb.ssl.keystore.password                              | false    | No Default                                           | The password for the key store file. Specify a password only if the mongodb.ssl.keystore is configured.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| mongodb.ssl.keystore.type                                  | false    | No Default                                           | The type of key store file. Specify a type only if the mongodb.ssl.keystore is configured.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| mongodb.ssl.truststore                                     | false    | No Default                                           | The location of the trust store file for the server certificate verification.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| mongodb.ssl.truststore.password                            | false    | No Default                                           | The password for the trust store file. Used to check the integrity of the truststore, and unlock the truststore. Specify a password only if the mongodb.ssl.truststore is configured.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| mongodb.ssl.truststore.type                                | false    | No Default                                           | The type of trust store file. Specify a type only if the mongodb.ssl.truststore is configured.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| source.struct.version                                      | false    | v2                                                   | Schema version for the source block in CDC events. Debezium 0.10 introduced a few breaking<br />changes to the structure of the source block in order to unify the exposed structure across all the connectors.<br />By setting this option to v1 the structure used in earlier versions can be produced. Note that this setting is not recommended and is planned for removal in a future Debezium version.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| heartbeat.interval.ms                                      | false    | 0                                                    | Controls how frequently heartbeat messages are sent.<br />This property contains an interval in milliseconds that defines how frequently the connector sends messages into a heartbeat topic. This can be used to monitor whether the connector is still receiving change events from the database. You also should leverage heartbeat messages in cases where only records in non-captured collections are changed for a longer period of time. In such situation the connector would proceed to read the oplog/change stream from the database but never emit any change messages into Kafka, which in turn means that no offset updates are committed to Kafka. This will cause the oplog files to be rotated out but connector will not notice it so on restart some events are no longer available which leads to the need of re-execution of the initial snapshot.<br /><br />Set this parameter to 0 to not send heartbeat messages at all.<br />Disabled by default.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| skipped.operations                                         | false    | t                                                    | A comma-separated list of the operation types that you want the connector to skip during streaming. You can configure the connector to skip the following types of operations:<br /><br />c (insert/create)<br /><br />u (update)<br /><br />d (delete)<br /><br />t (truncate)<br /><br />Set the value to none if you do not want the connector to skip any operations. Because MongoDB does not support truncate change events, setting the default t value has the same effect as setting the value to none.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| snapshot.collection.filter.overrides                       | false    | No default                                           | Controls which collection items are included in snapshot. This property affects snapshots only. Specify a comma-separated list of collection names in the form databaseName.collectionName.<br /><br />For each collection that you specify, also specify another configuration property: snapshot.collection.filter.overrides.databaseName.collectionName. For example, the name of the other configuration property might be: snapshot.collection.filter.overrides.customers.orders. Set this property to a valid filter expression that retrieves only the items that you want in the snapshot. When the connector performs a snapshot, it retrieves only the items that matches the filter expression.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| snapshot.delay.ms                                          | false    | No default                                           | An interval in milliseconds that the connector should wait before taking a snapshot after starting up;<br />Can be used to avoid snapshot interruptions when starting multiple connectors in a cluster, which may cause re-balancing of connectors.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| streaming.delay.ms                                         | false    | 0                                                    | Specifies the time, in milliseconds, that the connector delays the start of the streaming process after it completes a snapshot. Setting a delay interval helps to prevent the connector from restarting snapshots in the event that a failure occurs immediately after the snapshot completes, but before the streaming process begins. Set a delay value that is higher than the value of the offset.flush.interval.ms property that is set for the Kafka Connect worker.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| snapshot.fetch.size                                        | false    | 0                                                    | Specifies the maximum number of documents that should be read in one go from each collection while taking a snapshot. The connector will read the collection contents in multiple batches of this size.<br />Defaults to 0, which indicates that the server chooses an appropriate fetch size.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| snapshot.include.collection.list                           | false    | All collections specified in collection.include.list | An optional, comma-separated list of regular expressions that match the fully-qualified names (.) of the schemas that you want to include in a snapshot. The specified items must be named in the connectors’s collection.include.list property. This property takes effect only if the connector’s snapshot.mode property is set to a value other than never.<br />This property does not affect the behavior of incremental snapshots.<br /><br /><br />To match the name of a schema, Debezium applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire name string of the schema; it does not match substrings that might be present in a schema name.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| snapshot.max.threads                                       | false    | 1                                                    | Positive integer value that specifies the maximum number of threads used to perform an intial sync of the collections in a replica set. Defaults to 1.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| snapshot.mode                                              | false    | initial                                              | Specifies the criteria for performing a snapshot when the connector starts. Set the property to one of the following values:<br /><br />**always**: The connector performs a snapshot every time that it starts. The snapshot includes the structure and data of the captured tables. Specify this value to populate topics with a complete representation of the data from the captured tables every time that the connector starts. After the snapshot completes, the connector begins to stream event records for subsequent database changes.<br /><br />**initial**: When the connector starts, it performs an initial database snapshot. After the snapshot completes, the connector begins to stream event records for subsequent database changes.<br /><br />**initial\_only**: The connector performs a database a snapshot only when no offsets have been recorded for the logical server name. After the snapshot completes, the connector stops. It does not transition to streaming event records for subsequent database changes.<br /><br />**never**: Deprecated, see no\_data.<br /><br />**no\_data**: The connector runs a snapshot that captures the structure of all relevant tables, but it does not create READ events to represent the data set at the point of the connector’s start-up.<br /><br />**when\_needed**: After the connector starts, it performs a snapshot only if it detects one of the following circumstances:<br /><br />It cannot detect any topic offsets.<br /><br />A previously recorded offset specifies a log position that is not available on the server.<br /><br />**configuration\_based**: With this option, you control snapshot behavior through a set of connector properties that have the prefix 'snapshot.mode.configuration.based'.<br /><br />**custom**: The custom snapshot mode lets you inject your own implementation of the io.debezium.spi.snapshot.Snapshotter interface. Set the snapshot.mode.custom.name configuration property to the name provided by the name() method of your implementation.<br /><br />For more information, see custom snapshotter SPI. |
| snapshot.mode.configuration.based.snapshot.data            | false    | false                                                | If the snapshot.mode is set to configuration\_based, set this property to specify whether the connector includes table data when it performs a snapshot.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| snapshot.mode.configuration.based.snapshot.schema          | false    | false                                                | If the snapshot.mode is set to configuration\_based, set this property to specify whether the connector includes the table schema when it performs a snapshot.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| snapshot.mode.configuration.based.start.stream             | false    | false                                                | If the snapshot.mode is set to configuration\_based, set this property to specify whether the connector begins to stream change events after a snapshot completes.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| snapshot.mode.configuration.based.snapshot.on.schema.error | false    | false                                                | If the snapshot.mode is set to configuration\_based, set this property to specify whether the connector includes table schema in a snapshot if the schema history topic is not available.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| snapshot.mode.configuration.based.snapshot.on.data.error   | false    | false                                                | If the snapshot.mode is set to configuration\_based, this property specifies whether the connector attempts to snapshot table data if it does not find the last committed offset in the transaction log.<br />Set the value to true to instruct the connector to perform a new snapshot.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| snapshot.mode.custom.name                                  | false    | No default                                           | If snapshot.mode is set to custom, use this setting to specify the name of the custom implementation that is provided in the name() method that is defined in the 'io.debezium.spi.snapshot.Snapshotter' interface. After a connector restart, Debezium calls the specified custom implementation to determine whether to perform a snapshot. For more information, see custom snapshotter SPI.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| provide.transaction.metadata                               | false    | false                                                | When set to true Debezium generates events with transaction boundaries and enriches data events envelope with transaction metadata.<br /><br />See Transaction Metadata for additional details.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| retriable.restart.connector.wait.ms                        | false    | 10000 (10 seconds)                                   | The number of milliseconds to wait before restarting a connector after a retriable error occurs.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| mongodb.poll.interval.ms                                   | false    | 30000                                                | The interval in which the connector polls for new, removed, or changed replica sets.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| mongodb.connect.timeout.ms                                 | false    | 10000 (10 seconds)                                   | The number of milliseconds the driver will wait before a new connection attempt is aborted.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| mongodb.heartbeat.frequency.ms                             | false    | 10000 (10 seconds)                                   | The frequency that the cluster monitor attempts to reach each server.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| mongodb.socket.timeout.ms                                  | false    | 0                                                    | The number of milliseconds before a send/receive on the socket can take before a timeout occurs. A value of 0 disables this behavior.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| mongodb.server.selection.timeout.ms                        | false    | 30000 (30 seconds)                                   | The number of milliseconds the driver will wait to select a server before it times out and throws an error.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| cursor.pipeline                                            | false    | No default                                           | When streaming changes, this setting applies processing to change stream events as part of the standard MongoDB aggregation stream pipeline. A pipeline is a MongoDB aggregation pipeline composed of instructions to the database to filter or transform data. This can be used customize the data that the connector consumes. The value of this property must be an array of permitted aggregation pipeline stages in JSON format. Note that this is appended after the internal pipeline used to support the connector (e.g. filtering operation types, database names, collection names, etc.).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| cursor.pipeline.order                                      | false    | internal\_first                                      | The order used to construct the effective MongoDB aggregation stream pipeline. Set the property to one of the following values:<br /><br />**internal\_first**: Internal stages defined by the connector are applied first. This means that only the events which ought to be captured by the connector are fed to the user defined stages (configured by setting cursor.pipeline).<br /><br />**user\_first**: Stages defined by the 'cursor.pipeline' property are applied first. In this mode all events, included those not captured by the connector, are fed to user defined pipeline stages. This mode can have negative performance impact if the value of cursor.pipeline contains complex operations.<br /><br />**user\_only**: Stages defined by the 'cursor.pipeline' property will replace internal stages defined by the connector. This mode is intended only for expert users since all events are processed only by user defined pipeline stages. This mode can have negative impact on performance and overall functionality of the connector!                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| cursor.oversize.handling.mode                              | false    | fail                                                 | The strategy used to handle change events for documents exceeding specified BSON size. Set the property to one of the following values:<br /><br />**fail**: The connector fails if the total size of change event exceed the maximum BSON size.<br /><br />**skip**: Any change events for documents exceeding the maximum (specified by the cursor.oversize.skip.threshold property) size will be ignored<br /><br />**split**: Change events exceeding the maximum BSON size will be split using the \$changeStreamSplitLargeEvent aggregation. This option requires MongoDB 6.0.9 or newer.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| cursor.oversize.skip.threshold                             | false    | 0                                                    | The maximum allowed size in bytes of the stored document for which change events are processed. This includes both, the size before and after database operation, more specifically this limits the size of fullDocument and fullDocumentBeforeChange filed of MongoDB change events.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| cursor.max.await.time.ms                                   | false    | 0                                                    | Specifies the maximum number of milliseconds the oplog/change stream cursor will wait for the server to produce a result before causing an execution timeout exception. A value of 0 indicates using the server/driver default wait timeout.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| signal.data.collection                                     | false    | No default                                           | Fully-qualified name of the data collection that is used to send signals to the connector. Use the following format to specify the collection name:<br />.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| signal.enabled.channels                                    | false    | source                                               | List of the signaling channel names that are enabled for the connector. By default, the following channels are available:<br /><br />source<br /><br />kafka<br /><br />file<br /><br />jmx Optionally, you can also implement a custom signaling channel.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| notification.enabled.channels                              | false    | No default                                           | List of notification channel names that are enabled for the connector. By default, the following channels are available:<br /><br />sink<br /><br />log<br /><br />jmx Optionally, you can also implement a custom notification channel.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| incremental.snapshot.chunk.size                            | false    | 1024                                                 | The maximum number of documents that the connector fetches and reads into memory during an incremental snapshot chunk. Increasing the chunk size provides greater efficiency, because the snapshot runs fewer snapshot queries of a greater size. However, larger chunk sizes also require more memory to buffer the snapshot data. Adjust the chunk size to a value that provides the best performance in your environment.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| incremental.snapshot.watermarking.strategy                 | false    | insert\_insert                                       | Specifies the watermarking mechanism that the connector uses during an incremental snapshot to deduplicate events that might be captured by an incremental snapshot and then recaptured after streaming resumes.<br />You can specify one of the following options:<br /><br />**insert\_insert**: When you send a signal to initiate an incremental snapshot, for every chunk that Debezium reads during the snapshot, it writes an entry to the signaling data collection to record the signal to open the snapshot window. After the snapshot completes, Debezium inserts a second entry that records the signal to close the window.<br /><br />**insert\_delete**: When you send a signal to initiate an incremental snapshot, for every chunk that Debezium reads, it writes a single entry to the signaling data collection to record the signal to open the snapshot window. After the snapshot completes, this entry is removed. No entry is created for the signal to close the snapshot window. Set this option to prevent rapid growth of the signaling data collection.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| topic.naming.strategy                                      | false    | io.debezium.schema.DefaultTopicNamingStrategy        | The name of the TopicNamingStrategy class that should be used to determine the topic name for data change, schema change, transaction, heartbeat event etc., defaults to DefaultTopicNamingStrategy.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| topic.delimiter                                            | false    | .                                                    | Specify the delimiter for topic name, defaults to ..                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| topic.cache.size                                           | false    | 10000                                                | The size used for holding the topic names in bounded concurrent hash map. This cache will help to determine the topic name corresponding to a given data collection.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| topic.heartbeat.prefix                                     | false    | \_\_debezium-heartbeat                               | Controls the name of the topic to which the connector sends heartbeat messages. The topic name has this pattern:<br /><br />topic.heartbeat.prefix.topic.prefix<br /><br />For example, if the topic prefix is fulfillment, the default topic name is \_\_debezium-heartbeat.fulfillment.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| topic.transaction                                          | false    | transaction                                          | Controls the name of the topic to which the connector sends transaction metadata messages. The topic name has this pattern:<br /><br />topic.prefix.topic.transaction<br /><br />For example, if the topic prefix is fulfillment, the default topic name is fulfillment.transaction.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| custom.metric.tags                                         | false    | No default                                           | Defines tags that customize MBean object names by adding metadata that provides contextual information. Specify a comma-separated list of key-value pairs. Each key represents a tag for the MBean object name, and the corresponding value represents a value for the key, for example,<br />k1=v1,k2=v2<br /><br />The connector appends the specified tags to the base MBean object name. Tags can help you to organize and categorize metrics data. You can define tags to identify particular application instances, environments, regions, versions, and so forth. For more information, see Customized MBean names.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| errors.max.retries                                         | false    | -1                                                   | Specifies how the connector responds after an operation that results in a retriable error, such as a connection error.<br />Set one of the following options:<br /><br />-1<br /><br />No limit. The connector always restarts automatically, and retries the operation, regardless of the number of previous failures.<br /><br />0<br /><br />Disabled. The connector fails immediately, and never retries the operation. User intervention is required to restart the connector.<br /><br />> 0<br /><br />The connector restarts automatically until it reaches the specified maximum number of retries. After the next failure, the connector stops, and user intervention is required to restart it.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |

For more information about the configuration properties, see the [Official Debezium MongoDB Connector documentation](https://debezium.io/documentation/reference/stable/connectors/mongodb.html#connector-properties).


# Kafka connect debezium mysql
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-debezium-mysql/current/kafka-connect-debezium-mysql

Kafka Connect Debezium MySQL Source connector

The Debezium MySQL Source connector is a Kafka Connect connector that captures row-level changes in a MySQL database and streams them to Kafka topics.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

### Prerequisites

* A running MySQL server
* Binlog enabled in MySQL server

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)

2. Create a JSON file like the following:

   ```json theme={null}
   {
       "name": "debezium-mysql-source",
       "config": {
           "connector.class": "io.debezium.connector.mysql.MySqlConnector",
           "tasks.max": "1",
           "database.hostname": "{host}",
           "database.port": "3306",
           "database.user": "{username}",
           "database.password": "{password}",
           "database.server.id": "1",
           "topic.prefix": "fullfillment",
           "database.include.list": "inventory",
           "database.history.kafka.topic": "dbhistory.inventory",
           "schema.history.internal.kafka.bootstrap.servers": "kafka:9092",
           "schema.history.internal.kafka.topic": "schemahistory.fullfillment",
           "sn.passthrough.bootstrapServer.fields": "schema.history.internal.kafka.bootstrap.servers",
           "sn.passthrough.kafka.consumer.field.prefixes": "schema.history.internal",
           "sn.passthrough.kafka.producer.field.prefixes": "schema.history.internal"
       }
   }
   ```

   <Note>
     `sn.passthrough.bootstrapServer.fields` and `sn.passthrough.kafka.consumer|producer.field.prefixes` are required to make StreamNative Cloud automatically configure the schema history Kafka client settings for the connector.<br />
     You shouldn't change these properties unless you want to override the default behavior.
   </Note>

3. Run the following command to create the connector:

```bash theme={null}
kcctl apply -f <filename>.json
```

### Configuration

The Debezium MySQL Source connector is configured using the following properties:

| Property                                         | Required | Default                      | Description                                                                                                                                                                                                                                                                                                                                                               |
| ------------------------------------------------ | -------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| name                                             | true     | No default                   | Unique name for the connector. Attempting to register again with the same name will fail. (This property is required by all Kafka Connect connectors.)                                                                                                                                                                                                                    |
| connector.class                                  | true     | No default                   | The name of the Java class for the connector. Always use a value of io.debezium.connector.mysql.MySqlConnector for the MySQL connector.                                                                                                                                                                                                                                   |
| database.hostname                                | true     | No default                   | The address of the MySQL database server.                                                                                                                                                                                                                                                                                                                                 |
| database.port                                    | true     | 3306                         | The port number of the MySQL database server.                                                                                                                                                                                                                                                                                                                             |
| database.user                                    | true     | No default                   | The name of the MySQL database user to be used when connecting to the database.                                                                                                                                                                                                                                                                                           |
| database.password                                | true     | No default                   | The password to be used when connecting to the database.                                                                                                                                                                                                                                                                                                                  |
| database.server.id                               | true     | No default                   | A numeric ID for the database server. This ID must be unique among all database servers in the same cluster.                                                                                                                                                                                                                                                              |
| topic.prefix                                     | true     | No default                   | A logical name for the database server. This name is used as a prefix for all Kafka topics that receive records from this connector.                                                                                                                                                                                                                                      |
| schema.history.internal.kafka.topic              | true     | No default                   | The full name of the Kafka topic where the connector stores the database schema history.                                                                                                                                                                                                                                                                                  |
| schema.history.internal.kafka.bootstrap.servers  | true     | No default                   | A list of host/port pairs that the connector uses for establishing an initial connection to the Kafka cluster                                                                                                                                                                                                                                                             |
| field.name.adjustment.mode                       | false    | No default                   | Specifies how field names should be adjusted for compatibility with the message converter used by the connector.                                                                                                                                                                                                                                                          |
| schema.name.adjustment.mode                      | false    | No default                   | Specifies how the connector adjusts schema names for compatibility with the message converter used by the connector.                                                                                                                                                                                                                                                      |
| bigint.unsigned.handling.mode                    | false    | long                         | Specifies how the connector represents BIGINT UNSIGNED columns in change events.                                                                                                                                                                                                                                                                                          |
| binary.handling.mode                             | false    | bytes                        | Specifies how the connector represents values for binary columns, such as, blob, binary, varbinary, in change events.                                                                                                                                                                                                                                                     |
| column.exclude.list                              | false    | empty string                 | An optional, comma-separated list of regular expressions that match the fully-qualified names of columns to exclude from change event record values. Other columns in the source record are captured as usual. Fully-qualified names for columns are of the form databaseName.tableName.columnName.                                                                       |
| column.include.list                              | false    | empty string                 | An optional, comma-separated list of regular expressions that match the fully-qualified names of columns to include in change event record values. Other columns are omitted from the event record. Fully-qualified names for columns are of the form databaseName.tableName.columnName.                                                                                  |
| column.mask.hash.v2.hashAlgorithm.with.salt.salt | false    | No default                   | An optional, comma-separated list of regular expressions that match the fully-qualified names of character-based columns. Fully-qualified names for columns are of the form \[databaseName].\[tableName].\[columnName].                                                                                                                                                   |
| column.mask.with.length.chars                    | false    | No default                   | An optional, comma-separated list of regular expressions that match the fully-qualified names of character-based columns.                                                                                                                                                                                                                                                 |
| column.propagate.source.type                     | false    | No default                   | An optional, comma-separated list of regular expressions that match the fully-qualified names of columns for which you want the connector to emit extra parameters that represent column metadata.                                                                                                                                                                        |
| column.truncate.to.length.chars                  | false    | No default                   | An optional, comma-separated list of regular expressions that match the fully-qualified names of character-based columns. Set this property if you want to truncate the data in a set of columns when it exceeds the number of characters specified by the length in the property name. Set length to a positive integer value, for example, column.truncate.to.20.chars. |
| connect.timeout.ms                               | false    | 30                           | A positive integer value that specifies the maximum time in milliseconds that the connector waits to establish a connection to the \[connector-name] database server before the connection request times out.                                                                                                                                                             |
| database.exclude.list                            | false    | empty string                 | An optional comma-separated list of regular expressions that match database names to be excluded from monitoring.                                                                                                                                                                                                                                                         |
| database.include.list                            | false    | empty string                 | An optional comma-separated list of regular expressions that match database names to be monitored.                                                                                                                                                                                                                                                                        |
| decimal.handling.mode                            | false    | precise                      | Specifies how the connector handles values for DECIMAL and NUMERIC columns in change events.                                                                                                                                                                                                                                                                              |
| gtid.source.excludes                             | false    | No default                   | A comma-separated list of regular expressions that match source domain IDs in the GTID set that the connector uses to find the binlog position on the \[connector-name] server.                                                                                                                                                                                           |
| gtid.source.includes                             | false    | No default                   | A comma-separated list of regular expressions that match source domain IDs in the GTID set used that the connector uses to find the binlog position on the \[connector-name] server.                                                                                                                                                                                      |
| include.query                                    | false    | false                        | Boolean value that specifies whether the change event that the connector emits includes the SQL query that generated the change.                                                                                                                                                                                                                                          |
| include.schema.changes                           | false    | true                         | Boolean value that specifies whether the connector publishes changes in the database schema to a Kafka topic with the same name as the topic prefix.                                                                                                                                                                                                                      |
| include.schema.comments                          | false    | false                        | Boolean value that specifies whether the connector parses and publishes table and column comments on metadata objects.                                                                                                                                                                                                                                                    |
| inconsistent.schema.handling.mode                | false    | fail                         | Specifies how the connector responds to binlog events that refer to tables that are not present in the internal schema representation. That is, the internal representation is not consistent with the database.                                                                                                                                                          |
| message.key.columns                              | false    | No default                   | A list of expressions that specify the columns that the connector uses to form custom message keys for change event records that it publishes to the Kafka topics for specified tables.                                                                                                                                                                                   |
| skip.messages.without.change                     | false    | false                        | Specifies whether the connector emits messages for records when it does not detect a change in the included columns.                                                                                                                                                                                                                                                      |
| table.exclude.list                               | false    | empty string                 | An optional comma-separated list of regular expressions that match fully-qualified table names to be excluded from monitoring.                                                                                                                                                                                                                                            |
| table.include.list                               | false    | empty string                 | An optional comma-separated list of regular expressions that match fully-qualified table names to be monitored.                                                                                                                                                                                                                                                           |
| time.precision.mode                              | false    | adaptive\_time\_microseconds | Specifies the type of precision that the connector uses to represent time, date, and timestamps values.                                                                                                                                                                                                                                                                   |
| tombstones.on.delete                             | false    | true                         | Specifies whether a delete event is followed by a tombstone event. After a source record is deleted, the connector can emit a tombstone event (the default behavior) to enable Kafka to completely delete all events that pertain to the key of the deleted row in case \[link-kafka-docs]/#compaction\[log compaction] is enabled for the topic.                         |

For more information about the configuration properties, see the [Official Debezium MySQL Connector documentation](https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-connector-properties).


# Kafka connect debezium postgresql
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-debezium-postgresql/current/kafka-connect-debezium-postgresql

Kafka Connect Debezium PostgreSQL Source connector

The Debezium PostgreSQL Source connector is a Kafka Connect connector that captures row-level changes in a PostgreSQL database and streams them to Kafka topics.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

### Prerequisites

* A running PostgreSQL server
* PostgreSQL version 10 or later

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)
2. Create a JSON file like the following:

   ```json theme={null}
   {
       "name": "debezium-postgres-source",
       "config": {
           "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
           "tasks.max": "1",
           "database.hostname": "{host}",
           "database.port": "5432",
           "database.user": "{username}",
           "database.password": "{password}",
           "database.dbname" : "postgres",
           "topic.prefix": "fullfillment",
           "schema.include.list": "public",
           "plugin.name": "pgoutput"
       }
   }
   ```
3. Run the following command to create the connector:

   ```bash theme={null}
   kcctl apply -f &lt;filename&gt;.json
   ```

### Configuration

The Debezium PostgreSQL Source connector is configured using the following properties:

| Property                                         | Required | Default          | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| ------------------------------------------------ | -------- | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| name                                             | true     | No default       | Unique name for the connector. Attempting to register again with the same name will fail. This property is required by all Kafka Connect connectors.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| connector.class                                  | true     | No default       | The name of the Java class for the connector. Always use a value of io.debezium.connector.postgresql.PostgresConnector for the PostgreSQL connector.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| plugin.name                                      | false    | decoderbufs      | The name of the PostgreSQL logical decoding plug-in installed on the PostgreSQL server.<br /><br />Supported values are decoderbufs, and pgoutput.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| slot.name                                        | false    | debezium         | The name of the PostgreSQL logical decoding slot that was created for streaming changes from a particular plug-in for a particular database/schema. The server uses this slot to stream events to the Debezium connector that you are configuring.<br /><br />Slot names must conform to PostgreSQL replication slot naming rules, which state: "Each replication slot has a name, which can contain lower-case letters, numbers, and the underscore character."                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| slot.drop.on.stop                                | false    | false            | Whether or not to delete the logical replication slot when the connector stops in a graceful, expected way. The default behavior is that the replication slot remains configured for the connector when the connector stops. When the connector restarts, having the same replication slot enables the connector to start processing where it left off.<br /><br />Set to true in only testing or development environments. Dropping the slot allows the database to discard WAL segments. When the connector restarts it performs a new snapshot or it can continue from a persistent offset in the Kafka Connect offsets topic.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| slot.failover                                    | false    | false            | Specifies whether the connector creates a failover slot. If you omit this setting, or if the primary server runs PostgreSQL 16 or earlier, the connector does not create a failover slot.<br /><br />PostgreSQL uses the synchronized\_standby\_slots parameter to configure replication slot synchronization between primary and standby servers. Set this parameter on the primary server to specify the physical replication slots that it synchronizes with on standby servers.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| publication.name                                 | false    | dbz\_publication | The name of the PostgreSQL publication created for streaming changes when using pgoutput.<br /><br />This publication is created at start-up if it does not already exist and it includes all tables. Debezium then applies its own include/exclude list filtering, if configured, to limit the publication to change events for the specific tables of interest. The connector user must have superuser permissions to create this publication, so it is usually preferable to create the publication before starting the connector for the first time.<br /><br />If the publication already exists, either for all tables or configured with a subset of tables, Debezium uses the publication as it is defined.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| database.hostname                                | true     | No default       | IP address or hostname of the PostgreSQL database server.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| database.port                                    | true     | 5432             | Integer port number of the PostgreSQL database server.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| database.user                                    | false    | No default       | Name of the PostgreSQL database user for connecting to the PostgreSQL database server.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| database.password                                | false    | No default       | Password to use when connecting to the PostgreSQL database server.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| database.dbname                                  | true     | No default       | The name of the PostgreSQL database from which to stream the changes.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| topic.prefix                                     | true     | No default       | Topic prefix that provides a namespace for the particular PostgreSQL database server or cluster in which Debezium is capturing changes. The prefix should be unique across all other connectors, since it is used as a topic name prefix for all Kafka topics that receive records from this connector. Only alphanumeric characters, hyphens, dots and underscores must be used in the database server logical name.<br /><br /><br /><br />Do not change the value of this property. If you change the name value, after a restart, instead of continuing to emit events to the original topics, the connector emits subsequent events to topics whose names are based on the new value.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| schema.include.list                              | false    | No default       | An optional, comma-separated list of regular expressions that match names of schemas for which you want to capture changes. Any schema name not included in schema.include.list is excluded from having its changes captured. By default, all non-system schemas have their changes captured.<br /><br /><br />To match the name of a schema, Debezium applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire identifier for the schema; it does not match substrings that might be present in a schema name.<br />If you include this property in the configuration, do not also set the schema.exclude.list property.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| schema.exclude.list                              | false    | No default       | An optional, comma-separated list of regular expressions that match names of schemas for which you do not want to capture changes. Any schema whose name is not included in schema.exclude.list has its changes captured, with the exception of system schemas.<br /><br /><br />To match the name of a schema, Debezium applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire identifier for the schema; it does not match substrings that might be present in a schema name.<br />If you include this property in the configuration, do not set the schema.include.list property.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| table.include.list                               | false    | No default       | An optional, comma-separated list of regular expressions that match fully-qualified table identifiers for tables whose changes you want to capture. When this property is set, the connector captures changes only from the specified tables. Each identifier is of the form schemaName.tableName. By default, the connector captures changes in every non-system table in each schema whose changes are being captured.<br /><br /><br />To match the name of a table, Debezium applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire identifier for the table; it does not match substrings that might be present in a table name.<br />If you include this property in the configuration, do not also set the table.exclude.list property.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| table.exclude.list                               | false    | No default       | An optional, comma-separated list of regular expressions that match fully-qualified table identifiers for tables whose changes you do not want to capture. Each identifier is of the form schemaName.tableName. When this property is set, the connector captures changes from every table that you do not specify.<br /><br /><br />To match the name of a table, Debezium applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire identifier for the table; it does not match substrings that might be present in a table name.<br />If you include this property in the configuration, do not set the table.include.list property.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| column.include.list                              | false    | No default       | An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be included in change event record values. Fully-qualified names for columns are of the form schemaName.tableName.columnName.<br /><br /><br />To match the name of a column, Debezium applies the regular expression that you specify as an anchored regular expression. That is, the expression is used to match the entire name string of the column; it does not match substrings that might be present in a column name.<br />If you include this property in the configuration, do not also set the column.exclude.list property.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| column.exclude.list                              | false    | No default       | An optional, comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event record values. Fully-qualified names for columns are of the form schemaName.tableName.columnName.<br /><br /><br />To match the name of a column, Debezium applies the regular expression that you specify as an anchored regular expression. That is, the expression is used to match the entire name string of the column; it does not match substrings that might be present in a column name.<br />If you include this property in the configuration, do not set the column.include.list property.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| skip.messages.without.change                     | false    | false            | Specifies whether to skip publishing messages when there is no change in included columns. This would essentially filter messages if there is no change in columns included as per column.include.list or column.exclude.list properties.<br /><br />This property is applied only when the REPLICA IDENTITY of the table is set to FULL.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| time.precision.mode                              | false    | adaptive         | Time, date, and timestamps can be represented with different kinds of precision:<br /><br />adaptive captures the time and timestamp values exactly as in the database using either millisecond, microsecond, or nanosecond precision values based on the database column’s type.<br /><br />adaptive\_time\_microseconds captures the date, datetime and timestamp values exactly as in the database using either millisecond, microsecond, or nanosecond precision values based on the database column’s type. An exception is TIME type fields, which are always captured as microseconds.<br /><br />connect always represents time and timestamp values by using Kafka Connect’s built-in representations for Time, Date, and Timestamp, which use millisecond precision regardless of the database columns' precision. For more information, see temporal values.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| decimal.handling.mode                            | false    | precise          | Specifies how the connector should handle values for DECIMAL and NUMERIC columns:<br /><br />precise represents values by using java.math.BigDecimal to represent values in binary form in change events.<br /><br />double represents values by using double values, which might result in a loss of precision but which is easier to use.<br /><br />string encodes values as formatted strings, which are easy to consume but semantic information about the real type is lost. For more information, see Decimal types.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| hstore.handling.mode                             | false    | json             | Specifies how the connector should handle values for hstore columns:<br /><br />map represents values by using MAP.<br /><br />json represents values by using json string. This setting encodes values as formatted strings such as `{"key":"val"}`. For more information, see PostgreSQL HSTORE type.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| interval.handling.mode                           | false    | numeric          | Specifies how the connector should handle values for interval columns:<br /><br />numeric represents intervals using approximate number of microseconds.<br /><br />string represents intervals exactly by using the string pattern representation `P<years>Y<months>M<days>DT<hours>H<minutes>M<seconds>S`. For example: P1Y2M3DT4H5M6.78S. For more information, see PostgreSQL basic types.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| database.sslmode                                 | false    | prefer           | Whether to use an encrypted connection to the PostgreSQL server. Options include:<br /><br />disable uses an unencrypted connection.<br /><br />allow attempts to use an unencrypted connection first and, failing that, a secure (encrypted) connection.<br /><br />prefer attempts to use a secure (encrypted) connection first and, failing that, an unencrypted connection.<br /><br />require uses a secure (encrypted) connection, and fails if one cannot be established.<br /><br />verify-ca behaves like require but also verifies the server TLS certificate against the configured Certificate Authority (CA) certificates, or fails if no valid matching CA certificates are found.<br /><br />verify-full behaves like verify-ca but also verifies that the server certificate matches the host to which the connector is trying to connect. For more information, see the PostgreSQL documentation.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| database.sslcert                                 | false    | No default       | The path to the file that contains the SSL certificate for the client. For more information, see the PostgreSQL documentation.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| database.sslkey                                  | false    | No default       | The path to the file that contains the SSL private key of the client. For more information, see the PostgreSQL documentation.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| database.sslpassword                             | false    | No default       | The password to access the client private key from the file specified by database.sslkey. For more information, see the PostgreSQL documentation.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| database.sslrootcert                             | false    | No default       | The path to the file that contains the root certificate(s) against which the server is validated. For more information, see the PostgreSQL documentation.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| database.sslfactory                              | false    | No default       | A name of the class that creates SSL Sockets. Use org.postgresql.ssl.NonValidatingFactory to disable SSL validation in development environments.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| database.tcpKeepAlive                            | false    | true             | Enable TCP keep-alive probe to verify that the database connection is still alive. For more information, see the PostgreSQL documentation.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| tombstones.on.delete                             | false    | true             | Controls whether a delete event is followed by a tombstone event.<br /><br />true - a delete operation is represented by a delete event and a subsequent tombstone event.<br /><br />false - only a delete event is emitted.<br /><br />After a source record is deleted, emitting a tombstone event (the default behavior) allows Kafka to completely delete all events that pertain to the key of the deleted row in case log compaction is enabled for the topic.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| column.truncate.to.length.chars                  | false    | n/a              | An optional, comma-separated list of regular expressions that match the fully-qualified names of character-based columns. Set this property if you want to truncate the data in a set of columns when it exceeds the number of characters specified by the length in the property name. Set length to a positive integer value, for example, column.truncate.to.20.chars.<br /><br />The fully-qualified name of a column observes the following format: \<schemaName>.\<tableName>.\<columnName>. To match the name of a column, Debezium applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire name string of the column; the expression does not match substrings that might be present in a column name.<br /><br />You can specify multiple properties with different lengths in a single configuration.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| column.mask.with.length.chars                    | false    | n/a              | An optional, comma-separated list of regular expressions that match the fully-qualified names of character-based columns. Set this property if you want the connector to mask the values for a set of columns, for example, if they contain sensitive data. Set length to a positive integer to replace data in the specified columns with the number of asterisk (\*) characters specified by the length in the property name. Set length to 0 (zero) to replace data in the specified columns with an empty string.<br /><br />The fully-qualified name of a column observes the following format: schemaName.tableName.columnName. To match the name of a column, Debezium applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire name string of the column; the expression does not match substrings that might be present in a column name.<br /><br />You can specify multiple properties with different lengths in a single configuration.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| column.mask.hash.hashAlgorithm.with.salt.salt    | false    | n/a              | An optional, comma-separated list of regular expressions that match the fully-qualified names of character-based columns. Fully-qualified names for columns are of the form \<schemaName>.\<tableName>.\<columnName>.<br />To match the name of a column Debezium applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire name string of the column; the expression does not match substrings that might be present in a column name. In the resulting change event record, the values for the specified columns are replaced with pseudonyms.<br /><br /><br />A pseudonym consists of the hashed value that results from applying the specified hashAlgorithm and salt. Based on the hash function that is used, referential integrity is maintained, while column values are replaced with pseudonyms. Supported hash functions are described in the MessageDigest section of the Java Cryptography Architecture Standard Algorithm Name Documentation.<br /><br />In the following example, CzQMA0cB5K is a randomly selected salt.<br /><br /><br />column.mask.hash.SHA-256.with.salt.CzQMA0cB5K = inventory.orders.customerName, inventory.shipment.customerName<br /><br />If necessary, the pseudonym is automatically shortened to the length of the column. The connector configuration can include multiple properties that specify different hash algorithms and salts.<br /><br />Depending on the hashAlgorithm used, the salt selected, and the actual data set, the resulting data set might not be completely masked.<br /><br />Hashing strategy version 2 should be used to ensure fidelity if the value is being hashed in different places or systems.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| column.mask.hash.v2.hashAlgorithm.with.salt.salt | false    | n/a              | An optional, comma-separated list of regular expressions that match the fully-qualified names of character-based columns. Fully-qualified names for columns are of the form \<schemaName>.\<tableName>.\<columnName>.<br />To match the name of a column Debezium applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire name string of the column; the expression does not match substrings that might be present in a column name. In the resulting change event record, the values for the specified columns are replaced with pseudonyms.<br /><br /><br />A pseudonym consists of the hashed value that results from applying the specified hashAlgorithm and salt. Based on the hash function that is used, referential integrity is maintained, while column values are replaced with pseudonyms. Supported hash functions are described in the MessageDigest section of the Java Cryptography Architecture Standard Algorithm Name Documentation.<br /><br />In the following example, CzQMA0cB5K is a randomly selected salt.<br /><br /><br />column.mask.hash.SHA-256.with.salt.CzQMA0cB5K = inventory.orders.customerName, inventory.shipment.customerName<br /><br />If necessary, the pseudonym is automatically shortened to the length of the column. The connector configuration can include multiple properties that specify different hash algorithms and salts.<br /><br />Depending on the hashAlgorithm used, the salt selected, and the actual data set, the resulting data set might not be completely masked.<br /><br />Hashing strategy version 2 should be used to ensure fidelity if the value is being hashed in different places or systems.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| column.propagate.source.type                     | false    | n/a              | An optional, comma-separated list of regular expressions that match the fully-qualified names of columns for which you want the connector to emit extra parameters that represent column metadata. When this property is set, the connector adds the following fields to the schema of event records:<br /><br />\_\_debezium.source.column.type<br /><br /><br />\_\_debezium.source.column.length<br /><br /><br />\_\_debezium.source.column.scale<br /><br /><br />These parameters propagate a column’s original type name and length (for variable-width types), respectively.<br />Enabling the connector to emit this extra data can assist in properly sizing specific numeric or character-based columns in sink databases.<br /><br />The fully-qualified name of a column observes one of the following formats: databaseName.tableName.columnName, or databaseName.schemaName.tableName.columnName.<br />To match the name of a column, Debezium applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire name string of the column; the expression does not match substrings that might be present in a column name.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| datatype.propagate.source.type                   | false    | n/a              | An optional, comma-separated list of regular expressions that specify the fully-qualified names of data types that are defined for columns in a database. When this property is set, for columns with matching data types, the connector emits event records that include the following extra fields in their schema:<br /><br />\_\_debezium.source.column.type<br /><br /><br />\_\_debezium.source.column.length<br /><br /><br />\_\_debezium.source.column.scale<br /><br /><br />These parameters propagate a column’s original type name and length (for variable-width types), respectively.<br />Enabling the connector to emit this extra data can assist in properly sizing specific numeric or character-based columns in sink databases.<br /><br />The fully-qualified name of a column observes one of the following formats: databaseName.tableName.typeName, or databaseName.schemaName.tableName.typeName.<br />To match the name of a data type, Debezium applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire name string of the data type; the expression does not match substrings that might be present in a type name.<br /><br />For the list of PostgreSQL-specific data type names, see the PostgreSQL data type mappings.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| message.key.columns                              | false    | empty string     | A list of expressions that specify the columns that the connector uses to form custom message keys for change event records that it publishes to the Kafka topics for specified tables.<br /><br />By default, Debezium uses the primary key column of a table as the message key for records that it emits. In place of the default, or to specify a key for tables that lack a primary key, you can configure custom message keys based on one or more columns.<br /><br />To establish a custom message key for a table, list the table, followed by the columns to use as the message key. Each list entry takes the following format:<br /><br />\<fully-qualified\_tableName>:\<keyColumn>,\<keyColumn><br /><br />To base a table key on multiple column names, insert commas between the column names.<br /><br />Each fully-qualified table name is a regular expression in the following format:<br /><br />\<schemaName>.\<tableName><br /><br />The property can include entries for multiple tables. Use a semicolon to separate table entries in the list.<br /><br />The following example sets the message key for the tables inventory.customers and purchase.orders:<br /><br />inventory.customers:pk1,pk2;(.\*).purchaseorders:pk3,pk4<br /><br />In the example, the columns pk1 and pk2 are specified as the message key for the table inventory.customer. For the purchaseorders tables in any schema, the columns pk3 and pk4 serve as the message key.<br /><br />There is no limit to the number of columns that you use to create custom message keys. However, it’s best to use the minimum number that are required to specify a unique key.<br /><br />If the expressions that you specify for this property match columns that are not part of the table’s primary key, set the REPLICA IDENTITY of the table to FULL. If you set REPLICA IDENTITY to another value, such as DEFAULT, after delete operations, the connector fails to generate tombstone events with the expected null values.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| publication.autocreate.mode                      | false    | all\_tables      | Specifies whether and how the connector creates a publication. This setting applies only when the connector streams changes by using the pgoutput plug-in.<br /><br />To create publications, the connector must access PostgreSQL through a database account that has specific permissions. For more information, see Setting privileges to enable Debezium to create PostgreSQL publications.<br /><br />Specify one of the following values:<br /><br />all\_tables<br /><br />If a publication exists, the connector uses it.<br />If a publication does not exist, the connector creates a publication for all tables in the database from which the connector captures changes. The connector runs the following SQL command to create a publication:<br /><br />CREATE PUBLICATION \<publication\_name> FOR ALL TABLES;<br /><br />disabled<br /><br />The connector does not attempt to create a publication. A database administrator or the user configured to perform replications must have created the publication before running the connector. If the connector cannot find the publication, the connector throws an exception and stops.<br /><br />filtered<br /><br />If a publication does not exist, the connector creates one by running a SQL command in the following format:<br /><br />CREATE PUBLICATION \<publication\_name> FOR TABLE \<tbl1, tbl2, tbl3><br />The resulting publication includes tables that match the current filter configuration, as specified by the schema.include.list, schema.exclude.list, table.include.list, and table.exclude.list connector configuration properties.<br />If the publication exists, the connector updates the publication for tables that match the current filter configuration by running a SQL command in the following format:<br /><br />ALTER PUBLICATION \<publication\_name> SET TABLE \<tbl1, tbl2, tbl3>.<br /><br />no\_tables<br /><br />If a publication exists, the connector uses it. If a publication does not exist, the connector creates a publication without specifying any table by running a SQL command in the following format:<br /><br />CREATE PUBLICATION \<publication\_name>;<br /><br />Set the no\_tables option if you want the connector to capture only logical decoding messages, and not capture any other change events, such as those caused by INSERT, UPDATE, and DELETE operations on any table.<br /><br />If you select this option, to prevent the connector from emitting and processing READ events, you can specify names of schemas or tables for which you do not want to capture changes, for example, by using "table.exclude.list": "public.\*" or "schema.exclude.list": "public". |
| replica.identity.autoset.values                  | false    | empty string     | Set this property to apply specific replica identity settings to a subset of the tables that a connector captures, based on the table name. The replica identity values that the property sets overwrite the replica identity values that are set in the database.<br /><br />The property accepts a comma-separated list of key-value pairs. Each key is a regular expression that matches fully-qualified table names; the corresponding value specifies a replica identity type. For example:<br /><br />\<fqTableNameA>:\<replicaIdentity1>,\<fqTableNameB>:\<replicaIdentity2>,\<fqTableNameC>:\<replicaIdentity3><br /><br />Use the following format to specify the fully qualified table name:<br />SchemaName.TableName<br /><br />Set the replica identity to one of the following values:<br /><br />DEFAULT<br /><br />Records the value, if one existed, that was set for the primary key column before the change event. This is the default setting for non-system tables.<br /><br />INDEX indexName<br /><br />Records the values that were set for all columns defined for a specified index before the change event. The index must be unique, not partial, not deferrable, and must include only columns marked NOT NULL. If the specified index is dropped, the resulting behavior is the same as if you set the value to NOTHING.<br /><br />FULL<br /><br />Records the values that were set for all columns in the row before the change event.<br /><br />NOTHING<br /><br />Records no information about the row state before the change event. This is the default value for system tables.<br /><br />Example:<br />schema1.\*:FULL,schema2.table2:NOTHING,schema2.table3:INDEX idx\_name<br /><br />The replica.identity.autoset.values property applies only to tables that the connector captures. Other tables are ignored, even if they match the specified expression. Use the following connector properties to designate the tables to capture:<br /><br />table.include.list<br /><br />table.exclude.list<br /><br />schema.include.list<br /><br />schema.exclude.list                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| binary.handling.mode                             | false    | bytes            | Specifies how binary (bytea) columns should be represented in change events. Specify one of the following values:<br /><br />bytes<br /><br />Represents binary data as a byte array.<br /><br />base64<br /><br />Represents binary data as base64-encoded strings.<br /><br />base64-url-safe<br /><br />Represents binary data as base64-url-safe-encoded strings.<br /><br />hex<br /><br />Represents binary data as hex-encoded (base16) strings.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| schema.name.adjustment.mode                      | false    | none             | Specifies how schema names should be adjusted for compatibility with the message converter used by the connector. Set one of the following values:<br /><br />none<br /><br />Does not apply any adjustment.<br /><br />avro<br /><br />Replaces the characters that cannot be used in the Avro type name with underscore.<br /><br />avro\_unicode<br /><br />Replaces the underscore or characters that cannot be used in the Avro type name with corresponding Unicode characters, such as *uxxxx.<br /><br />In the preceding example, the underscore character (*) represents an escape sequence, equivalent to a backslash in Java.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| field.name.adjustment.mode                       | false    | none             | Specifies how field names should be adjusted for compatibility with the message converter used by the connector. Specify one of the following values:<br /><br />none<br /><br />Do not apply any adjustment.<br /><br />avro<br /><br />Replace characters that cannot be used in Avro type names with underscores.<br /><br />avro\_unicode<br /><br />Replace the underscore or characters that cannot be used in Avro type names with the corresponding Unicode characters, such as *uxxxx.<br /><br />In the preceding example, the underscore character (*) represents an escape sequence, equivalent to a backslash in Java.<br /><br />For more information, see Avro naming.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| money.fraction.digits                            | false    | 2                | Specifies how many decimal digits should be used when converting Postgres money type to java.math.BigDecimal, which represents the values in change events. Applicable only when decimal.handling.mode is set to precise.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| message.prefix.include.list                      | false    | No default       | An optional, comma-separated list of regular expressions that match the names of the logical decoding message prefixes that you want the connector to capture. By default, the connector captures all logical decoding messages. When this property is set, the connector captures only logical decoding message with the prefixes specified by the property. All other logical decoding messages are excluded.<br /><br />To match the name of a message prefix, Debezium applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire message prefix string; the expression does not match substrings that might be present in a prefix.<br /><br />If you include this property in the configuration, do not also set the message.prefix.exclude.list property.<br /><br />For information about the structure of message events and about their ordering semantics, see message events.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| message.prefix.exclude.list                      | false    | No default       | An optional, comma-separated list of regular expressions that match the names of the logical decoding message prefixes that you do not want the connector to capture. When this property is set, the connector does not capture logical decoding messages that use the specified prefixes. All other messages are captured.<br />To exclude all logical decoding messages, set the value of this property to .\*.<br /><br />To match the name of a message prefix, Debezium applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire message prefix string; the expression does not match substrings that might be present in a prefix.<br /><br />If you include this property in the configuration, do not also set message.prefix.include.list property.<br /><br /><br />For information about the structure of message events and about their ordering semantics, see message events.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |

For more information about the configuration properties, see the [Official Debezium PostgreSQL Connector documentation](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-connector-properties).


# Kafka connect debezium spanner
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-debezium-spanner/current/kafka-connect-debezium-spanner

Kafka Connect Debezium Cloud Spanner Source connector

The Debezium Cloud Spanner Source connector streams transactional change events from a Google Cloud Spanner database into Kafka topics.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

### Prerequisites

* Google Cloud project with a provisioned Spanner instance and database
* Service account key with `spanner.databaseReader` and `monitoring.viewer` roles
* Enable the Cloud Spanner change streams API in the target project

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)

2. Create a JSON file similar to the following:

   ```json theme={null}
   {
       "name": "debezium-spanner-source",
       "config": {
           "connector.class": "io.debezium.connector.spanner.SpannerConnector",
           "tasks.max": "1",
           "gcp.spanner.change.stream": "changeStreamAll",
           "gcp.spanner.project.id": "${GCP_PROJECT}",
           "gcp.spanner.instance.id": "${SPANNER_INSTANCE}",
           "gcp.spanner.database.id": "${SPANNER_DATABASE}",
           "gcp.spanner.credentials.json": "${SERVICE_ACCOUNT_JSON}",
           "sn.passthrough.bootstrapServer.fields": "connector.spanner.sync.kafka.bootstrap.servers",
           "sn.passthrough.kafka.client.field.prefixes": "kafka.internal.client"
       }
   }
   ```

   <Note>
     `sn.passthrough.bootstrapServer.fields` and `sn.passthrough.kafka.client.field.prefixes` are required to make StreamNative Cloud automatically configure the internal Kafka client settings for the connector.<br />
     You shouldn't change these properties unless you want to override the default behavior.
   </Note>

3. Deploy the connector:

   ```bash theme={null}
   kcctl apply -f <filename>.json
   ```

### Configuration

The Debezium Cloud Spanner Source connector accepts the following common options:

| Property                                                 | Required | Default                                                 | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| -------------------------------------------------------- | -------- | ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| name                                                     | true     | No default                                              | Unique name for the connector. Attempting to register again with the same name will fail. This property is required by all Kafka Connect connectors.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| connector.class                                          | true     | No default                                              | The name of the Java class for the connector. Always use a value of io.debezium.connector.spanner.SpannerConnector for the Spanner connector.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| tasks.max                                                | true     | 1                                                       | The maximum number of tasks that should be created for this connector. The Spanner connector can use more than 1 tasks if you enable offset.storage.per.task mode.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| gcp.spanner.project.id                                   | true     | No default                                              | The GCP project ID                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| gcp.spanner.instance.id                                  | true     | No default                                              | The Spanner instance ID                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| gcp.spanner.database.id                                  | true     | No default                                              | The Spanner database ID                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| gcp.spanner.change.stream                                | true     | No default                                              | The Spanner change stream                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| gcp.spanner.credentials.path                             | true     | No default                                              | The file path to the GCP service account key JSON.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| gcp.spanner.credentials.json                             | true     | No default                                              | The GCP service account key JSON. Required if gcp.spanner.credentials.path is not provided.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| schema.name.adjustment.mode                              | true     | none                                                    | Specifies how schema names should be adjusted for compatibility with the message converter used by the connector. Possible settings:<br /><br /><br />none does not apply any adjustment.<br /><br /><br />avro replaces the characters that cannot be used in the Avro type name with underscore.<br /><br /><br />avro\_unicode replaces the underscore or characters that cannot be used in the Avro type name with corresponding unicode like \_uxxxx. Note: \_ is an escape sequence like backslash in Java                                                                                                                                                                                           |
| field.name.adjustment.mode                               | true     | none                                                    | Specifies how field names should be adjusted for compatibility with the message converter used by the connector. Possible settings:<br /><br /><br />none does not apply any adjustment.<br /><br /><br />avro replaces the characters that cannot be used in the Avro type name with underscore.<br /><br /><br />avro\_unicode replaces the underscore or characters that cannot be used in the Avro type name with corresponding unicode like \_uxxxx. Note: \_ is an escape sequence like backslash in Java<br /><br /><br />See Avro naming for more details.                                                                                                                                         |
| gcp.spanner.low-watermark.enabled                        | false    | false                                                   | Whether or not the low watermark is enabled for the connector.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| gcp.spanner.low-watermark.update-period.ms               | false    | 1000 ms                                                 | The interval at which the low watermark is updated.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| heartbeat.interval.ms                                    | false    | 300000                                                  | The Spanner heartbeat interval.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| gcp.spanner.start.time                                   | false    | current time                                            | The connector start time.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| gcp.spanner.end.time                                     | false    | indefinite end time                                     | The connector end time.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| gcp.spanner.stream.event.queue.capacity                  | false    | 10000                                                   | The Spanner event queue capacity. Increase this capacity if the remaining stream event queue capacity approaches zero during connector runtime.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| connector.spanner.task.state.change.event.queue.capacity | false    | 1000                                                    | The task state change event queue capacity. Increase this capacity if the remaining task state change event queue capacity approaches zero during connector runtime.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| connector.spanner.max.missed.heartbeats                  | false    | 5                                                       | The maximum number of missed heartbeats for a change stream query before an exception is thrown                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| scaler.monitor.enabled                                   | false    | false                                                   | Whether or not task autoscaling is enabled                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| connector.spanner.sync.topic                             | false    | *sync\_topic\_spanner\_connector*\$connectorname        | The name for the Sync topic. The Sync topic is an internal connector topic used to store communication between tasks.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| connector.spanner.sync.poll.duration                     | false    | 500 ms                                                  | The poll duration for the sync topic.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| connector.spanner.sync.request.timeout.ms                | false    | 5000 ms                                                 | The timeout for requests to the sync topic.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| connector.spanner.sync.delivery.timeout.ms               | false    | 15000 ms                                                | The timeout for publishing to the sync topic.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| connector.spanner.sync.commit.offsets.timeout.ms         | false    | 5000 ms                                                 | The timeout for committing offsets for the sync topic.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| connector.spanner.sync.commit.offsets.interval.ms        | false    | 60000 ms                                                | The interval at which offsets are committed for the sync topic.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| connector.spanner.sync.publisher.wait.timeout            | false    | 5 ms                                                    | The interval at which messages are published to the sync topic.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| connector.spanner.rebalancing.topic                      | false    | *rebalancing\_topic\_spanner\_connector*\$connectorname | The name for the rebalancing topic. The rebalancing topic is an internal connector topic used to determine task aliveness.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| connector.spanner.rebalancing.poll.duration              | false    | 5000                                                    | The poll duration for the rebalancing topic.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| connector.spanner.rebalancing.commit.offsets.timeout     | false    | 5000                                                    | The timeout for committing offsets for the rebalance topic.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| connector.spanner.rebalancing.commit.offsets.interval.ms | false    | 60000 ms                                                | The interval at which offsets are committed for the sync topic.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| connector.spanner.rebalancing.task.waiting.timeout       | false    | 1000 ms                                                 | The duration of time a task waits before processing a rebalancing event.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| custom.metric.tags                                       | false    | No default                                              | Defines tags that customize MBean object names by adding metadata that provides contextual information. Specify a comma-separated list of key-value pairs. Each key represents a tag for the MBean object name, and the corresponding value represents a value for the key, for example,<br />k1=v1,k2=v2<br /><br />The connector appends the specified tags to the base MBean object name. Tags can help you to organize and categorize metrics data. You can define tags to identify particular application instances, environments, regions, versions, and so forth. For more information, see Customized MBean names.                                                                                 |
| errors.max.retries                                       | false    | -1                                                      | Specifies how the connector responds after an operation that results in a retriable error, such as a connection error.<br />Set one of the following options:<br /><br />-1<br /><br />No limit. The connector always restarts automatically, and retries the operation, regardless of the number of previous failures.<br /><br />0<br /><br />Disabled. The connector fails immediately, and never retries the operation. User intervention is required to restart the connector.<br /><br />> 0<br /><br />The connector restarts automatically until it reaches the specified maximum number of retries. After the next failure, the connector stops, and user intervention is required to restart it. |
| extended.headers.enabled                                 | false    | true                                                    | This property specifies whether Debezium adds context headers with the prefix \_\_debezium.context. to the messages that it emits.<br /><br />These headers are required by the OpenLineage integration and provide metadata that enables downstream processing systems to track and identify the sources of change events.<br /><br />The property adds following headers:<br /><br />\_\_debezium.context.connectorLogicalName<br /><br />The logical name of the Debezium connector.<br /><br />\_\_debezium.context.taskId<br /><br />The unique identifier of the connector task.<br /><br />\_\_debezium.context.connectorName<br /><br />The name of the Debezium connector.                        |

Refer to the [official Debezium Cloud Spanner documentation](https://debezium.io/documentation/reference/stable/connectors/spanner.html) for a complete property reference and advanced deployment guidance.


# Kafka connect debezium sqlserver
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-debezium-sqlserver/current/kafka-connect-debezium-sqlserver

Kafka Connect Debezium SQL Server Source connector

The Debezium SQL Server Source connector is a Kafka Connect connector that captures row-level changes in a SQL Server database and streams them to Kafka topics.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

### Prerequisites

* A running SQL Server
* Change Data Capture (CDC) enabled in SQL Server

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)

2. Create a JSON file like the following:

   ```json theme={null}
   {
       "name": "debezium-sqlserver-source",
       "config": {
           "connector.class": "io.debezium.connector.sqlserver.SqlServerConnector",
           "tasks.max": "1",
           "database.hostname": "{host}",
           "database.port": "1433",
           "database.user": "{username}",
           "database.password": "{password}",
           "database.names": "testDB1,testDB2",
           "topic.prefix": "sqlserver",
           "table.include.list": "dbo.customers",
           "schema.history.internal.kafka.bootstrap.servers": "kafka:9092",
           "schema.history.internal.kafka.topic": "dbhistory.fullfillment",
           "sn.passthrough.bootstrapServer.fields": "schema.history.internal.kafka.bootstrap.servers",
           "sn.passthrough.kafka.consumer.field.prefixes": "schema.history.internal",
           "sn.passthrough.kafka.producer.field.prefixes": "schema.history.internal"
       }
   }
   ```

   <Note>
     `sn.passthrough.bootstrapServer.fields` and `sn.passthrough.kafka.consumer|producer.field.prefixes` are required to make StreamNative Cloud automatically configure the schema history Kafka client settings for the connector.<br />
     You shouldn't change these properties unless you want to override the default behavior.
   </Note>

3. Run the following command to create the connector:

   ```bash theme={null}
   kcctl apply -f <filename>.json
   ```

### Configuration

The Debezium SQL Server Source connector is configured using the following properties:

| Property                                                   | Required | Default                                      | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| ---------------------------------------------------------- | -------- | -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| name                                                       | true     | No default                                   | Unique name for the connector. Attempting to register again with the same name will fail. (This property is required by all Kafka Connect connectors.)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| connector.class                                            | true     | No default                                   | The name of the Java class for the connector. Always use a value of io.debezium.connector.sqlserver.SqlServerConnector for the SQL Server connector.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| tasks.max                                                  | true     | 1                                            | Specifies the maximum number of tasks that the connector can use to capture data from the database instance. If the database.names list contains more than one element, you can increase the value of this property to a number less than or equal to the number of elements in the list.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| database.hostname                                          | true     | No default                                   | IP address or hostname of the SQL Server database server.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| database.port                                              | false    | 1433                                         | Integer port number of the SQL Server database server. If both database.port and database.instance are specified, database.instance is ignored. See JDBC driver for SQL server documentation for more details.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| database.user                                              | true     | No default                                   | Username to use when connecting to the SQL Server database server. Can be omitted when using Kerberos authentication, which can be configured using pass-through properties.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| database.password                                          | true     | No default                                   | Password to use when connecting to the SQL Server database server.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| database.instance                                          | false    | No default                                   | Specifies the instance name of the SQL Server named instance. If both database.port and database.instance are specified, database.instance is ignored. See JDBC driver for SQL server documentation for more details.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| database.names                                             | true     | No default                                   | The comma-separated list of the SQL Server database names from which to stream the changes.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| topic.prefix                                               | true     | No default                                   | Topic prefix that provides a namespace for the SQL Server database server that you want Debezium to capture. The prefix should be unique across all other connectors, since it is used as the prefix for all Kafka topic names that receive records from this connector. Only alphanumeric characters, hyphens, dots and underscores must be used in the database server logical name.<br /><br /><br /><br />Do not change the value of this property. If you change the name value, after a restart, instead of continuing to emit events to the original topics, the connector emits subsequent events to topics whose names are based on the new value. The connector is also unable to recover its database schema history topic.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| schema.include.list                                        | false    | No default                                   | An optional, comma-separated list of regular expressions that match names of schemas for which you want to capture changes. Any schema name not included in schema.include.list is excluded from having its changes captured. By default, the connector captures changes for all non-system schemas.<br /><br /><br />To match the name of a schema, Debezium applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire name string of the schema; it does not match substrings that might be present in a schema name.<br />If you include this property in the configuration, do not also set the schema.exclude.list property.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| schema.exclude.list                                        | false    | No default                                   | An optional, comma-separated list of regular expressions that match names of schemas for which you do not want to capture changes. Any schema whose name is not included in schema.exclude.list has its changes captured, with the exception of system schemas.<br /><br /><br />To match the name of a schema, Debezium applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire name string of the schema; it does not match substrings that might be present in a schema name.<br />If you include this property in the configuration, do not set the schema.include.list property.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| table.include.list                                         | false    | No default                                   | An optional comma-separated list of regular expressions that match fully-qualified table identifiers for tables that you want Debezium to capture. By default, the connector captures all non-system tables for the designated schemas. When this property is set, the connector captures changes only from the specified tables. Each identifier is of the form schemaName.tableName.<br /><br /><br />To match the name of a table, Debezium applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire name string of the table; it does not match substrings that might be present in a table name.<br />If you include this property in the configuration, do not also set the table.exclude.list property.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| table.exclude.list                                         | false    | No default                                   | An optional comma-separated list of regular expressions that match fully-qualified table identifiers for the tables that you want to exclude from being captured. Debezium captures all tables that are not included in table.exclude.list. Each identifier is of the form schemaName.tableName.<br /><br /><br />To match the name of a table, Debezium applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire name string of the table; it does not match substrings that might be present in a table name.<br />If you include this property in the configuration, do not also set the table.include.list property.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| column.include.list                                        | false    | empty string                                 | An optional comma-separated list of regular expressions that match the fully-qualified names of columns that should be included in the change event message values. Fully-qualified names for columns are of the form schemaName.tableName.columnName.<br /><br /><br />Each change event record that Debezium emits for a table includes an event key that contains fields for each column in the table’s primary key or unique key. To ensure that event keys are generated correctly, if you set this property, be sure to explicitly list the primary key columns of any captured tables.<br /><br />To match the name of a column, Debezium applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire name string of the column; it does not match substrings that might be present in a column name.<br />If you include this property in the configuration, do not also set the column.exclude.list property.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| column.exclude.list                                        | false    | empty string                                 | An optional comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event message values. Fully-qualified names for columns are of the form schemaName.tableName.columnName. Note that primary key columns are always included in the event’s key, also if excluded from the value.<br /><br /><br />To match the name of a column, Debezium applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire name string of the column; it does not match substrings that might be present in a column name.<br />If you include this property in the configuration, do not also set the column.include.list property.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| skip.messages.without.change                               | false    | false                                        | Specifies whether to skip publishing messages when there is no change in included columns. This would essentially filter messages if there is no change in columns included as per column.include.list or column.exclude.list properties.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| column.mask.hash.hashAlgorithm.with.salt.salt              | false    | n/a                                          | An optional, comma-separated list of regular expressions that match the fully-qualified names of character-based columns. Fully-qualified names for columns are of the form `<schemaName>.<tableName>.<columnName>`.<br />To match the name of a column Debezium applies the regular expression that you specify as an \_anchored regular expression. That is, the specified expression is matched against the entire name string of the column; the expression does not match substrings that might be present in a column name. In the resulting change event record, the values for the specified columns are replaced with pseudonyms.<br /><br /><br />A pseudonym consists of the hashed value that results from applying the specified hashAlgorithm and salt. Based on the hash function that is used, referential integrity is maintained, while column values are replaced with pseudonyms. Supported hash functions are described in the MessageDigest section of the Java Cryptography Architecture Standard Algorithm Name Documentation.<br /><br />In the following example, CzQMA0cB5K is a randomly selected salt.<br /><br /><br />column.mask.hash.SHA-256.with.salt.CzQMA0cB5K = inventory.orders.customerName, inventory.shipment.customerName<br /><br />If necessary, the pseudonym is automatically shortened to the length of the column. The connector configuration can include multiple properties that specify different hash algorithms and salts.<br /><br />Depending on the hashAlgorithm used, the salt selected, and the actual data set, the resulting data set might not be completely masked.<br /><br />Hashing strategy version 2 should be used to ensure fidelity if the value is being hashed in different places or systems.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| column.mask.hash.v2.hashAlgorithm.with.salt.salt           | false    | n/a                                          | An optional, comma-separated list of regular expressions that match the fully-qualified names of character-based columns. Fully-qualified names for columns are of the form `<schemaName>.<tableName>.<columnName>`.<br />To match the name of a column Debezium applies the regular expression that you specify as an \_anchored regular expression. That is, the specified expression is matched against the entire name string of the column; the expression does not match substrings that might be present in a column name. In the resulting change event record, the values for the specified columns are replaced with pseudonyms.<br /><br /><br />A pseudonym consists of the hashed value that results from applying the specified hashAlgorithm and salt. Based on the hash function that is used, referential integrity is maintained, while column values are replaced with pseudonyms. Supported hash functions are described in the MessageDigest section of the Java Cryptography Architecture Standard Algorithm Name Documentation.<br /><br />In the following example, CzQMA0cB5K is a randomly selected salt.<br /><br /><br />column.mask.hash.SHA-256.with.salt.CzQMA0cB5K = inventory.orders.customerName, inventory.shipment.customerName<br /><br />If necessary, the pseudonym is automatically shortened to the length of the column. The connector configuration can include multiple properties that specify different hash algorithms and salts.<br /><br />Depending on the hashAlgorithm used, the salt selected, and the actual data set, the resulting data set might not be completely masked.<br /><br />Hashing strategy version 2 should be used to ensure fidelity if the value is being hashed in different places or systems.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| time.precision.mode                                        | false    | adaptive                                     | Time, date, and timestamps can be represented with different kinds of precision, including: adaptive (the default) captures the time and timestamp values exactly as in the database using either millisecond, microsecond, or nanosecond precision values based on the database column’s type; or connect always represents time and timestamp values using Kafka Connect’s built-in representations for Time, Date, and Timestamp, which uses millisecond precision regardless of the database columns' precision. For more information, see temporal values.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| decimal.handling.mode                                      | false    | precise                                      | Specifies how the connector should handle values for DECIMAL and NUMERIC columns:<br /><br />precise (the default) represents them precisely using java.math.BigDecimal values represented in change events in a binary form.<br /><br />double represents them using double values, which may result in a loss of precision but is easier to use.<br /><br />string encodes values as formatted strings, which is easy to consume but semantic information about the real type is lost.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| include.schema.changes                                     | false    | true                                         | Boolean value that specifies whether the connector publishes changes in the database schema to a Kafka topic with the same name as the topic prefix. The connector records each schema change with a key that contains the database name, and a value that is a JSON structure that describes the schema update. This mechanism for recording schema changes is independent of the connector’s internal recording of changes to the database schema history.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| tombstones.on.delete                                       | false    | true                                         | Controls whether a delete event is followed by a tombstone event.<br /><br />true - a delete operation is represented by a delete event and a subsequent tombstone event.<br /><br />false - only a delete event is emitted.<br /><br />After a source record is deleted, emitting a tombstone event (the default behavior) allows Kafka to completely delete all events that pertain to the key of the deleted row in case log compaction is enabled for the topic.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| column.truncate.to.length.chars                            | false    | n/a                                          | An optional, comma-separated list of regular expressions that match the fully-qualified names of character-based columns. Set this property if you want to truncate the data in a set of columns when it exceeds the number of characters specified by the length in the property name. Set length to a positive integer value, for example, column.truncate.to.20.chars.<br /><br />The fully-qualified name of a column observes the following format: `<schemaName>.<tableName>.<columnName>`. To match the name of a column, Debezium applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire name string of the column; the expression does not match substrings that might be present in a column name.<br /><br />You can specify multiple properties with different lengths in a single configuration.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| column.mask.with.length.chars                              | false    | n/a                                          | An optional, comma-separated list of regular expressions that match the fully-qualified names of character-based columns. Set this property if you want the connector to mask the values for a set of columns, for example, if they contain sensitive data. Set length to a positive integer to replace data in the specified columns with the number of asterisk (\*) characters specified by the length in the property name. Set length to 0 (zero) to replace data in the specified columns with an empty string.<br /><br />The fully-qualified name of a column observes the following format: schemaName.tableName.columnName. To match the name of a column, Debezium applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire name string of the column; the expression does not match substrings that might be present in a column name.<br /><br />You can specify multiple properties with different lengths in a single configuration.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| column.propagate.source.type                               | false    | n/a                                          | An optional, comma-separated list of regular expressions that match the fully-qualified names of columns for which you want the connector to emit extra parameters that represent column metadata. When this property is set, the connector adds the following fields to the schema of event records:<br /><br />\_\_debezium.source.column.type<br /><br /><br />\_\_debezium.source.column.length<br /><br /><br />\_\_debezium.source.column.scale<br /><br /><br />These parameters propagate a column’s original type name and length (for variable-width types), respectively.<br />Enabling the connector to emit this extra data can assist in properly sizing specific numeric or character-based columns in sink databases.<br /><br />The fully-qualified name of a column observes the following format: schemaName.tableName.columnName.<br />To match the name of a column, Debezium applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire name string of the column; the expression does not match substrings that might be present in a column name.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| datatype.propagate.source.type                             | false    | n/a                                          | An optional, comma-separated list of regular expressions that specify the fully-qualified names of data types that are defined for columns in a database. When this property is set, for columns with matching data types, the connector emits event records that include the following extra fields in their schema:<br /><br />\_\_debezium.source.column.type<br /><br /><br />\_\_debezium.source.column.length<br /><br /><br />\_\_debezium.source.column.scale<br /><br /><br />These parameters propagate a column’s original type name and length (for variable-width types), respectively.<br />Enabling the connector to emit this extra data can assist in properly sizing specific numeric or character-based columns in sink databases.<br /><br />The fully-qualified name of a column observes the following format: schemaName.tableName.typeName.<br />To match the name of a data type, Debezium applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire name string of the data type; the expression does not match substrings that might be present in a type name.<br /><br />For the list of SQL Server-specific data type names, see the SQL Server data type mappings.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| message.key.columns                                        | false    | n/a                                          | A list of expressions that specify the columns that the connector uses to form custom message keys for change event records that it publishes to the Kafka topics for specified tables.<br /><br />By default, Debezium uses the primary key column of a table as the message key for records that it emits. In place of the default, or to specify a key for tables that lack a primary key, you can configure custom message keys based on one or more columns.<br /><br />To establish a custom message key for a table, list the table, followed by the columns to use as the message key. Each list entry takes the following format:<br /><br />`<fully-qualified_tableName>:<keyColumn>,<keyColumn>`<br /><br />To base a table key on multiple column names, insert commas between the column names.<br /><br />Each fully-qualified table name is a regular expression in the following format:<br /><br />`<schemaName>.<tableName>`<br /><br />The property can include entries for multiple tables. Use a semicolon to separate table entries in the list.<br /><br />The following example sets the message key for the tables inventory.customers and purchase.orders:<br /><br />inventory.customers:pk1,pk2;(.\*).purchaseorders:pk3,pk4<br /><br />For the table inventory.customer, the columns pk1 and pk2 are specified as the message key. For the purchaseorders tables in any schema, the columns pk3 and pk4 server as the message key.<br /><br />There is no limit to the number of columns that you use to create custom message keys. However, it’s best to use the minimum number that are required to specify a unique key.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| binary.handling.mode                                       | false    | bytes                                        | Specifies how binary (binary, varbinary) columns should be represented in change events, including: bytes represents binary data as byte array (default), base64 represents binary data as base64-encoded String, base64-url-safe represents binary data as base64-url-safe-encoded String, hex represents binary data as hex-encoded (base16) String                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| schema.name.adjustment.mode                                | false    | none                                         | Specifies how schema names should be adjusted for compatibility with the message converter used by the connector. Possible settings:<br /><br /><br />none does not apply any adjustment.<br /><br /><br />avro replaces the characters that cannot be used in the Avro type name with underscore.<br /><br /><br />avro\_unicode replaces the underscore or characters that cannot be used in the Avro type name with corresponding unicode like \_uxxxx. Note: \_ is an escape sequence like backslash in Java                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| field.name.adjustment.mode                                 | false    | none                                         | Specifies how field names should be adjusted for compatibility with the message converter used by the connector. Possible settings:<br /><br /><br />none does not apply any adjustment.<br /><br /><br />avro replaces the characters that cannot be used in the Avro type name with underscore.<br /><br /><br />avro\_unicode replaces the underscore or characters that cannot be used in the Avro type name with corresponding unicode like \_uxxxx. Note: \_ is an escape sequence like backslash in Java<br /><br /><br />For more information, see Avro naming.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| converters                                                 | false    | No default                                   | Enumerates a comma-separated list of the symbolic names of the custom converter instances that the connector can use. For example,<br /><br /><br />isbn<br /><br />You must set the converters property to enable the connector to use a custom converter.<br /><br />For each converter that you configure for a connector, you must also add a .type property, which specifies the fully-qualified name of the class that implements the converter interface. The .type property uses the following format:<br /><br /><br />`<converterSymbolicName>.type`<br /><br /><br />For example,<br /><br /><br />isbn.type: io.debezium.test.IsbnConverter<br /><br />If you want to further control the behavior of a configured converter, you can add one or more configuration parameters to pass values to the converter. To associate any additional configuration parameter with a converter, prefix the parameter names with the symbolic name of the converter. For example,<br /><br /><br />isbn.schema.name: io.debezium.sqlserver.type.Isbn                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| snapshot.mode                                              | false    | initial                                      | A mode for taking an initial snapshot of the structure and optionally data of captured tables. Once the snapshot is complete, the connector will continue reading change events from the database’s redo logs. The following values are supported:<br /><br />always<br /><br />Perform snapshot on each connector start. After the snapshot completes, the connector begins to stream event records for subsequent database changes.<br /><br />initial<br /><br />The connector performs a database snapshot as described in the default workflow for creating an initial snapshot. After the snapshot completes, the connector begins to stream event records for subsequent database changes.<br /><br />initial\_only<br /><br />The connector performs a database snapshot and stops before streaming any change event records, not allowing any subsequent change events to be captured.<br /><br />schema\_only<br /><br />Deprecated, see no\_data.<br /><br />no\_data<br /><br />The connector captures the structure of all relevant tables, performing all the steps described in the default snapshot workflow, except that it does not create READ events to represent the data set at the point of the connector’s start-up (Step 7.b).<br /><br />recovery<br /><br />Set this option to restore a database schema history topic that is lost or corrupted. After a restart, the connector runs a snapshot that rebuilds the topic from the source tables. You can also set the property to periodically prune a database schema history topic that experiences unexpected growth.<br /><br /><br />Do not use this mode to perform a snapshot if schema changes were committed to the database after the last connector shutdown.<br />when\_needed<br /><br />After the connector starts, it performs a snapshot only if it detects one of the following circumstances:<br /><br />It cannot detect any topic offsets.<br /><br />A previously recorded offset specifies a log position that is not available on the server.<br /><br />configuration\_based<br /><br />With this option, you control snapshot behavior through a set of connector properties that have the prefix 'snapshot.mode.configuration.based'.<br /><br />custom<br /><br />The custom snapshot mode lets you inject your own implementation of the io.debezium.spi.snapshot.Snapshotter interface. Set the snapshot.mode.custom.name configuration property to the name provided by the name() method of your implementation.<br /><br />For more information, see custom snapshotter SPI. |
| snapshot.mode.configuration.based.snapshot.data            | false    | false                                        | If the snapshot.mode is set to configuration\_based, set this property to specify whether the connector includes table data when it performs a snapshot.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| snapshot.mode.configuration.based.snapshot.schema          | false    | false                                        | If the snapshot.mode is set to configuration\_based, set this property to specify whether the connector includes the table schema when it performs a snapshot.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| snapshot.mode.configuration.based.start.stream             | false    | false                                        | If the snapshot.mode is set to configuration\_based, set this property to specify whether the connector begins to stream change events after a snapshot completes.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| snapshot.mode.configuration.based.snapshot.on.schema.error | false    | false                                        | If the snapshot.mode is set to configuration\_based, set this property to specify whether the connector includes table schema in a snapshot if the schema history topic is not available.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| snapshot.mode.configuration.based.snapshot.on.data.error   | false    | false                                        | If the snapshot.mode is set to configuration\_based, this property specifies whether the connector attempts to snapshot table data if it does not find the last committed offset in the transaction log.<br />Set the value to true to instruct the connector to perform a new snapshot.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| snapshot.mode.custom.name                                  | false    | No default                                   | If snapshot.mode is set to custom, use this setting to specify the name of the custom implementation that is provided in the name() method that is defined in the 'io.debezium.spi.snapshot.Snapshotter' interface. After a connector restart, Debezium calls the specified custom implementation to determine whether to perform a snapshot. For more information, see custom snapshotter SPI.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| snapshot.locking.mode                                      | false    | exclusive                                    | Controls whether and for how long the connector holds a table lock. Table locks prevent certain types of changes table operations from occurring while the connector performs a snapshot. You can set the following values:<br /><br />exclusive<br /><br />Controls how the connector holds locks on tables while performing the schema snapshot when snapshot.isolation.mode is REPEATABLE\_READ or EXCLUSIVE.<br />The connector will hold a table lock for exclusive table access for just the initial portion of the snapshot while the database schemas and other metadata are being read. The remaining work in a snapshot involves selecting all rows from each table, and this is done using a flashback query that requires no locks. However, in some cases it may be desirable to avoid locks entirely which can be done by specifying none. This mode is only safe to use if no schema changes are happening while the snapshot is taken.<br /><br />none<br /><br />Prevents the connector from acquiring any table locks during the snapshot. Use this setting only if no schema changes might occur during the creation of the snapshot.<br /><br />custom<br /><br />The connector performs a snapshot according to the implementation specified by the snapshot.locking.mode.custom.name property, which is a custom implementation of the io.debezium.spi.snapshot.SnapshotLock interface.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| snapshot.locking.mode.custom.name                          | false    | No default                                   | When snapshot.locking.mode is set as custom, use this setting to specify the name of the custom implementation provided in the name() method that is defined by the 'io.debezium.spi.snapshot.SnapshotLock' interface. For more information, see custom snapshotter SPI.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| snapshot.query.mode                                        | false    | select\_all                                  | Specifies how the connector queries data while performing a snapshot.<br />Set one of the following options:<br /><br />select\_all<br /><br />The connector performs a select all query by default, optionally adjusting the columns selected based on the column include and exclude list configurations.<br /><br />custom<br /><br />The connector performs a snapshot query according to the implementation specified by the snapshot.query.mode.custom.name property, which defines a custom implementation of the io.debezium.spi.snapshot.SnapshotQuery interface.<br /><br /><br />This setting enables you to manage snapshot content in a more flexible manner compared to using the snapshot.select.statement.overrides property.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| snapshot.query.mode.custom.name                            | false    | No default                                   | When snapshot.query.mode is set to custom, use this setting to specify the name of the custom implementation provided in the name() method that is defined by the 'io.debezium.spi.snapshot.SnapshotQuery' interface. For more information, see custom snapshotter SPI.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| snapshot.include.collection.list                           | false    | All tables specified in table.include.list   | An optional, comma-separated list of regular expressions that match the fully-qualified names (`<dbName>.<schemaName>.<tableName>`) of the tables to include in a snapshot. The specified items must be named in the connector’s table.include.list property. This property takes effect only if the connector’s snapshot.mode property is set to a value other than never.<br />This property does not affect the behavior of incremental snapshots.<br /><br /><br />To match the name of a table, Debezium applies the regular expression that you specify as an anchored regular expression. That is, the specified expression is matched against the entire name string of the table; it does not match substrings that might be present in a table name.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| snapshot.isolation.mode                                    | false    | repeatable\_read                             | Mode to control which transaction isolation level is used and how long the connector locks tables that are designated for capture. The following values are supported:<br /><br />read\_uncommitted<br /><br />read\_committed<br /><br />repeatable\_read<br /><br />snapshot<br /><br />exclusive (exclusive mode uses repeatable read isolation level, however, it takes the exclusive lock on all tables to be read).<br /><br /><br />The snapshot, read\_committed and read\_uncommitted modes do not prevent other transactions from updating table rows during initial snapshot. The exclusive and repeatable\_read modes do prevent concurrent updates.<br /><br /><br />Mode choice also affects data consistency. Only exclusive and snapshot modes guarantee full consistency, that is, initial snapshot and streaming logs constitute a linear history. In case of repeatable\_read and read\_committed modes, it might happen that, for instance, a record added appears twice - once in initial snapshot and once in streaming phase. Nonetheless, that consistency level should do for data mirroring. For read\_uncommitted there are no data consistency guarantees at all (some data might be lost or corrupted).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| event.processing.failure.handling.mode                     | false    | fail                                         | Specifies how the connector should react to exceptions during processing of events. fail will propagate the exception (indicating the offset of the problematic event), causing the connector to stop.<br />warn will cause the problematic event to be skipped and the offset of the problematic event to be logged.<br />skip will cause the problematic event to be skipped.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| poll.interval.ms                                           | false    | 500 (0.5 seconds)                            | Positive integer value that specifies the number of milliseconds that the connector waits before it checks the database for new change events.<br /><br />The value that you specify influences the behavior of heartbeat.interval.ms. The connector can emit heartbeat messages only during the specified polling cycle.<br /><br /><br />To prevent this setting from delaying heartbeat emissions, set it to a value that is less than or equal to the value of heartbeat.interval.ms.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| max.queue.size                                             | false    | 8192                                         | Positive integer value that specifies the maximum number of records that the blocking queue can hold. When Debezium reads events streamed from the database, it places the events in the blocking queue before it writes them to Kafka. The blocking queue can provide backpressure for reading change events from the database in cases where the connector ingests messages faster than it can write them to Kafka, or when Kafka becomes unavailable. Events that are held in the queue are disregarded when the connector periodically records offsets. Always set the value of max.queue.size to be larger than the value of max.batch.size.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| max.queue.size.in.bytes                                    | false    | 0                                            | A long integer value that specifies the maximum volume of the blocking queue in bytes. By default, volume limits are not specified for the blocking queue. To specify the number of bytes that the queue can consume, set this property to a positive long value.<br />If max.queue.size is also set, writing to the queue is blocked when the size of the queue reaches the limit specified by either property. For example, if you set max.queue.size=1000, and max.queue.size.in.bytes=5000, writing to the queue is blocked after the queue contains 1000 records, or after the volume of the records in the queue reaches 5000 bytes.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| max.batch.size                                             | false    | 2048                                         | Positive integer value that specifies the maximum size of each batch of events that should be processed during each iteration of this connector.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| heartbeat.interval.ms                                      | false    | 0                                            | Specifies an interval in milliseconds that determines how frequently the connector sends messages to a Kafka heartbeat topic, regardless of whether changes occur in the database.<br />By default, the connector does not send heartbeat messages.<br /><br />Setting this property can help to confirm whether the connector is still receiving change events from the database. This can be especially important in databases where captured tables remain unchanged for long periods. When a database experiences frequent long intervals during which no changes occur in captured tables, although the connector continues to read from the transaction log as usual, it only rarely commits offset values to Kafka. As a result, after a connector restart, because the offset value is stale, the connector must send a high number of change events.<br /><br />By contrast, when you configure the connector to send regular heartbeat messages, it can update the offset in Kafka more frequently. Because the offset values in Kafka remain current, fewer change events must be re-sent after a connector restarts.<br /><br />Heartbeats are only emitted during polling cycles. That is, in a Debezium environment, the actual interval between sending heartbeat messages is jointly controlled by the settings of the heartbeat.interval.ms and poll.interval.ms properties. The actual frequency for sending heartbeat messages is based on the lower of the two values. To prevent delays in sending heartbeat messages, reducing their effectiveness, set this property to a value that is greater than or equal to the value of poll.interval.ms. For example, if you set poll.interval.ms to 100, set heartbeat.interval.ms to 5000.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| heartbeat.action.query                                     | false    | No default                                   | Specifies a query that the connector executes on the source database when the connector sends a heartbeat message.<br /><br />This is useful for keeping offsets from becoming stale when capturing changes from a low-traffic database. Create a heartbeat table in the low-traffic database, and set this property to a statement that inserts records into that table, for example:<br /><br />INSERT INTO test\_heartbeat\_table (text) VALUES ('test\_heartbeat')<br /><br />This allows the connector to receive changes from the low-traffic database and acknowledge their LSNs, which prevents offsets from become stale.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| snapshot.delay.ms                                          | false    | No default                                   | An interval in milli-seconds that the connector should wait before taking a snapshot after starting up;<br />Can be used to avoid snapshot interruptions when starting multiple connectors in a cluster, which may cause re-balancing of connectors.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| streaming.delay.ms                                         | false    | 0                                            | Specifies the time, in milliseconds, that the connector delays the start of the streaming process after it completes a snapshot. Setting a delay interval helps to prevent the connector from restarting snapshots in the event that a failure occurs immediately after the snapshot completes, but before the streaming process begins. Set a delay value that is higher than the value of the offset.flush.interval.ms property that is set for the Kafka Connect worker.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| snapshot.fetch.size                                        | false    | 2000                                         | Specifies the maximum number of rows that should be read in one go from each table while taking a snapshot. The connector will read the table contents in multiple batches of this size. Defaults to 2000.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| query.fetch.size                                           | false    | No default                                   | Specifies the number of rows that will be fetched for each database round-trip of a given query. Defaults to the JDBC driver’s default fetch size.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| snapshot.lock.timeout.ms                                   | false    | 10000                                        | An integer value that specifies the maximum amount of time (in milliseconds) to wait to obtain table locks when performing a snapshot. If table locks cannot be acquired in this time interval, the snapshot will fail (also see snapshots).<br />When set to 0 the connector will fail immediately when it cannot obtain the lock. Value -1 indicates infinite waiting.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| snapshot.select.statement.overrides                        | false    | No default                                   | Specifies the table rows to include in a snapshot. Use the property if you want a snapshot to include only a subset of the rows in a table. This property affects snapshots only. It does not apply to events that the connector reads from the log.<br /><br />The property contains a comma-separated list of fully-qualified table names in the form `<schemaName>.<tableName>`. For example,<br /><br />"snapshot.select.statement.overrides": "inventory.products,customers.orders"<br /><br />For each table in the list, add a further configuration property that specifies the SELECT statement for the connector to run on the table when it takes a snapshot. The specified SELECT statement determines the subset of table rows to include in the snapshot. Use the following format to specify the name of this SELECT statement property:<br /><br />snapshot.select.statement.overrides.`<schemaName>.<tableName>`. For example, snapshot.select.statement.overrides.customers.orders.<br /><br />Example:<br /><br />From a customers.orders table that includes the soft-delete column, delete\_flag, add the following properties if you want a snapshot to include only those records that are not soft-deleted:<br /><br />"snapshot.select.statement.overrides": "customer.orders",<br />"snapshot.select.statement.overrides.customer.orders": "SELECT \* FROM customers.orders WHERE delete\_flag = 0 ORDER BY id DESC"<br /><br />In the resulting snapshot, the connector includes only the records for which delete\_flag = 0.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| source.struct.version                                      | false    | v2                                           | Schema version for the source block in CDC events; Debezium 0.10 introduced a few breaking<br />changes to the structure of the source block in order to unify the exposed structure across all the connectors.<br />By setting this option to v1 the structure used in earlier versions can be produced. Note that this setting is not recommended and is planned for removal in a future Debezium version.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| provide.transaction.metadata                               | false    | false                                        | When set to true Debezium generates events with transaction boundaries and enriches data events envelope with transaction metadata.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| retriable.restart.connector.wait.ms                        | false    | 10000 (10 seconds)                           | The number of milli-seconds to wait before restarting a connector after a retriable error occurs.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| skipped.operations                                         | false    | t                                            | A comma-separated list of the operation types that you want the connector to skip during streaming. You can configure the connector to skip the following types of operations:<br /><br />c (insert/create)<br /><br />u (update)<br /><br />d (delete)<br /><br />t (truncate)<br /><br />Set the value to none if you do not want the connector to skip any operations. Because the Debezium SQL Server connector does not support truncate change events, setting the default t value has the same effect as setting the value to none.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| signal.data.collection                                     | false    | No default value                             | Fully-qualified name of the data collection that is used to send signals to the connector.<br />Use the following format to specify the collection name:<br />`<databaseName>.<schemaName>.<tableName>`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| signal.enabled.channels                                    | false    | source                                       | List of the signaling channel names that are enabled for the connector. By default, the following channels are available:<br /><br />source<br /><br />kafka<br /><br />file<br /><br />jmx Optionally, you can also implement a custom signaling channel.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| notification.enabled.channels                              | false    | No default                                   | List of notification channel names that are enabled for the connector. By default, the following channels are available:<br /><br />sink<br /><br />log<br /><br />jmx Optionally, you can also implement a custom notification channel.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| incremental.snapshot.allow\.schema.changes                 | false    | false                                        | Allow schema changes during an incremental snapshot. When enabled the connector will detect schema change during an incremental snapshot and re-select a current chunk to avoid locking DDLs.<br /><br />Note that changes to a primary key are not supported and can cause incorrect results if performed during an incremental snapshot. Another limitation is that if a schema change affects only columns' default values, then the change won’t be detected until the DDL is processed from the transaction log stream. This doesn’t affect the snapshot events' values, but the schema of snapshot events may have outdated defaults.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| incremental.snapshot.chunk.size                            | false    | 1024                                         | The maximum number of rows that the connector fetches and reads into memory during an incremental snapshot chunk. Increasing the chunk size provides greater efficiency, because the snapshot runs fewer snapshot queries of a greater size. However, larger chunk sizes also require more memory to buffer the snapshot data. Adjust the chunk size to a value that provides the best performance in your environment.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| incremental.snapshot.watermarking.strategy                 | false    | insert\_insert                               | Specifies the watermarking mechanism that the connector uses during an incremental snapshot to deduplicate events that might be captured by an incremental snapshot and then recaptured after streaming resumes.<br />You can specify one of the following options:<br /><br />insert\_insert<br /><br />When you send a signal to initiate an incremental snapshot, for every chunk that Debezium reads during the snapshot, it writes an entry to the signaling data collection to record the signal to open the snapshot window. After the snapshot completes, Debezium inserts a second entry that records the signal to close the window.<br /><br />insert\_delete<br /><br />When you send a signal to initiate an incremental snapshot, for every chunk that Debezium reads, it writes a single entry to the signaling data collection to record the signal to open the snapshot window. After the snapshot completes, this entry is removed. No entry is created for the signal to close the snapshot window. Set this option to prevent rapid growth of the signaling data collection.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| max.iteration.transactions                                 | false    | 500                                          | Specifies the maximum number of transactions per iteration to be used to reduce the memory footprint when streaming changes from multiple tables in a database. When set to 0, the connector uses the current maximum LSN as the range to fetch changes from. When set to a value greater than zero, the connector uses the n-th LSN specified by this setting as the range to fetch changes from. Defaults to 500.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| incremental.snapshot.option.recompile                      | false    | false                                        | Uses OPTION(RECOMPILE) query option to all SELECT statements used during an incremental snapshot. This can help to solve parameter sniffing issues that may occur but can cause increased CPU load on the source database, depending on the frequency of query execution.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| topic.naming.strategy                                      | false    | io.debezium.schema.SchemaTopicNamingStrategy | The name of the TopicNamingStrategy class that should be used to determine the topic name for data change, schema change, transaction, heartbeat event etc., defaults to SchemaTopicNamingStrategy.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| topic.delimiter                                            | false    | .                                            | Specify the delimiter for topic name, defaults to ..                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| topic.cache.size                                           | false    | 10000                                        | The size used for holding the topic names in bounded concurrent hash map. This cache will help to determine the topic name corresponding to a given data collection.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| topic.heartbeat.prefix                                     | false    | \_\_debezium-heartbeat                       | Controls the name of the topic to which the connector sends heartbeat messages. The topic name has this pattern:<br /><br />topic.heartbeat.prefix.topic.prefix<br /><br />For example, if the topic prefix is fulfillment, the default topic name is \_\_debezium-heartbeat.fulfillment.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| topic.transaction                                          | false    | transaction                                  | Controls the name of the topic to which the connector sends transaction metadata messages. The topic name has this pattern:<br /><br />topic.prefix.topic.transaction<br /><br />For example, if the topic prefix is fulfillment, the default topic name is fulfillment.transaction.<br /><br />For more information, see Transaction Metadata.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| snapshot.max.threads                                       | false    | 1                                            | Specifies the number of threads that the connector uses when performing an initial snapshot. To enable parallel initial snapshots, set the property to a value greater than 1. In a parallel initial snapshot, the connector processes multiple tables concurrently.<br /><br /><br />When you enable parallel initial snapshots, the threads that perform each table snapshot can require varying times to complete their work. If a snapshot for one table requires significantly more time to complete than the snapshots for other tables, threads that have completed their work sit idle. In some environments, a network device such as a load balancer or firewall, terminates connections that remain idle for an extended interval. After the snapshot completes, the connector is unable to close the connection, resulting in an exception, and an incomplete snapshot, even in cases where the connector successfully transmitted all snapshot data.<br /><br />If you experience this problem, revert the value of snapshot.max.threads to 1, and retry the snapshot.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| custom.metric.tags                                         | false    | No default                                   | Defines tags that customize MBean object names by adding metadata that provides contextual information. Specify a comma-separated list of key-value pairs. Each key represents a tag for the MBean object name, and the corresponding value represents a value for the key, for example,<br />k1=v1,k2=v2<br /><br />The connector appends the specified tags to the base MBean object name. Tags can help you to organize and categorize metrics data. You can define tags to identify particular application instances, environments, regions, versions, and so forth. For more information, see Customized MBean names.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| errors.max.retries                                         | false    | -1                                           | Specifies how the connector responds after an operation that results in a retriable error, such as a connection error.<br />Set one of the following options:<br /><br />-1<br /><br />No limit. The connector always restarts automatically, and retries the operation, regardless of the number of previous failures.<br /><br />0<br /><br />Disabled. The connector fails immediately, and never retries the operation. User intervention is required to restart the connector.<br /><br />> 0<br /><br />The connector restarts automatically until it reaches the specified maximum number of retries. After the next failure, the connector stops, and user intervention is required to restart it.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| data.query.mode                                            | false    | function                                     | Controls how the connector queries CDC data. The following modes are supported:<br /><br />function: The data is queried by calling cdc.\[fn\_cdc\_get\_all\_changes\_#] function. This is the default mode.<br /><br />direct: Makes the connector to query change tables directly.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| database.query.timeout.ms                                  | false    | 600000 (10 minutes)                          | Specifies the time, in milliseconds, that the connector waits for a query to complete. Set the value to 0 (zero) to remove the timeout limit.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| streaming.fetch.size                                       | false    | 0                                            | Specifies the maximum number of rows that should be read in one go from each table while streaming. The connector will read the table contents in multiple batches of this size. Defaults to 0 which means no limit.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |

For more information about the configuration properties, see the [Official Debezium SQL Server Connector documentation](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html#sqlserver-connector-properties).


# Kafka connect elasticsearch sink
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-elasticsearch-sink/current/kafka-connect-elasticsearch-sink

The official ElasticSearch Kafka Connect Sink connector.

The ElasticSearch Kafka Connect Sink connector is a Kafka Connect connector that writes data from Kafka topics to ElasticSearch.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

### Prerequisites

* A running ElasticSearch cluster

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)
2. Set up a ElasticSearch cluster
3. Create a JSON file like the following:

   ```json theme={null}
   {
       "name": "elasticsearch-sink",
       "config": {
           "connector.class": "io.aiven.connect.elasticsearch.ElasticsearchSinkConnector",
           "tasks.max": "1",
           "topics": "kafka-elastic-input",
           "connection.url": "http://elastic:9200",
           "type.name": "kafka-connect",
           "key.ignore": "true",
           "schema.ignore": "true",
           "value.converter": "org.apache.kafka.connect.json.JsonConverter",
           "value.converter.schemas.enable": "false"
       }
   }
   ```
4. Run the following command to create the connector:

   ```bash theme={null}
   kcctl apply -f <filename>.json
   ```

### Configuration

The ElasticSearch Kafka Connect Sink connector is configured using the following properties:

| Parameter                       | Required | Description                                                                                                                                       | Default |   |
| ------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | - |
| connection.url                  | true     | List of Elasticsearch HTTP connection URLs e.g. `http://eshost1:9200, http://eshost2:9200`                                                        |         |   |
| type.name                       | true     | The Elasticsearch type name to use when indexing.                                                                                                 |         |   |
| connection.username             | false    | The username used to authenticate with Elasticsearch.                                                                                             |         |   |
| connection.password             | false    | The password used to authenticate with Elasticsearch.                                                                                             |         |   |
| batch.size                      | false    | The number of records to process as a batch when writing to Elasticsearch.                                                                        | 2000    |   |
| max.in.flight.requests          | false    | The maximum number of indexing requests that can be in-flight to Elasticsearch before blocking further requests.                                  | 5       |   |
| max.buffered.records            | false    | The maximum number of records each task will buffer before blocking acceptance of more records.                                                   | 20000   |   |
| linger.ms                       | false    | Linger time in milliseconds for batching.                                                                                                         | 1       |   |
| flush.timeout.ms                | false    | The timeout in milliseconds to use for periodic flushing.                                                                                         | 10000   |   |
| max.retries                     | false    | The maximum number of retries that are allowed for failed indexing requests.                                                                      | 5       |   |
| retry.backoff.ms                | false    | How long to wait in milliseconds before attempting the first retry of a failed indexing.                                                          | 100     |   |
| key.ignore                      | false    | Whether to ignore the record key for the purpose of forming the Elasticsearch document ID.                                                        | false   |   |
| topic.key.ignore                | false    | List of topics for which `key.ignore` should be true.                                                                                             |         |   |
| schema.ignore                   | false    | Whether to ignore schemas during indexing.                                                                                                        | false   |   |
| topic.schema.ignore             | false    | List of topics for which `schema.ignore` should be true.                                                                                          |         |   |
| drop.invalid.message            | false    | Whether to drop kafka message when it cannot be converted to output message.                                                                      | false   |   |
| compact.map.entries             | false    | Defines how map entries with string keys within record values should be written to JSON.                                                          | true    |   |
| connection.timeout.ms           | false    | How long to wait in milliseconds when establishing a connection to the Elasticsearch server.                                                      | 1000    |   |
| read.timeout.ms                 | false    | How long to wait in milliseconds for the Elasticsearch server to send a response.                                                                 | 3000    |   |
| behavior.on.null.values         | false    | How to handle records with a non-null key and a null value, Valid options are 'ignore', 'delete', and 'fail'.                                     | ignore  |   |
| behavior.on.malformed.documents | false    | How to handle records that Elasticsearch rejects due to some malformation of the document itself, Valid options are 'ignore', 'warn', and 'fail'. | fail    |   |

For more information about the configuration properties, see the [Official ElasticSearch Kafka Connect Sink Connector documentation](https://github.com/Aiven-Open/elasticsearch-connector-for-apache-kafka/blob/v7.0.0/README.md).


# Kafka connect google bigtable sink
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-google-bigtable-sink/current/kafka-connect-google-bigtable-sink

The official Google Cloud Bigtable Sink connector.

The Kafka Connect Bigtable sink is a dedicated connector designed to stream data into Bigtable in real time with as little latency as possible.

### Prerequisites

You must have a GCP project in order to use Cloud Bigtable.

Follow these [setup steps](https://docs.cloud.google.com/bigtable/docs/overview)
for Bigtable before doing the [quickstart](#quick-start).

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)
2. Create a Bigtable instance in your GCP project.
3. (Optional) Create a Bigtable table with column families to receive data(you can set the connector to automatically create tables and column families if they do not exist).
4. Create a JSON file like the following:

```json theme={null}
{
    "name": "bigtable-sink",
    "config": {
        "connector.class": "com.google.cloud.kafka.connect.bigtable.BigtableSinkConnector",
        "gcp.bigtable.project.id": "${GCP_PROJECT_ID}",
        "gcp.bigtable.instance.id": "${BIGTABLE_INSTANCE_ID}",
        "gcp.bigtable.credentials.json": "${GCP_CREDENTIALS_JSON}",
        "topics": "${KAFKA_TOPIC_NAME}",
        "key.converter": "org.apache.kafka.connect.storage.StringConverter",
        "value.converter": "org.apache.kafka.connect.json.JsonConverter",
        "value.converter.schemas.enable": "false"
    }
}
```

5. Run the following command to create the connector:

```shell theme={null}
kcctl apply -f <filename>.json
```

### Configuration

The Google Bigtable sink connector is configured using the following properties:

| Config                        | Type    | Required              | Description                                                                                                                                 | Default      |
| ----------------------------- | ------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | ------------ |
| gcp.bigtable.project.id       | String  | REQUIRED (No default) | The ID of the GCP project.                                                                                                                  |              |
| gcp.bigtable.instance.id      | String  | REQUIRED (No default) | The ID of the Cloud Bigtable instance.                                                                                                      |              |
| gcp.bigtable.app.profile.id   | String  | Optional              | The application profile that the connector should use.                                                                                      | default      |
| gcp.bigtable.credentials.path | String  | Optional              | The path to the JSON service key file.                                                                                                      |              |
| gcp.bigtable.credentials.json | String  | Optional              | GCP credentials JSON blob.                                                                                                                  |              |
| insert.mode                   | String  | Optional              | Defines the insertion mode to use. Supported modes are: INSERT, UPSERT                                                                      | INSERT       |
| max.batch.size                | Int     | Optional              | The maximum number of records that can be batched into a batch of upserts.                                                                  | 1            |
| value.null.mode               | String  | Optional              | Defines what to do with `null`s within Kafka values. Supported modes are: WRITE, IGNORE, DELETE.                                            | WRITE        |
| error.mode                    | String  | Optional              | Specifies how to handle errors that result from writes, after retries. Supported modes are: FAIL, WARN, IGNORE                              | FAIL         |
| table.name.format             | String  | Optional              | Name of the destination table. Use `${topic}` within the table name to specify the originating topic name.                                  | `${topic}`   |
| row\.key.definition           | String  | Optional              | A comma separated list of Kafka Record key field names that specifies the order of Kafka key fields to be concatenated to form the row key. |              |
| row\.key.delimiter            | String  | Optional              | The delimiter used in concatenating Kafka key fields in the row key.                                                                        |              |
| auto.create.tables            | Boolean | Optional              | Whether to automatically create the destination table if it is found to be mission.                                                         | false        |
| auto.create.column.families   | Boolean | Optional              | Whether to automatically create missing columns families in the table relative to the record schema.                                        | false        |
| default.column.family         | String  | Optional              | Any root-level fields on the SinkRecord that aren't objects will be added to this column family.                                            | `${topic}`   |
| default.column.qualifier      | String  | Optional              | Any root-level values on the SinkRecord that aren't objects will be added to this column within default column family.                      | KAFKA\_VALUE |
| retry.timeout.ms              | Long    | Optional              | Maximum time in milliseconds allocated for retrying database operations before trying other error handling mechanisms.                      | 90000        |


# Kafka connect google cloud storage sink
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-google-cloud-storage-sink/current/kafka-connect-google-cloud-storage-sink

The official Google Cloud Storage Kafka Connect Sink connector.

This is a sink Apache Kafka Connect connector that stores Kafka messages in a Google Cloud Storage (GCS) bucket.

### Prerequisites

You must have a GCP project in order to use GCS.

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)
2. Create a GCS bucket in your GCP project.
3. Create a JSON file like the following:

   ```json theme={null}
   {
       "name": "gcs-sink",
       "config": {
           "connector.class": "io.aiven.kafka.connect.gcs.GcsSinkConnector",
           "tasks.max": "1",
           "topics": "kafka-gcs-input",
           "format.output.type": "json",
           "gcs.bucket.name": "${GCS_BUCKET_NAME}"
       }
   }
   ```
4. Run the following command to create the connector:

   ```bash theme={null}
   kcctl apply -f <filename>.json
   ```

### Configuration

The GCS Kafka Connect Sink connector is configured using the following properties:

| Parameter                                       | Required | Description                                                                                                                                                                                                                                                                       | Default                                                                |
| ----------------------------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| connector.class                                 | Yes      | The Java class for the GCS Sink connector.                                                                                                                                                                                                                                        |                                                                        |
| tasks.max                                       | Yes      | The maximum number of tasks that should be created for this connector.                                                                                                                                                                                                            |                                                                        |
| topics                                          | No       | A comma-separated list of Kafka topics to consume from, Only one of topics or topics.regex should be specified.                                                                                                                                                                   |                                                                        |
| topics.regex                                    | No       | Regular expression giving topics to consume. Under the hood, the regex is compiled to a java.util.regex.Pattern. Only one of topics or topics.regex should be specified.                                                                                                          |                                                                        |
| gcs.bucket.name                                 | Yes      | The name of the GCS bucket where the data will be stored.                                                                                                                                                                                                                         |                                                                        |
| gcs.credentials.json                            | No       | The GCP credentials in JSON format. If not provided, the connector will use the default application credentials.                                                                                                                                                                  |                                                                        |
| gcs.credentials.path                            | No       | The path to a GCP credentials file. Cannot be set together with "gcs.credentials.json or "gcs.credentials.default.                                                                                                                                                                |                                                                        |
| gcs.credentials.default                         | No       | Whether to connect using default the GCP SDK default credential discovery. When set to null (the default) or false, will fall back to connecting with No Credentials.Cannot be set together with "gcs.credentials.json" or "gcs.credentials.path".                                |                                                                        |
| gcs.object.content.encoding                     | No       | The GCS object metadata value of Content-Encoding.                                                                                                                                                                                                                                |                                                                        |
| gcs.endpoint                                    | No       | Explicit GCS Endpoint Address, mainly for testing.                                                                                                                                                                                                                                |                                                                        |
| gcs.retry.backoff.initial.delay.ms              | No       | Initial retry delay in milliseconds. The default value is 1000.                                                                                                                                                                                                                   | 1000                                                                   |
| gcs.retry.backoff.max.delay.ms                  | No       | Maximum retry delay in milliseconds. The default value is 32000.                                                                                                                                                                                                                  | 32000                                                                  |
| gcs.retry.backoff.delay.multiplier              | No       | Retry delay multiplier. The default value is 2.0.                                                                                                                                                                                                                                 | 2.0                                                                    |
| gcs.retry.backoff.max.attempts                  | No       | Retry max attempts. The default value is 6.                                                                                                                                                                                                                                       | 6                                                                      |
| gcs.retry.backoff.total.timeout.ms              | No       | Retry total timeout in milliseconds. The default value is 50000.                                                                                                                                                                                                                  | 50000                                                                  |
| gcs.user.agent                                  | No       | A custom user agent used while contacting google.                                                                                                                                                                                                                                 | "Google GCS Sink/3.4.1 (GPN: Aiven;)"                                  |
| file.name.prefix                                | No       | The prefix to be added to the name of each file put on GCS.                                                                                                                                                                                                                       |                                                                        |
| file.name.template                              | No       | The template for file names on GCS. Supports \{\{ variable }} placeholders for substituting variables. Currently supported variables are topic, partition, and start\_offset (the offset of the first record in the file).                                                        | `{{topic}}-{{partition:padding=false}}-{{start_offset:padding=false}}` |
| file.compression.type                           | No       | The compression type used for files put on GCS. The supported values are: 'none', 'gzip', 'snappy', 'zstd'.                                                                                                                                                                       | none                                                                   |
| file.max.records                                | No       | The maximum number of records to put in a single file. Must be a non-negative integer number. 0 is interpreted as "unlimited", which is the default.                                                                                                                              | 0                                                                      |
| file.name.timestamp.timezone                    | No       | Specifies the timezone in which the dates and time for the timestamp variable will be treated. Use standard shot and long names. Default is UTC.                                                                                                                                  | UTC                                                                    |
| file.name.timestamp.source                      | No       | Specifies the the timestamp variable source. Default is wall-clock.                                                                                                                                                                                                               | WALLCLOCK                                                              |
| format.output.type                              | No       | The format type of output contentThe supported values are: 'avro', 'csv', 'json', 'jsonl', 'parquet'.                                                                                                                                                                             | csv                                                                    |
| format.output.fields                            | No       | Fields to put into output files. The supported values are: 'key', 'value', 'offset', 'timestamp', 'headers'.                                                                                                                                                                      | value                                                                  |
| format.output.fields.value.encoding             | No       | The type of encoding for the value field. The supported values are: 'none', 'base64'.                                                                                                                                                                                             | base64                                                                 |
| format.output.envelope                          | No       | Whether to enable envelope for entries with single field.                                                                                                                                                                                                                         | true                                                                   |
| errors.deadletterqueue.topic.name               | No       | The name of the topic to be used as the dead letter queue (DLQ) for messages that result in an error when processed by this sink connector, or its transformations or converters. The topic name is blank by default, which means that no messages are to be recorded in the DLQ. |                                                                        |
| errors.deadletterqueue.topic.replication.factor | No       | Replication factor used to create the dead letter queue topic when it doesn't already exist.                                                                                                                                                                                      | 3                                                                      |
| errors.deadletterqueue.context.headers.enable   | No       | If true, add headers containing error context to the messages written to the dead letter queue. To avoid clashing with headers from the original record, all error context header keys, all error context header keys will start with \_\_connect.errors.                         | false                                                                  |


# Kafka connect google pubsub lite sink
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-google-pubsub-lite-sink/current/kafka-connect-google-pubsub-lite-sink

The official Google Cloud Pub/Sub Lite Sink connector.

The Google Cloud Pub/Sub Group Kafka Connector library provides Google Cloud
Platform (GCP) first-party connectors for Pub/Sub products with
[Kafka Connect](http://kafka.apache.org/documentation.html#connect).
You can use the library to transmit data from
[Apache Kafka](http://kafka.apache.org)
to [Cloud Pub/Sub](https://cloud.google.com/pubsub/docs/) or
[Pub/Sub Lite](https://cloud.google.com/pubsub/lite/docs) and vice versa.

### Prerequisites

You must have a GCP project in order to use Cloud Pub/Sub or Pub/Sub Lite.

Follow these [setup steps](https://cloud.google.com/pubsub/docs/publish-receive-messages-client-library#before-you-begin)
for Pub/Sub before doing the [quickstart](#quickstart).

Follow these [setup steps](https://cloud.google.com/pubsub/lite/docs/publish-receive-messages-console#before-you-begin)
for Pub/Sub Lite before doing the [quickstart](#quickstart).

For general information on how to authenticate with GCP when using the Google
Cloud Pub/Sub Group Kafka Connector library, please visit [Provide credentials
for Application Default Credentials](https://cloud.google.com/docs/authentication/provide-credentials-adc).

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)
2. Create a Pub/Sub topic in your GCP project.
3. Create a JSON file like the following:

```json theme={null}
{
    "name": "pubsub-sink",
    "config": {
        "connector.class": "com.google.pubsublite.kafka.sink.PubSubLiteSinkConnector",
        "pubsublite.project": "${GCP_PROJECT_ID}",
        "pubsublite.topic": "${PUBSUB_TOPIC_NAME}",
        "pubsublite.location": "${PUBSUB_LOCATION}",
        "gcp.credentials.json": "${GCP_CREDENTIALS_JSON}",
        "topics": "${KAFKA_TOPIC_NAME}"
    }
}
```

4. Run the following command to create the connector:

```shell theme={null}
kcctl apply -f <filename>.json
```

### Configuration

The Google Pub/Sub Lite sink connector is configured using the following properties:

| Config                    | Value Range | Default               | Description                                                                                                                                                                                                                                                                 |
| ------------------------- | ----------- | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| pubsublite.topic          | String      | REQUIRED (No default) | The Pub/Sub Lite topic ID, e.g. "foo" for topic "/projects/bar/locations/europe-south7-q/topics/foo".                                                                                                                                                                       |
| pubsublite.project        | String      | REQUIRED (No default) | The project containing the Pub/Sub Lite topic, e.g. "bar" from above.                                                                                                                                                                                                       |
| pubsublite.location       | String      | REQUIRED (No default) | The location of the Pub/Sub Lite topic, e.g. "europe-south7-q" from above.                                                                                                                                                                                                  |
| gcp.credentials.file.path | String      | Optional              | The filepath, which stores GCP credentials. If not defined, the environment variable GOOGLE\_APPLICATION\_CREDENTIALS is used. If specified, use the explicitly handed credentials. Consider using the externalized secrets feature in Kafka Connect for passing the value. |
| gcp.credentials.json      | String      | Optional              | GCP credentials JSON blob. If specified, use the explicitly handed credentials. Consider using the externalized secrets feature in Kafka Connect for passing the value.                                                                                                     |

The full properties are also available from the [Official Google Pub/Sub Lite Kafka Sink Connector documentation](https://github.com/googleapis/java-pubsub-group-kafka-connector?tab=readme-ov-file#sink-connector-1).


# Kafka connect google pubsub lite source
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-google-pubsub-lite-source/current/kafka-connect-google-pubsub-lite-source

The official Google Cloud Pub/Sub Lite Source connector.

The Google Cloud Pub/Sub Group Kafka Connector library provides Google Cloud
Platform (GCP) first-party connectors for Pub/Sub products with
[Kafka Connect](http://kafka.apache.org/documentation.html#connect).
You can use the library to transmit data from
[Apache Kafka](http://kafka.apache.org)
to [Cloud Pub/Sub](https://cloud.google.com/pubsub/docs/) or
[Pub/Sub Lite](https://cloud.google.com/pubsub/lite/docs) and vice versa.

### Prerequisites

You must have a GCP project in order to use Cloud Pub/Sub or Pub/Sub Lite.

Follow these [setup steps](https://cloud.google.com/pubsub/docs/publish-receive-messages-client-library#before-you-begin)
for Pub/Sub before doing the [quickstart](#quickstart).

Follow these [setup steps](https://cloud.google.com/pubsub/lite/docs/publish-receive-messages-console#before-you-begin)
for Pub/Sub Lite before doing the [quickstart](#quickstart).

For general information on how to authenticate with GCP when using the Google
Cloud Pub/Sub Group Kafka Connector library, please visit [Provide credentials
for Application Default Credentials](https://cloud.google.com/docs/authentication/provide-credentials-adc).

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)
2. Create a Pub/Sub topic in your GCP project.
3. Create a JSON file like the following:

```json theme={null}
{
    "name": "pubsub-source",
    "config": {
        "connector.class": "com.google.pubsublite.kafka.source.PubSubLiteSourceConnector",
        "pubsublite.project": "${GCP_PROJECT_ID}",
        "pubsublite.topic": "${PUBSUB_TOPIC_NAME}",
        "pubsublite.subscription": "${PUBSUB_SUBSCRIPTION_NAME}",
        "pubsublite.location": "${PUBSUB_LOCATION}",
        "gcp.credentials.json": "${GCP_CREDENTIALS_JSON}",
        "kafka.topic": "${KAFKA_TOPIC_NAME}"
    }
}
```

4. Run the following command to create the connector:

```shell theme={null}
kcctl apply -f <filename>.json
```

### Configuration

The Google Pub/Sub Lite source connector is configured using the following properties:

| Config                                       | Value Range | Default               | Description                                                                                                                                                                                                                                                                 |
| -------------------------------------------- | ----------- | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| pubsublite.subscription                      | String      | REQUIRED (No default) | The Pub/Sub Lite subscription ID, e.g. "baz" for the subscription "/projects/bar/locations/europe-south7-q/subscriptions/baz".                                                                                                                                              |
| pubsublite.project                           | String      | REQUIRED (No default) | The project containing the Pub/Sub Lite subscription, e.g. "bar" from above.                                                                                                                                                                                                |
| pubsublite.location                          | String      | REQUIRED (No default) | The location of the Pub/Sub Lite subscription, e.g. "europe-south7-q" from above.                                                                                                                                                                                           |
| kafka.topic                                  | String      | REQUIRED (No default) | The Kafka topic which will receive messages from Pub/Sub Lite.                                                                                                                                                                                                              |
| pubsublite.partition\_flow\_control.messages | Long        | Long.MAX\_VALUE       | The maximum number of outstanding messages per Pub/Sub Lite partition.                                                                                                                                                                                                      |
| pubsublite.partition\_flow\_control.bytes    | Long        | 20,000,000            | The maximum number of outstanding bytes per Pub/Sub Lite partition.                                                                                                                                                                                                         |
| gcp.credentials.file.path                    | String      | Optional              | The filepath, which stores GCP credentials. If not defined, the environment variable GOOGLE\_APPLICATION\_CREDENTIALS is used. If specified, use the explicitly handed credentials. Consider using the externalized secrets feature in Kafka Connect for passing the value. |
| gcp.credentials.json                         | String      | Optional              | GCP credentials JSON blob. If specified, use the explicitly handed credentials. Consider using the externalized secrets feature in Kafka Connect for passing the value.                                                                                                     |

The full properties are also available from the [Official Google Pub/Sub Lite Kafka Source Connector documentation](https://github.com/googleapis/java-pubsub-group-kafka-connector?tab=readme-ov-file#source-connector-1).


# Kafka connect google pubsub sink
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-google-pubsub-sink/current/kafka-connect-google-pubsub-sink

The official Google Cloud Pub/Sub Sink connector.

The Google Cloud Pub/Sub Group Kafka Connector library provides Google Cloud
Platform (GCP) first-party connectors for Pub/Sub products with
[Kafka Connect](http://kafka.apache.org/documentation.html#connect).
You can use the library to transmit data from
[Apache Kafka](http://kafka.apache.org)
to [Cloud Pub/Sub](https://cloud.google.com/pubsub/docs/) or
[Pub/Sub Lite](https://cloud.google.com/pubsub/lite/docs) and vice versa.

### Prerequisites

You must have a GCP project in order to use Cloud Pub/Sub or Pub/Sub Lite.

Follow these [setup steps](https://cloud.google.com/pubsub/docs/publish-receive-messages-client-library#before-you-begin)
for Pub/Sub before doing the [quickstart](#quickstart).

Follow these [setup steps](https://cloud.google.com/pubsub/lite/docs/publish-receive-messages-console#before-you-begin)
for Pub/Sub Lite before doing the [quickstart](#quickstart).

For general information on how to authenticate with GCP when using the Google
Cloud Pub/Sub Group Kafka Connector library, please visit [Provide credentials
for Application Default Credentials](https://cloud.google.com/docs/authentication/provide-credentials-adc).

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)
2. Create a Pub/Sub topic in your GCP project.
3. Create a JSON file like the following:

```json theme={null}
{
    "name": "pubsub-sink",
    "config": {
        "connector.class": "com.google.pubsub.kafka.sink.CloudPubSubSinkConnector",
        "cps.project": "${GCP_PROJECT_ID}",
        "cps.topic": "${PUBSUB_TOPIC_NAME}",
        "gcp.credentials.json": "${GCP_CREDENTIALS_JSON}",
        "topics": "${KAFKA_TOPIC_NAME}"
    }
}
```

4. Run the following command to create the connector:

```shell theme={null}
kcctl apply -f <filename>.json
```

### Configuration

The Google Pub/Sub sink connector is configured using the following properties:

| Config                     | Type                          | Default                     | Description                                                                                                                                                                                                                                                                                                                         |
| -------------------------- | ----------------------------- | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| cps.topic                  | String                        | REQUIRED (No default)       | The Pub/Sub topic ID, e.g. "foo" for topic "/projects/bar/topics/foo".                                                                                                                                                                                                                                                              |
| cps.project                | String                        | REQUIRED (No default)       | The project containing the Pub/Sub topic, e.g. "bar" from above.                                                                                                                                                                                                                                                                    |
| cps.endpoint               | String                        | "pubsub.googleapis.com:443" | The Pub/Sub endpoint to use.                                                                                                                                                                                                                                                                                                        |
| maxBufferSize              | Integer                       | 100                         | The maximum number of messages that can be received for the messages on a topic partition before publishing them to Pub/Sub.                                                                                                                                                                                                        |
| maxBufferBytes             | Long                          | 10,000,000                  | The maximum number of bytes that can be received for the messages on a topic partition before publishing them to Pub/Sub.                                                                                                                                                                                                           |
| maxOutstandingRequestBytes | Long                          | Long.MAX\_VALUE             | The maximum number of total bytes that can be outstanding (including incomplete and pending batches) before the publisher will block further publishing.                                                                                                                                                                            |
| maxOutstandingMessages     | Long                          | Long.MAX\_VALUE             | The maximum number of messages that can be outstanding (including incomplete and pending batches) before the publisher will block further publishing.                                                                                                                                                                               |
| maxDelayThresholdMs        | Integer                       | 100                         | The maximum amount of time to wait to reach maxBufferSize or maxBufferBytes before publishing outstanding messages to Pub/Sub.                                                                                                                                                                                                      |
| maxRequestTimeoutMs        | Integer                       | 10,000                      | The timeout for individual publish requests to Pub/Sub.                                                                                                                                                                                                                                                                             |
| maxTotalTimeoutMs          | Integer                       | 60,000                      | The total timeout for a call to publish (including retries) to Pub/Sub.                                                                                                                                                                                                                                                             |
| maxShutdownTimeoutMs       | Integer                       | 60,000                      | The maximum amount of time to wait for a publisher to shutdown when stopping task in Kafka Connect.                                                                                                                                                                                                                                 |
| gcp.credentials.file.path  | String                        | Optional                    | The filepath, which stores GCP credentials. If not defined, GOOGLE\_APPLICATION\_CREDENTIALS env is used.                                                                                                                                                                                                                           |
| gcp.credentials.json       | String                        | Optional                    | GCP credentials JSON blob. If specified, use the explicitly handed credentials. Consider using the externalized secrets feature in Kafka Connect for passing the value.                                                                                                                                                             |
| metadata.publish           | Boolean                       | false                       | When true, include the Kafka topic, partition, offset, and timestamp as message attributes when a message is published to Pub/Sub.                                                                                                                                                                                                  |
| headers.publish            | Boolean                       | false                       | When true, include any headers as attributes when a message is published to Pub/Sub.                                                                                                                                                                                                                                                |
| orderingKeySource          | String (none, key, partition) | none                        | When set to "none", do not set the ordering key. When set to "key", uses a message's key as the ordering key. If set to "partition", converts the partition number to a String and uses that as the ordering key. Note that using "partition" should only be used for low-throughput topics or topics with thousands of partitions. |
| messageBodyName            | String                        | "cps\_message\_body"        | When using a struct or map value schema, this field or key name indicates that the corresponding value will go into the Pub/Sub message body.                                                                                                                                                                                       |
| enableCompression          | Boolean                       | false                       | When true, enable publish-side compression in order to save on networking costs between Kafka Connect and Cloud Pub/Sub.                                                                                                                                                                                                            |
| compressionBytesThreshold  | Long                          | 240                         | When enableCompression is true, the minimum size of publish request (in bytes) to compress.                                                                                                                                                                                                                                         |

The full properties are also available from the [official Google Pub/Sub Kafka Sink Connector documentation](https://github.com/googleapis/java-pubsub-group-kafka-connector?tab=readme-ov-file#sink-connector).


# Kafka connect google pubsub source
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-google-pubsub-source/current/kafka-connect-google-pubsub-source

The official Google Cloud Pub/Sub Source connector.

The Google Cloud Pub/Sub Group Kafka Connector library provides Google Cloud
Platform (GCP) first-party connectors for Pub/Sub products with
[Kafka Connect](http://kafka.apache.org/documentation.html#connect).
You can use the library to transmit data from
[Apache Kafka](http://kafka.apache.org)
to [Cloud Pub/Sub](https://cloud.google.com/pubsub/docs/) or
[Pub/Sub Lite](https://cloud.google.com/pubsub/lite/docs) and vice versa.

### Prerequisites

You must have a GCP project in order to use Cloud Pub/Sub or Pub/Sub Lite.

Follow these [setup steps](https://cloud.google.com/pubsub/docs/publish-receive-messages-client-library#before-you-begin)
for Pub/Sub before doing the [quickstart](#quickstart).

Follow these [setup steps](https://cloud.google.com/pubsub/lite/docs/publish-receive-messages-console#before-you-begin)
for Pub/Sub Lite before doing the [quickstart](#quickstart).

For general information on how to authenticate with GCP when using the Google
Cloud Pub/Sub Group Kafka Connector library, please visit [Provide credentials
for Application Default Credentials](https://cloud.google.com/docs/authentication/provide-credentials-adc).

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)
2. Create a Pub/Sub topic and a Subscription on this topic in your GCP project.
3. Create a JSON file like the following:

```json theme={null}
{
    "name": "pubsub-source",
    "config": {
        "connector.class": "com.google.pubsub.kafka.source.CloudPubSubSourceConnector",
        "cps.project": "${GCP_PROJECT_ID}",
        "cps.topic": "${PUBSUB_TOPIC_NAME}",
        "cps.subscription": "${PUBSUB_SUBSCRIPTION_NAME}",
        "gcp.credentials.json": "${GCP_CREDENTIALS_JSON}",
        "kafka.topic": "${KAFKA_TOPIC_NAME}"
    }
}
```

4. Run the following command to create the connector:

```shell theme={null}
kcctl apply -f <filename>.json
```

### Configuration

The Google Pub/Sub sink connector is configured using the following properties:

| Config                                 | Value Range                                                             | Default                        | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| -------------------------------------- | ----------------------------------------------------------------------- | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| cps.subscription                       | String                                                                  | REQUIRED (No default)          | The Pub/Sub subscription ID, e.g. "baz" for subscription "/projects/bar/subscriptions/baz".                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| cps.project                            | String                                                                  | REQUIRED (No default)          | The project containing the Pub/Sub subscription, e.g. "bar" from above.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| cps.endpoint                           | String                                                                  | "pubsub.googleapis.com:443"    | The Pub/Sub endpoint to use.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| kafka.topic                            | String                                                                  | REQUIRED (No default)          | The Kafka topic which will receive messages from the Pub/Sub subscription.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| cps.maxBatchSize                       | Integer                                                                 | 100                            | The maximum number of messages per batch in a pull request to Pub/Sub.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| cps.makeOrderingKeyAttribute           | Boolean                                                                 | false                          | When true, copy the ordering key to the set of attributes set in the Kafka message.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| kafka.key.attribute                    | String                                                                  | null                           | The Pub/Sub message attribute to use as a key for messages published to Kafka. If set to "orderingKey", use the message's ordering key.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| kafka.partition.count                  | Integer                                                                 | 1                              | The number of Kafka partitions for the Kafka topic in which messages will be published to. NOTE: this parameter is ignored if partition scheme is "kafka\_partitioner".                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| kafka.partition.scheme                 | round\_robin, hash\_key, hash\_value, kafka\_partitioner, ordering\_key | round\_robin                   | The scheme for assigning a message to a partition in Kafka. The scheme "round\_robin" assigns partitions in a round robin fashion, while the schemes "hash\_key" and "hash\_value" find the partition by hashing the message key and message value respectively. "kafka\_partitioner" scheme delegates partitioning logic to Kafka producer, which by default detects number of partitions automatically and performs either murmur hash based partition mapping or round robin depending on whether message key is provided or not. "ordering\_key" uses the hash code of a message's ordering key. If no ordering key is present, uses "round\_robin". |
| gcp.credentials.file.path              | String                                                                  | Optional                       | The filepath, which stores a GCP Service Account credentials. If not defined, GOOGLE\_APPLICATION\_CREDENTIALS env is used. If specified, use the explicitly handed credentials. Consider using the externalized secrets feature in Kafka Connect for passing the value.                                                                                                                                                                                                                                                                                                                                                                                 |
| gcp.credentials.json                   | String                                                                  | Optional                       | GCP Service Account credentials JSON blob. If specified, use the explicitly handed credentials. Consider using the externalized secrets feature in Kafka Connect for passing the value.                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| kafka.record.headers                   | Boolean                                                                 | false                          | Use Kafka record headers to store Pub/Sub message attributes.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| cps.streamingPull.enabled              | Boolean                                                                 | false                          | Whether to use streaming pull for the connector to connect to Pub/Sub. If provided, cps.maxBatchSize is ignored.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| cps.streamingPull.flowControlMessages  | Long                                                                    | 1,000                          | The maximum number of outstanding messages per task when using streaming pull.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| cps.streamingPull.flowControlBytes     | Long                                                                    | 100L \* 1024 \* 1024 (100 MiB) | The maximum number of outstanding message bytes per task when using streaming pull.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| cps.streamingPull.parallelStreams      | Integer                                                                 | 1                              | The number of streams to pull messages from the subscription when using streaming pull.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| cps.streamingPull.maxAckExtensionMs    | Long                                                                    | 0                              | The maximum number of milliseconds the subscribe deadline will be extended to in milliseconds when using streaming pull. A value of 0 implies the java-pubsub library default value.                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| cps.streamingPull.maxMsPerAckExtension | Long                                                                    | 0                              | The maximum number of milliseconds to extend the subscribe deadline for at a time when using streaming pull. A value of 0 implies the java-pubsub library default value.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |

The full properties are also available from the [offical Google Pub/Sub Kafka Source Connector documentation](https://github.com/googleapis/java-pubsub-group-kafka-connector?tab=readme-ov-file#source-connector).


# Kafka connect iceberg sink
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-iceberg/current/kafka-connect-iceberg-sink

The official Apache Iceberg Kafka Connect Sink connector.

The Apache Iceberg Kafka Connect Sink connector is a Kafka Connect connector that writes data from Kafka topics to Apache Iceberg tables.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

### Prerequisites

* Setup the [Iceberg Catalog](https://iceberg.apache.org/concepts/catalog/)
* Create the Iceberg connector control topic, which cannot be used by other connectors.

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)

2. Set up the Iceberg Catalog, we can use the below yaml file to create a local iceberg catalog in k8s:
   ```yaml theme={null}
   apiVersion: v1
   data:
     spark-defaults.conf: |
       #
       # Licensed to the Apache Software Foundation (ASF) under one or more
       # contributor license agreements.  See the NOTICE file distributed with
       # this work for additional information regarding copyright ownership.
       # The ASF licenses this file to You under the Apache License, Version 2.0
       # (the "License"); you may not use this file except in compliance with
       # the License.  You may obtain a copy of the License at
       #
       #    http://www.apache.org/licenses/LICENSE-2.0
       #
       # Unless required by applicable law or agreed to in writing, software
       # distributed under the License is distributed on an "AS IS" BASIS,
       # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
       # See the License for the specific language governing permissions and
       # limitations under the License.
       #

       # Default system properties included when running spark-submit.
       # This is useful for setting default environmental settings.

       # Example:
       spark.sql.extensions                   org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
       spark.sql.catalog.demo                 org.apache.iceberg.spark.SparkCatalog
       spark.sql.catalog.demo.type            rest
       spark.sql.catalog.demo.uri             http://iceberg-rest.default.svc.cluster.local:8181
       spark.sql.catalog.demo.io-impl         org.apache.iceberg.aws.s3.S3FileIO
       spark.sql.catalog.demo.warehouse       s3://warehouse/
       spark.sql.catalog.demo.s3.endpoint     http://minio.default.svc.cluster.local:9000
       spark.sql.defaultCatalog               demo
       spark.eventLog.enabled                 true
       spark.eventLog.dir                     /home/iceberg/spark-events
       spark.history.fs.logDirectory          /home/iceberg/spark-events
       spark.sql.catalogImplementation        in-memory
   kind: ConfigMap
   metadata:
     name: spark-config
     namespace: default
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: spark-iceberg
     namespace: default
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: spark-iceberg
     template:
       metadata:
         labels:
           app: spark-iceberg
       spec:
         containers:
         - name: spark-iceberg
           image: tabulario/spark-iceberg
           ports:
             - name: one
               containerPort: 8888
             - name: two
               containerPort: 8080
             - name: three
               containerPort: 10000
             - name: four
               containerPort: 10001
           env:
             - name: AWS_ACCESS_KEY_ID
               value: admin
             - name: AWS_SECRET_ACCESS_KEY
               value: password
             - name: AWS_REGION
               value: us-east-1
           volumeMounts:
             - name: spark-config
               mountPath: /opt/spark/conf/spark-defaults.conf
               subPath: spark-defaults.conf
         volumes:
           - name: spark-config
             configMap:
               name: spark-config
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: spark-iceberg
     namespace: default
   spec:
     selector:
       app: spark-iceberg
     ports:
       - protocol: TCP
         name: one
         port: 8888
         targetPort: 8888
       - protocol: TCP
         name: two
         port: 8080
         targetPort: 8080
       - protocol: TCP
         name: three
         port: 10000
         targetPort: 10000
       - protocol: TCP
         port: 10001
         name: four
         targetPort: 10001
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: iceberg-rest
     namespace: default
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: iceberg-rest
     template:
       metadata:
         labels:
           app: iceberg-rest
       spec:
         containers:
         - name: iceberg-rest
           image: tabulario/iceberg-rest
           ports:
             - name: one
               containerPort: 8181
           env:
             - name: AWS_ACCESS_KEY_ID
               value: admin
             - name: AWS_SECRET_ACCESS_KEY
               value: password
             - name: AWS_REGION
               value: us-east-1
             - name: CATALOG_WAREHOUSE
               value: s3://warehouse/
             - name: CATALOG_IO__IMPL
               value: org.apache.iceberg.aws.s3.S3FileIO
             - name: CATALOG_S3_ENDPOINT
               value: http://minio.default.svc.cluster.local:9000
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: iceberg-rest
     namespace: default
   spec:
     selector:
       app: iceberg-rest
     ports:
       - protocol: TCP
         name: one
         port: 8181
         targetPort: 8181
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: minio
     namespace: default
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: minio
     template:
       metadata:
         labels:
           app: minio
       spec:
         hostname: warehouse
         subdomain: minio
         containers:
         - name: minio
           image: minio/minio
           args:
             - server
             - /data
             - --console-address
             - ":9001"
           ports:
             - name: one
               containerPort: 9000
             - name: two
               containerPort: 9001
           env:
             - name: MINIO_ROOT_USER
               value: admin
             - name: MINIO_ROOT_PASSWORD
               value: password
             - name: MINIO_DOMAIN
               value: minio.default.svc.cluster.local
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: minio
     namespace: default
   spec:
     selector:
       app: minio
     ports:
       - protocol: TCP
         name: one
         port: 9000
         targetPort: 9000
       - protocol: TCP
         name: two
         port: 9001
         targetPort: 9001
   ```

3. Initialize the Iceberg table:

   ```bash theme={null}
   kubectl apply -f iceberg-spark.yaml
   kubectl wait -l app=spark-iceberg --for=condition=Ready pod --timeout=5m
   kubectl wait -l app=iceberg-rest --for=condition=Ready pod --timeout=5m
   kubectl wait -l app=minio --for=condition=Ready pod --timeout=5m

   sleep 30

   # initialize the bucket
   minio_pod_name=$(kubectl get pods -l app=minio -o=jsonpath='{.items[0].metadata.name}')
   kubectl exec $minio_pod_name -- /usr/bin/mc config host add minio http://minio.default.svc.cluster.local:9000 admin password
   kubectl exec $minio_pod_name -- /usr/bin/mc rm -r --force minio/warehouse || true
   kubectl exec $minio_pod_name -- /usr/bin/mc mb minio/warehouse
   kubectl exec $minio_pod_name -- /usr/bin/mc policy set public minio/warehouse
   ```

4. Create a JSON file like the following:

   ```json theme={null}
   {
       "name": "iceberg-sink",
       "config": {
           "connector.class": "io.tabular.iceberg.connect.IcebergSinkConnector",
           "topics": "kafka-iceberg-input",
           "iceberg.tables": "sink.kafka",
           "iceberg.catalog": "demo",
           "iceberg.catalog.type": "rest",
           "iceberg.catalog.uri": "http://iceberg-rest.default.svc.cluster.local:8181",
           "iceberg.catalog.client.region": "us-east-1",
           "iceberg.catalog.io-impl": "org.apache.iceberg.aws.s3.S3FileIO",
           "iceberg.catalog.warehouse": "s3://warehouse",
           "iceberg.catalog.s3.endpoint": "http://minio.default.svc.cluster.local:9000",
           "iceberg.catalog.s3.path-style-access": "true",
           "iceberg.catalog.s3.access-key-id": "admin",
           "iceberg.catalog.s3.secret-access-key": "password",
           "iceberg.tables.auto-create-enabled": "true",
           "iceberg.tables.evolve-schema-enabled": "true",
           "iceberg.control.commit.interval-ms": "1000",
           "key.converter": "org.apache.kafka.connect.storage.StringConverter",
           "value.converter": "org.apache.kafka.connect.json.JsonConverter",
           "value.converter.schemas.enable": "false",
           "tasks.max": "1"
       }
   }
   ```

5. Run the following command to create the connector:

   ```bash theme={null}
   kcctl apply -f <filename>.json
   ```

### Quick Start 2 - Write to AWS S3 Table

This is a real example which sink data to AWS S3 Iceberg table, below are the steps.

1. Create an AWS S3 **table bucket**, this is a **new bucket type**, A regular S3 bucket won’t work for S3 Tables, here we use `kc-test-iceberg-table` as an example.

2. Create a new database in the table bucket, you can go to the **AWS Lake Formation** console, and then go to the **Data Catalog**/**Databases** section to create it, here we use `my_s3_namespace` as an example.

3. Create an AWS IAM user and attach the following policy to it, replace `eu-north-1`, `account-id`, `kc-test-iceberg-table` and `my_s3_namespace` with your real `region`, `account-id`, `table-bucket` and `database` respectively.:
   ```json theme={null}
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "VisualEditor0",
               "Effect": "Allow",
               "Action": "lakeformation:GetDataAccess",
               "Resource": "*"
           },
           {
               "Sid": "GlueAndCatalogForS3Tables",
               "Effect": "Allow",
               "Action": [
                   "glue:GetCatalog",
                   "glue:GetDatabase",
                   "glue:GetDatabases",
                   "glue:GetTable",
                   "glue:GetTables",
                   "glue:CreateTable",
                   "glue:UpdateTable"
               ],
               "Resource": [
                   "arn:aws:glue:eu-north-1:{account_id}:catalog",
                   "arn:aws:glue:eu-north-1:{account_id}:catalog/s3tablescatalog",
                   "arn:aws:glue:eu-north-1:{account_id}:catalog/s3tablescatalog/kc-test-iceberg-table",
                   "arn:aws:glue:eu-north-1:{account_id}:database/s3tablescatalog/kc-test-iceberg-table/my_s3_namespace",
                   "arn:aws:glue:eu-north-1:{account_id}:table/s3tablescatalog/kc-test-iceberg-table/my_s3_namespace/*"
               ]
           }
       ]
   }
   ```

4. Create an access key for the IAM user, you will need the `Access Key ID` and `Secret Access Key` later.

5. In the **Lake Formation** console, grant the user database permissions (**Create table**, **Describe**) on \[account-id]:s3tablescatalog/kc-test-iceberg-table/my\_s3\_namespace, and table permissions (**Super** on **ALL\_TABLES** or at least the table you’ll write).

6. Create a JSON file like the following, replace the fields such as `region`, `account-id`, `table_bucket`, `access_key_id` and `secret_access_key` with your real values:
   ```JSON theme={null}
   {
       "name": "test-ice",
       "config": {
           "name": "test-ice",
           "connector.class": "io.tabular.iceberg.connect.IcebergSinkConnector",
           "topics": "events",
           "tasks.max": "1",
           "iceberg.tables.auto-create-enabled": "true",
           "iceberg.tables": "my_s3_namespace.events",
           "iceberg.tables.evolve-schema-enabled": "true",
           "header.converter": "org.apache.kafka.connect.storage.SimpleHeaderConverter",
           "value.converter": "org.apache.kafka.connect.json.JsonConverter",
           "value.converter.schemas.enable": "false",
           "key.converter": "org.apache.kafka.connect.storage.StringConverter",
           "key.converter.schemas.enable": "false",
           "iceberg.catalog": "iceberg",
           "iceberg.catalog.type": "rest",
           "iceberg.catalog.uri": "https://glue.{region}.amazonaws.com/iceberg",
           "iceberg.catalog.warehouse": "{account-id}:s3tablescatalog/{table_bucket}",
           "iceberg.catalog.rest.access-key-id": "{access_key_id}",
           "iceberg.catalog.rest.secret-access-key": "{secret_access_key}",
           "iceberg.catalog.rest.sigv4-enabled": "true",
           "iceberg.catalog.rest.signing-name": "glue",
           "iceberg.catalog.rest.signing-region": "{region}",
           "iceberg.control.topic": "control-iceberg",
           "sn.connector.image.opts": "-Daws.region={region}"
       }
   }
   ```

7. Run the following command to create the connector:

   ```bash theme={null}
   kcctl apply -f <filename>.json
   ```

8. Produce some test data to the `events` topic, you can use the following command to produce some test data:

   ```bash theme={null}
   echo '{"id": 1, "name": "Jack", "message": "hello iceberg", "email": "Jack@test.com"}' | ~/kafka/kafka_2.13-3.1.0/bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic events
   ```

9. Wait until the data is written to the Iceberg table, you can check the logs of the connector to see something like below:

   ```
   2025-08-25T08:05:27,622+0000 [iceberg-coord] INFO  io.tabular.iceberg.connect.channel.Coordinator - Commit fa40ffe5-9424-4994-8439-cebe46919ff4 complete, committed to 1 table(s), vtts null
   ```

10. Check the Iceberg table in the AWS Athena console, you should see the `events` table in the `my_s3_namespace` database, and you can run a query like below to see the data:

    ```sql theme={null}
    SELECT * FROM "my_s3_namespace"."events" limit 10;
    ```

### Limitations

* Each Iceberg sink connector must have its own control topic.

### Configuration

The following *Required* properties are used to configure the connector.

| Parameter               | Description                                                                                                                                       |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `topics`                | Comma-separated list of the Kafka topics you want to replicate. (You can define either the `topics` or the `topics.regex` setting, but not both.) |
| `topics.regex`          | Java regular expression of topics to replicate. (You can define either the `topics` or the `topics.regex` setting, but not both.)                 |
| `iceberg.control.topic` | The name of the control topic. It cannot be used by other Iceberg connectors.                                                                     |
| `iceberg.catalog.type`  | The type of Iceberg catalog. Allowed options are: `REST`, `HIVE`, `HADOOP`.                                                                       |
| `iceberg.tables`        | Comma-separated list of Iceberg table names, which are specified using the format `{namespace}.{table}`.                                          |

The following *Advanced* properties are used to configure the connector.

| Parameter                           | Description                                                                                                                                      |
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `iceberg.control.commit.timeout-ms` | Commit timeout interval in ms. The default is 30000 (30 sec).                                                                                    |
| `iceberg.tables.route-field`        | For multi-table fan-out, the name of the field used to route records to tables. Required when `iceberg.tables.dynamic-enabled` is set to `true`. |
| `iceberg.tables.cdc-field`          | Name of the field containing the CDC operation, `I`, `U`, or `D`, default is none                                                                |

For more information about the properties, see the [official documentation](https://github.com/tabular-io/iceberg-kafka-connect/blob/v0.6.19/README.md).


# Aiven JDBC Kafka Connect Sink
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-jdbc-sink/current/kafka-connect-jdbc-sink

The Aiven JDBC Kafka Connect Sink connector.

The Aiven JDBC sink connector reads data from Kafka topics and writes data to any JDBC-compliant database.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

### Prerequisites

* Valid credentials for the target database.
* The `connection.url` for your database.

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)
2. Create a MySQL database in your Kubernetes cluster:

```shell theme={null}
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm install mysql bitnami/mysql \
  --set auth.rootPassword=secretpassword \
  --set auth.database=test
kubectl wait -l app.kubernetes.io/instance=mysql --for=condition=Ready pod --timeout=5m
```

3. Create a JSON file like the following:

```json theme={null}
{
    "name": "jdbc-sink",
    "config": {
        "connector.class": "io.aiven.connect.jdbc.JdbcSinkConnector",
        "connection.url": "jdbc:mysql://mysql.default.svc.cluster.local:3306/test",
        "connection.user": "root",
        "connection.password": "secretpassword",
        "insert.mode": "insert",
        "table.name.format": "test_table",
        "topics": "kafka-jdbc-input",
        "value.converter": "org.apache.kafka.connect.json.JsonConverter",
        "value.converter.schemas.enable": false,
        "tasks.max": "1"
    }
}
```

4. Run the following command to create the connector:

```shell theme={null}
kcctl apply -f <filename>.json
```

### Configuration

Configure the Aiven JDBC sink connector with the following properties:

| Property                   | Required | Default    | Description                                                                                                                                                                                                                                                                                                                                       |
| -------------------------- | -------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `connection.url`           | true     |            | JDBC connection URL.                                                                                                                                                                                                                                                                                                                              |
| `connection.user`          | true     | null       | JDBC connection user.                                                                                                                                                                                                                                                                                                                             |
| `connection.password`      | true     | null       | JDBC connection password.                                                                                                                                                                                                                                                                                                                         |
| `db.timezone`              | false    | UTC        | Name of the JDBC timezone that should be used in the connector when querying with time-based criteria. Defaults to UTC.                                                                                                                                                                                                                           |
| `dialect.name`             | false    | ""         | The name of the database dialect that should be used for this connector. By default this is empty, and the connector automatically determines the dialect based upon the JDBC connection URL. Use this if you want to override that behavior and use a specific dialect. All properly-packaged dialects in the JDBC connector plugin can be used. |
| `sql.quote.identifiers`    | false    | true       | Whether to delimit (in most databases, quote with double quotes) identifiers (e.g., table names and column names) in SQL statements.                                                                                                                                                                                                              |
| `insert.mode`              | true     | insert     | The insertion mode to use. Supported modes are: `insert`, `multi`, `upsert`, `update`.                                                                                                                                                                                                                                                            |
| `batch.size`               | false    | 3000       | Specifies how many records to attempt to batch together for insertion into the destination table, when possible.                                                                                                                                                                                                                                  |
| `delete.enabled`           | false    | false      | Enable deletion of rows based on tombstone messages.                                                                                                                                                                                                                                                                                              |
| `table.name.format`        | false    | `${topic}` | A format string for the destination table name, which may contain `${topic}` as a placeholder for the originating topic name.                                                                                                                                                                                                                     |
| `table.name.normalize`     | false    | false      | Whether or not to normalize destination table names for topics.                                                                                                                                                                                                                                                                                   |
| `topics.to.tables.mapping` | false    | null       | Kafka topics to database tables mapping.                                                                                                                                                                                                                                                                                                          |
| `pk.mode`                  | true     | none       | The primary key mode. Supported modes are: `none`, `kafka`, `record_key`, `record_value`.                                                                                                                                                                                                                                                         |
| `pk.fields`                | false    | ""         | List of comma-separated primary key field names.                                                                                                                                                                                                                                                                                                  |
| `fields.whitelist`         | false    | ""         | List of comma-separated record value field names.                                                                                                                                                                                                                                                                                                 |
| `auto.create`              | false    | false      | Whether to automatically create the destination table based on record schema if it is found to be missing.                                                                                                                                                                                                                                        |
| `auto.evolve`              | false    | false      | Whether to automatically add columns in the table schema when found to be missing relative to the record schema.                                                                                                                                                                                                                                  |
| `max.retries`              | false    | 10         | The maximum number of times to retry on errors before failing the task.                                                                                                                                                                                                                                                                           |
| `retry.backoff.ms`         | false    | 3000       | The time in milliseconds to wait following an error before a retry attempt is made.                                                                                                                                                                                                                                                               |

For full details, see the [Aiven JDBC sink connector configs](https://github.com/Aiven-Open/jdbc-connector-for-apache-kafka/blob/master/docs/sink-connector-config-options.rst)


# Aiven JDBC Kafka Connect Source
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-jdbc-source/current/kafka-connect-jdbc-source

The Aiven JDBC Kafka Connect Source connector.

The Aiven JDBC source connector reads data from any JDBC-compliant database and writes data to Kafka topics.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

### Prerequisites

* Valid credentials for the source database.
* The `connection.url` for your database.

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)
2. Create a MySQL database in your Kubernetes cluster:

```shell theme={null}
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm install mysql bitnami/mysql \
  --set auth.rootPassword=secretpassword \
  --set auth.database=test
kubectl wait -l app.kubernetes.io/instance=mysql --for=condition=Ready pod --timeout=5m
# Create a table and insert some data
kubectl run -i --rm --tty mysql-client --image=mysql:8.0 --restart=Never -- mysql -h mysql.default.svc.cluster.local -uroot -psecretpassword -Dtest -e "CREATE TABLE test_table (id INT PRIMARY KEY, name VARCHAR(255)); INSERT INTO test_table (id, name) VALUES (1, 'test-user');"
```

3. Create a JSON file like the following:

```json theme={null}
{
    "name": "jdbc-source",
    "config": {
        "connector.class": "io.aiven.connect.jdbc.JdbcSourceConnector",
        "connection.url": "jdbc:mysql://mysql.default.svc.cluster.local:3306/test",
        "connection.user": "root",
        "connection.password": "secretpassword",
        "mode": "incrementing",
        "table.whitelist": "test_table",
        "incrementing.column.name": "id",
        "topic.prefix": "jdbc_",
        "value.converter": "org.apache.kafka.connect.json.JsonConverter",
        "value.converter.schemas.enable": false,
        "tasks.max": "1"
    }
}
```

4. Run the following command to create the connector:

```shell theme={null}
kcctl apply -f <filename>.json
```

### Configuration

Configure the Aiven JDBC source connector with the following properties:

| Property                      | Required | Default | Description                                                                                                                                                 |
| ----------------------------- | -------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `connection.url`              | true     |         | JDBC connection URL.                                                                                                                                        |
| `connection.user`             | true     | null    | JDBC connection user.                                                                                                                                       |
| `connection.password`         | true     | null    | JDBC connection password.                                                                                                                                   |
| `connection.attempts`         | false    | 3       | Maximum number of attempts to retrieve a valid JDBC connection.                                                                                             |
| `connection.backoff.ms`       | false    | 10000   | Backoff time in milliseconds between connection attempts.                                                                                                   |
| `table.whitelist`             | false    | ""      | List of tables to include in copying.                                                                                                                       |
| `table.blacklist`             | false    | ""      | List of tables to exclude from copying.                                                                                                                     |
| `catalog.pattern`             | false    | null    | Catalog pattern to fetch table metadata from the database.                                                                                                  |
| `schema.pattern`              | false    | null    | Schema pattern to fetch table metadata from the database.                                                                                                   |
| `numeric.precision.mapping`   | false    | false   | Whether or not to attempt mapping NUMERIC values by precision to integral types. (deprecated)                                                               |
| `numeric.mapping`             | false    | null    | Map NUMERIC values by precision and optionally scale to integral or decimal types.                                                                          |
| `table.names.qualify`         | false    | true    | Whether to use fully-qualified table names when querying the database.                                                                                      |
| `db.timezone`                 | false    | UTC     | Name of the JDBC timezone that should be used in the connector when querying with time-based criteria.                                                      |
| `dialect.name`                | false    | ""      | The name of the database dialect that should be used for this connector.                                                                                    |
| `sql.quote.identifiers`       | false    | true    | Whether to delimit identifiers in SQL statements.                                                                                                           |
| `mode`                        | true     |         | The mode for updating a table each time it is polled. Valid Values: \[bulk, timestamp, incrementing, timestamp+incrementing]                                |
| `incrementing.column.name`    | false    | ""      | The name of the strictly incrementing column to use to detect new rows.                                                                                     |
| `timestamp.column.name`       | false    | ""      | Comma separated list of one or more timestamp columns to detect new or modified rows.                                                                       |
| `validate.non.null`           | false    | true    | By default, the JDBC connector will validate that all incrementing and timestamp tables have NOT NULL set for the columns being used as their ID/timestamp. |
| `query`                       | false    | ""      | If specified, the query to perform to select new or updated rows.                                                                                           |
| `timestamp.initial.ms`        | false    | 0       | The initial value of timestamp when selecting records.                                                                                                      |
| `incrementing.initial`        | false    | -1      | For the incrementing column, consider only the rows that have the value greater than this.                                                                  |
| `table.types`                 | false    | TABLE   | A comma-separated list of table types to extract.                                                                                                           |
| `poll.interval.ms`            | true     | 5000    | Frequency in ms to poll for new data in each table.                                                                                                         |
| `batch.max.rows`              | false    | 100     | Maximum number of rows to include in a single batch when polling for new data.                                                                              |
| `table.poll.interval.ms`      | false    | 60000   | Frequency in ms to poll for new or removed tables.                                                                                                          |
| `topic.prefix`                | true     |         | Prefix to prepend to table names to generate the name of the Kafka topic.                                                                                   |
| `timestamp.delay.interval.ms` | true     | 0       | How long to wait after a row with certain timestamp appears before we include it in the result.                                                             |

For full details, see the [Aiven JDBC source connector configs](https://github.com/Aiven-Open/jdbc-connector-for-apache-kafka/blob/master/docs/source-connector-config-options.rst)


# Kafka connect jr source
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-jr-source/current/kafka-connect-jr-source

JR Source Connector for Apache Kafka Connect. JR (jrnd.io) is a CLI program that helps you to stream quality random data for your applications.

`kafka-connect-jr-source` is a Kafka Connect connector for generating mock data for testing and is not suitable for production scenarios. It is available in the StreamNative Cloud.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)
2. Create a json file like below:
   ```json theme={null}
   {
       "name": "jr-quickstart",
       "config": {
           "connector.class": "io.jrnd.kafka.connect.connector.JRSourceConnector",
           "template": "net_device",
           "topic": "net_device",
           "frequency": 5000,
           "objects": 5,
           "tasks.max": 1
       }
   }
   ```
3. Run the following command to create the connector:
   ```
   kcctl apply -f <filename>.json
   ```

### Configuration

The JR Source Connector can be configured using the following properties:

| Parameter                             | Description                                                                                                                                                                                                                                                         | Default                                          |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ |
| `template`                            | A valid JR existing template name. Skipped when `embedded_template` is set.                                                                                                                                                                                         | net\_device                                      |
| `topic`                               | Destination topic on Kafka                                                                                                                                                                                                                                          |                                                  |
| `frequency`                           | Repeat the creation of a random object every 'frequency' milliseconds.                                                                                                                                                                                              | 5000                                             |
| `duration`                            | Set a time bound to the entire object creation. The duration is calculated starting from the first run and is expressed in milliseconds. At least one run will always been scheduled, regardless of the value for 'duration'. If not set creation will run forever. | -1                                               |
| `objects`                             | Number of objects to create at every run.                                                                                                                                                                                                                           | 1                                                |
| `key_field_name`                      | Name for key field, for example 'ID'. This is an *OPTIONAL* config, if not set, objects will be created without a key. Skipped when `key_embedded_template` is set. Value for key will be calculated using JR function `key`.                                       |                                                  |
| `key_value_interval_max`              | Maximum interval value for key value, for example 150 (0 to key\_value\_interval\_max). Skipped when `key_embedded_template` is set.                                                                                                                                | 100                                              |
| `value.converter`                     | One between `org.apache.kafka.connect.storage.StringConverter`, `io.confluent.connect.avro.AvroConverter`, `io.confluent.connect.json.JsonSchemaConverter` or `io.confluent.connect.protobuf.ProtobufConverter`                                                     | org.apache.kafka.connect.storage.StringConverter |
| `value.converter.schema.registry.url` | Only if `value.converter` is set to `io.confluent.connect.avro.AvroConverter`, `io.confluent.connect.json.JsonSchemaConverter` or `io.confluent.connect.protobuf.ProtobufConverter`. URL for Schema Registry.                                                       |                                                  |
| `key.converter`                       | One between `org.apache.kafka.connect.storage.StringConverter`, `io.confluent.connect.avro.AvroConverter`, `io.confluent.connect.json.JsonSchemaConverter` or `io.confluent.connect.protobuf.ProtobufConverter`                                                     | org.apache.kafka.connect.storage.StringConverter |
| `key.converter.schema.registry.url`   | Only if `key.converter` is set to `io.confluent.connect.avro.AvroConverter`, `io.confluent.connect.json.JsonSchemaConverter` or `io.confluent.connect.protobuf.ProtobufConverter`. URL for Schema Registry.                                                         |                                                  |

### Available templates on StreamNative Cloud

* csv\_product
* csv\_user
* finance\_stock\_trade
* fleet\_mgmt\_sensors
* fleetmgmt\_description
* fleetmgmt\_location
* fleetmgmt\_sensor
* gaming\_game
* gaming\_player
* gaming\_player\_activity
* insurance\_customer
* insurance\_customer\_activity
* insurance\_offer
* inventorymgmt\_inventory
* inventorymgmt\_product
* iot\_device\_information
* marketing\_campaign\_finance
* net\_device
* payment\_credit\_card
* payment\_transaction
* payroll\_bonus
* payroll\_employee
* payroll\_employee\_location
* pizzastore\_order
* pizzastore\_order\_cancelled
* pizzastore\_order\_completed
* pizzastore\_util
* shoestore\_clickstream
* shoestore\_customer
* shoestore\_order
* shoestore\_shoe
* shopping\_order
* shopping\_purchase
* shopping\_rating
* siem\_log
* store
* syslog\_log
* user
* user\_with\_key
* users
* users\_array\_map
* util\_ip
* util\_userid
* webanalytics\_clickstream
* webanalytics\_code
* webanalytics\_page\_view
* webanalytics\_user


# Kafka connect milvus sink
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-milvus-sink/current/kafka-connect-milvus-sink

The official Milvus Kafka Connect Sink connector.

The Milvus Kafka Connect Sink connector is a Kafka Connect connector that writes data from Kafka topics to Milvus.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

### Prerequisites

* If you don't already have a collection in Zilliz Cloud or Milvus cluster, create a collection with a vector field.
* Collect the `endpoint`, `token`, and `collection.name` parameters from your Zilliz Cloud instance or Milvus cluster.

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)
2. Set up a Milvus cluster, we can set up a local Milvus cluster using docker-compose:

   ```shell theme={null}
   wget -q https://github.com/milvus-io/milvus/releases/download/${MILVUS_VERSION}/milvus-standalone-docker-compose.yml -O "tmp/docker-compose.yml" > /dev/null 2>&1
   docker compose -f "tmp/docker-compose.yml" up -d > /dev/null 2>&1
   ```
3. Initialize the local Milvus cluster, below is a python script to do so:
   ```python theme={null}
   from pymilvus import MilvusClient, DataType
   import time

   client = MilvusClient(
   uri="http://localhost:19530",
   db_name="default"
   )

   # 3.1. Create schema
   schema = MilvusClient.create_schema(
   auto_id=False,
   enable_dynamic_field=False,
   )
   schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True)
   schema.add_field(field_name="title", datatype=DataType.VARCHAR, max_length=65535)
   schema.add_field(field_name="title_vector", datatype=DataType.FLOAT_VECTOR, dim=8)
   schema.add_field(field_name="link", datatype=DataType.VARCHAR, max_length=65535)

   index_params = client.prepare_index_params()

   index_params.add_index(
   field_name="id",
   index_type="STL_SORT"
   )

   index_params.add_index(
   field_name="title_vector", 
   index_type="IVF_FLAT",
   metric_type="IP",
   params={ "nlist": 128 }
   )

   client.create_collection(
   collection_name="demo",
   schema=schema,
   index_params=index_params
   )

   time.sleep(5)
   ```
4. Create a JSON file like the following:

   ```json theme={null}
   {
       "name": "mysink13",
       "config": {
           "connector.class": "com.milvus.io.kafka.MilvusSinkConnector",
           "topics": "mytopic",
           "public.endpoint": "http://{MILVUS_IP}:19530",
           "token": "",
           "database.name": "default",
           "collection.name": "demo",
           "value.converter": "org.apache.kafka.connect.json.JsonConverter",
           "value.converter.schemas.enable": "false",
           "tasks.max": "1"
       }
   }
   ```
5. Run the following command to create the connector:

   ```shell theme={null}
   kcctl apply -f <filename>.json
   ```

### Configuration

The Milvus Kafka Connect Sink connector is configured using the following *Required* properties:

| Parameter         | Description                                                   |
| ----------------- | ------------------------------------------------------------- |
| `public.endpoint` | The endpoint of your Zilliz Cloud instance or Milvus cluster. |
| `token`           | The token of your Zilliz Cloud instance or Milvus cluster.    |
| `database.name`   | The name of the database to write to.                         |
| `collection.name` | The name of the collection to write to.                       |
| `topics`          | The Kafka topics to read from.                                |

For more information about the configuration properties, see the [offical Milvus Kafka Connect Sink Connector documentation](https://github.com/zilliztech/kafka-connect-milvus/blob/v1.0.1/README_OSS.md).

### Known Issues

1. The Milvus Kafka Connect Sink connector does not support the dynamic fields feature yet, and when the dynamic fields feature enabled on the Milvus server, or enabled on the Zilliz Cloud instance, the connector will not work properly and throw below error:

```shell theme={null}
2024-09-23T21:21:24,441+0000 [task-thread-mysink13-0] ERROR org.apache.kafka.connect.runtime.WorkerSinkTask - WorkerSinkTask{id=mysink13-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted. Error: 'com.google.protobuf.Internal$ProtobufList io.milvus.grpc.JSONArray.emptyList(java.lang.Class)'
java.lang.NoSuchMethodError: 'com.google.protobuf.Internal$ProtobufList io.milvus.grpc.JSONArray.emptyList(java.lang.Class)'
```

When you encounter this issue, please disable the dynamic fields feature on the Milvus server or Zilliz Cloud instance.


# Kafka connect mongodb sink
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-mongo-sink/current/kafka-connect-mongodb-sink

The official MongoDB Kafka Connect Sink connector.

The MongoDB Kafka sink connector is a Kafka Connect connector that reads data from Kafka topics and writes data to MongoDB.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

### Prerequisites

* Valid credentials with the `readWrite` role on the database. For more granular access control, you can specify a custom role that allows `insert`, `remove`, and `update` actions on the databases or collections.
* The `connection.uri` is in form of `mongodb+srv://username:password@cluster0.xxx.mongodb.net`

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)

2. Create a MongoDB Cluster, you can create one in k8s cluster with below yaml file:
   ```yaml theme={null}
   apiVersion: v1
   kind: Service
   metadata:
     name: mongo
     labels:
       name: mongo
   spec:
     ports:
       - port: 27017
     clusterIP: None
     selector:
       role: mongo
   ---
   apiVersion: apps/v1
   kind: StatefulSet
   metadata:
     name: mongo-dbz
   spec:
     selector:
       matchLabels:
         role: mongo
     serviceName: "mongo"
     replicas: 1
     template:
       metadata:
         labels:
           role: mongo
       spec:
         terminationGracePeriodSeconds: 10
         containers:
           - name: mongo
             image: debezium/example-mongodb:2.6
             env:
               - name: MONGODB_USER
                 value: "debezium"
               - name: MONGODB_PASSWORD
                 value: "dbz"
             command:
               - mongod
               - "--replSet"
               - rs0
               - "--bind_ip" # bind mongo to all ip address to allow others to access
               - "0.0.0.0"
             ports:
               - containerPort: 27017
           - name: mongo-sidecar
             image: cvallance/mongo-k8s-sidecar
             env:
               - name: MONGO_SIDECAR_POD_LABELS
                 value: "role=mongo"
               - name: KUBE_NAMESPACE
                 value: default
               - name: KUBERNETES_MONGO_SERVICE_NAME
                 value: "mongo"
   ```

3. Initialize the local MongoDB cluster:
   ```shell theme={null}
   kubectl apply -f .ci/clusters/mongodb.yaml
   kubectl wait -l role=mongo --for=condition=Ready pod --timeout=5m
   # initialize the data
   kubectl exec mongo-dbz-0 -c mongo -- bash ./usr/local/bin/init-inventory.sh
   ```

4. Create a JSON file like the following:
   ```json theme={null}
   {
       "name": "mongo-sink",
       "config": {
           "connector.class": "com.mongodb.kafka.connect.MongoSinkConnector",
           "connection.uri": "mongodb://mongo.default.svc.cluster.local:27017/?authSource=admin",
           "database": "kafka-mongo-sink",
           "topics": "kafka-mongo-input",
           "value.converter": "org.apache.kafka.connect.json.JsonConverter",
           "value.converter.schemas.enable": false,
           "tasks.max": "1"
       }
   }
   ```

5. Run the following command to create the connector:
   ```shell theme={null}
   kcctl apply -f <filename>.json
   ```

### Limitations

If you want to use the MongoDB CDC handler for data sourced from MongoDB instances by MongoDB source connector, you will need to select `STRING` or `BYTES` as the value converter for both MongoDB source and MongoDB sink connectors. Details can be found [here](https://www.mongodb.com/docs/kafka-connector/v1.13/sink-connector/fundamentals/change-data-capture/).

### Configuration

The MongoDB Kafka sink connector is configured using the following *Required* properties:

| Parameter        | Description                                                                                                                                                           |
| ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `connection.uri` | The connection URI for the MongoDB server.                                                                                                                            |
| `database`       | The MongoDB database name.                                                                                                                                            |
| `topics`         | A list of Kafka topics that the sink connector watches. (You can define either the `topics` or the `topics.regex` setting, but not both.)                             |
| `topics.regex`   | A regular expression that matches the Kafka topics that the sink connector watches. (You can define either the `topics` or the `topics.regex` setting, but not both.) |

The full properties are also available from the [offical MongoDB Kafka Sink Connector documentation](https://www.mongodb.com/docs/kafka-connector/v1.13/sink-connector/configuration-properties/).


# Kafka connect mongodb source
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-mongo-source/current/kafka-connect-mongodb-source

The official MongoDB Kafka Connect Source connector.

The MongoDB Kafka source connector is a Kafka Connect connector that reads data from MongoDB and writes data to Kafka topics.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

### Prerequisites

* The `connection.uri` is in form of `mongodb+srv://username:password@cluster0.xxx.mongodb.net`
* Valid credentials with the `read` role on the database. For more granular access control, you can specify a custom role that allows `find`, and `changeStream` actions on the databases or collections.

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)

2. Create a MongoDB Cluster, you can create one in k8s cluster with below yaml file:
   ```yaml theme={null}
   apiVersion: v1
   kind: Service
   metadata:
     name: mongo
     labels:
       name: mongo
   spec:
     ports:
       - port: 27017
     clusterIP: None
     selector:
       role: mongo
   ---
   apiVersion: apps/v1
   kind: StatefulSet
   metadata:
     name: mongo-dbz
   spec:
     selector:
       matchLabels:
         role: mongo
     serviceName: "mongo"
     replicas: 1
     template:
       metadata:
         labels:
           role: mongo
       spec:
         terminationGracePeriodSeconds: 10
         containers:
           - name: mongo
             image: debezium/example-mongodb:2.6
             env:
               - name: MONGODB_USER
                 value: "debezium"
               - name: MONGODB_PASSWORD
                 value: "dbz"
             command:
               - mongod
               - "--replSet"
               - rs0
               - "--bind_ip" # bind mongo to all ip address to allow others to access
               - "0.0.0.0"
             ports:
               - containerPort: 27017
           - name: mongo-sidecar
             image: cvallance/mongo-k8s-sidecar
             env:
               - name: MONGO_SIDECAR_POD_LABELS
                 value: "role=mongo"
               - name: KUBE_NAMESPACE
                 value: default
               - name: KUBERNETES_MONGO_SERVICE_NAME
                 value: "mongo"
   ```

3. Initialize the local MongoDB cluster:
   ```shell theme={null}
   kubectl apply -f .ci/clusters/mongodb.yaml
   kubectl wait -l role=mongo --for=condition=Ready pod --timeout=5m
   # initialize the data
   kubectl exec mongo-dbz-0 -c mongo -- bash ./usr/local/bin/init-inventory.sh
   ```

4. Create a JSON file like the following:
   ```json theme={null}
   {
       "name": "mongo-source",
       "config": {
           "connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",
           "connection.uri": "mongodb://mongo.default.svc.cluster.local:27017/?authSource=admin",
           "database": "kafka-mongo",
           "collection": "source",
           "value.converter": "org.apache.kafka.connect.json.JsonConverter",
           "value.converter.schemas.enable": false,
           "tasks.max": "1"
       }
   }
   ```

5. Run the following command to create the connector:
   ```shell theme={null}
   kcctl apply -f <filename>.json
   ```

### Configuration

The MongoDB Kafka source connector is configured using the following *Required* properties:

| Parameter        | Description                                                                                                                                                                  |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `connection.uri` | The connection URI for the MongoDB server.                                                                                                                                   |
| `database`       | The MongoDb database from which the connector imports data into Kafka topics. The connector monitors changes in this database. Leave the field empty to watch all databases. |
| `collection`     | The collection in the MongoDB database to watch. If not set, then all collections are watched.                                                                               |
| `topic.prefix`   | The prefix for the Kafka topics that the connector creates. The connector appends a database name and collection name to this prefix to create the topic name.               |

The full properties are also available from the [offical MongoDB Kafka Source Connector documentation](https://www.mongodb.com/docs/kafka-connector/v1.13/source-connector/configuration-properties/).


# Kafka connect snowflake sink
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-snowflake-sink/current/kafka-connect-snowflake-sink

The official Snowflake Kafka Connect Sink connector.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

### Prerequisites

* A running Snowflake instance in [Snowflake](https://www.snowflake.com/en/)

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)

2. Create a Snowflake instance

3. Setup the database, user in Snowflake, please refer to: [Snowflake Documentation](https://docs.snowflake.com/en/user-guide/kafka-connector-install#creating-a-role-to-use-the-kafka-connector)

4. Setup keypair: refer to: [Using key pair authentication & key rotation](https://docs.snowflake.com/en/user-guide/kafka-connector-install#using-key-pair-authentication-key-rotation)

5. Create a secret in StreamNative Console, and save the private key's content and passphrase to the secret, please refer to: [doc](https://docs.streamnative.io/docs/kafka-connect-create#create-kafka-connect-with-secret), let's say the secret name is `gcp`, and key is `auth`

6. Create a JSON file like the following:

   ```json theme={null}
   {
       "name": "snowflake-demo",
       "config": {
           "connector.class": "com.snowflake.kafka.connector.SnowflakeSinkConnector",
           "key.converter": "org.apache.kafka.connect.storage.StringConverter",
           "key.converter.schemas.enable": "false",
           "value.converter": "org.apache.kafka.connect.json.JsonConverter",
           "value.converter.schemas.enable": "false",
           "snowflake.ingestion.method": "SNOWPIPE_STREAMING",
           "snowflake.role.name": "kafka_connector_role_1",
           "snowflake.user.name": "kafka_connector_user_1",
           "snowflake.url.name": "${SNOWFLAKE_URL}:443",
           "snowflake.private.key": "${snsecret:snowflake-demo:snowflake.private.key}",
           "snowflake.private.key.passphrase": "${snsecret:snowflake-demo:snowflake.private.key.passphrase}",
           "topics": "snowflake-input",
           "snowflake.database.name": "kafka_db",
           "snowflake.schema.name": "public",
           "tasks.max": "1"
       }
   }
   ```

7. Run the following command to create the connector:

   ```bash theme={null}
   kcctl apply -f <filename>.json
   ```

### Configuration

The Snowflake Kafka sink connector is configured using the following *Required* properties:

| Parameter                 | Description                                                                                                                                                           |
| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name`                    | The name of the connector.                                                                                                                                            |
| `connector.class`         | `com.snowflake.kafka.connector.SnowflakeSinkConnector` .                                                                                                              |
| `topics`                  | A list of Kafka topics that the sink connector watches. (You can define either the `topics` or the `topics.regex` setting, but not both.)                             |
| `topics.regex`            | A regular expression that matches the Kafka topics that the sink connector watches. (You can define either the `topics` or the `topics.regex` setting, but not both.) |
| `snowflake.url.name`      | The URL for accessing your Snowflake account.                                                                                                                         |
| `snowflake.user.name`     | User login name for the Snowflake account.                                                                                                                            |
| `snowflake.private.key`   | The private key to authenticate the user. Include only the key, not the header or footer. If the key is split across multiple lines, remove the line breaks.          |
| `snowflake.database.name` | The name of the database that contains the table to insert rows into.                                                                                                 |
| `snowflake.schema.name`   | The name of the schema that contains the table to insert rows into.                                                                                                   |
| `header.converter`        | Required only if the records are formatted in Avro and include a header.                                                                                              |
| `key.converter`           | Kafka record's key converter.                                                                                                                                         |
| `value.converter`         | Kafka record's value converter.                                                                                                                                       |

For the full list of configs, see the [Official Snowflake Kafka Connect documentation](https://docs.snowflake.com/en/user-guide/kafka-connector-install#kafka-configuration-properties)


# Kafka connect yugabyte cdc source
Source: https://docs.streamnative.io/connect/connectors/kafka-connect-yugabyte-cdc-source/current/kafka-connect-yugabyte-cdc-source

The official YugabyteDB CDC Kafka Source Connector for capturing change data from YugabyteDB.

The YugabyteDB CDC Kafka Source connector is a Kafka Connect connector that reads change data from YugabyteDB and writes it to Kafka topics.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

### Prerequisites

* Collect the following information about your YugabyteDB cluster:
  * Master addresses
  * DB Stream ID
  * DB User and password

### Quick Start

1. Setup the kcctl client: [doc](https://docs.streamnative.io/docs/kafka-connect-setup)
2. Setup a YugabyteDB cluster, you can create one in k8s cluster with below yaml file:
   ```shell theme={null}
   helm repo add yugabytedb https://charts.yugabyte.com
   helm repo update

   helm upgrade --install yb-demo yugabytedb/yugabyte --version 2024.1.0
   kubectl wait -l app=yb-master --for=condition=Ready pod --timeout=10m
   kubectl wait -l app=yb-tserver --for=condition=Ready pod --timeout=10m

   # wait for the pod to be ready
   sleep 30

   kubectl exec yb-tserver-0 -- /home/yugabyte/bin/ysqlsh -c "CREATE TABLE IF NOT EXISTS test_kafka_connect(id int primary key, message text);"

   # create change stream
   stream_id=$(kubectl exec yb-master-0 -- yb-admin --master_addresses yb-masters.default.svc.cluster.local:7100 create_change_data_stream ysql.yugabyte)
   stream_id=$(echo $stream_id | awk '{print $4}')
   ```
3. Create a JSON file like the following:
   ```json theme={null}
   {
       "name": "yugabyte-cdc-source",
       "config": {
           "connector.class": "io.debezium.connector.yugabytedb.YugabyteDBConnector",
           "kafka.topic": "kafka-yugabyte-output",
           "tasks.max": "1",
           "database.hostname": "yb-tservers.default.svc.cluster.local",
           "database.port": "5433",
           "database.master.addresses": "yb-masters.default.svc.cluster.local:7100",
           "database.dbname": "yugabyte",
           "table.include.list": "public.test_kafka_connect",
           "database.streamid": "${STREAM_ID}",
           "snapshot.mode": "never",
           "database.user": "yugabyte",
           "database.password": "yugabyte",
           "database.server.name": "dbserver1",
           "key.converter": "org.apache.kafka.connect.storage.StringConverter",
           "key.converter.schemas.enable": "false",
           "value.converter": "org.apache.kafka.connect.json.JsonConverter",
           "value.converter.schemas.enable": "false"
       }
   }
   ```
4. Run the following command to create the connector:
   ```shell theme={null}
   kcctl apply -f <filename>.json
   ```

### Configuration

The YugabyteDB CDC Kafka Source connector is configured using the following *Required* properties:

| Property                    | Description                                   |
| --------------------------- | --------------------------------------------- |
| `database.master.addresses` | The addresses of the YugabyteDB master nodes. |
| `database.server.name`      | The name of the YugabyteDB server.            |
| `database.dbname`           | The name of the YugabyteDB database.          |
| `database.user`             | The user name for the YugabyteDB database.    |
| `database.password`         | The password for the YugabyteDB database.     |
| `database.streamid`         | The stream ID for the YugabyteDB database.    |
| `database.hostname`         | The hostname for the YugabyteDB database.     |
| `database.port`             | The port for the YugabyteDB database.         |

For more information about the properties, see the [offical YugabyteDB CDC Kafka Source Connector documentation](https://github.com/yugabyte/debezium-connector-yugabytedb/blob/v1.9.5.y.220.3/README.md).


# Kafka sink
Source: https://docs.streamnative.io/connect/connectors/kafka-sink/current/kafka-sink

The Kafka sink connector pulls messages from Pulsar topics and persists the messages to Kafka topics.

The [Kafka](https://kafka.apache.org/) sink connector pulls messages from Pulsar topics and persists the messages to Kafka topics. For more information about connectors, see [Connector Overview](https://docs.streamnative.io/docs/connector-overview).

This document introduces how to get started with creating an Kafka sink connector and get it up and running.

## Quick start

### Prerequisites

The prerequisites for connecting an Kafka sink connector to external systems include:

Apache Kafka: Ensure you have a running Kafka instance. You can follow the official Kafka [Quickstart guide](https://kafka.apache.org/quickstart) to set up a Kafka instance if you don't have one already.

### 1. Create a connector

The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector,
you need to replace `--sink-type kafka` with `--archive /path/to/pulsar-io-kafka.nar`. You can find the button to download the `nar` package at the beginning of the document.

<Note title="For StreamNative Cloud User">
  If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first.
</Note>

```bash theme={null}
pulsarctl sinks create \
  --sink-type kafka \
  --name kafka-sink \
  --tenant public \
  --namespace default \
  --inputs "Your topic name" \
  --parallelism 1 \
  --sink-config \
  '{
    "bootstrapServers": "localhost:9092",
    "topic": "kafka-topic-name",
    "ack": 1
  }'
```

The `--sink-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.
If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference.

<Note title="Note">
  You can also choose to use a variety of other tools to create a connector:

  * [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [RestAPI](https://pulsar.apache.org/sink-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document.
</Note>

### 2. Send messages to the topic

<Note title="Note">
  If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information.
</Note>

```java theme={null}
   PulsarClient client = PulsarClient.builder()
            .serviceUrl("{{Your Pulsar URL}}")
            .build();

   Producer<String> producer = client.newProducer(Schema.STRING)
     .topic("{{Your topic name}}")
     .create();

   String message = "hello kafka";
   MessageId msgID = producer.send(message);
   System.out.println("Publish " + message + " and message ID " + msgID);

   producer.flush();
   producer.close();
   client.close();
```

You can also send the message using the command line:

```sh theme={null}
$ bin/pulsar-client produce pulsar-topic-name --messages "hello kafka"
```

### 3. Check the data on kafka topic

You can consume the data from the kafka topic using the command:

```sh theme={null}
$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic kafka-topic-name --from-beginning
```

If everything is set up correctly, you should see the message "hello kafka" in the Kafka consumer.

## Configuration Properties

This table outlines the properties of a Kafka sink connector.

| Name                        | Type   | Required | Default                                                   | Description                                                                                                                                                                                                                                                                         |
| --------------------------- | ------ | -------- | --------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `bootstrapServers`          | String | true     | " " (empty string)                                        | A comma-separated list of host and port pairs for establishing the initial connection to the Kafka cluster.                                                                                                                                                                         |
| `acks`                      | String | true     | " " (empty string)                                        | The number of acknowledgments that the producer requires the leader to receive before a request completes. <br />This controls the durability of the sent records.                                                                                                                  |
| `batchsize`                 | long   | false    | 16384L                                                    | The batch size that a Kafka producer attempts to batch records together before sending them to brokers.                                                                                                                                                                             |
| `maxRequestSize`            | long   | false    | 1048576L                                                  | The maximum size of a Kafka request in bytes.                                                                                                                                                                                                                                       |
| `topic`                     | String | true     | " " (empty string)                                        | The Kafka topic which receives messages from Pulsar.                                                                                                                                                                                                                                |
| `keyDeserializationClass`   | String | false    | org.apache.kafka.common.serialization.StringSerializer    | The serializer class for Kafka producers to serialize keys.                                                                                                                                                                                                                         |
| `valueDeserializationClass` | String | false    | org.apache.kafka.common.serialization.ByteArraySerializer | The serializer class for Kafka producers to serialize values.<br /><br />The serializer is set by a specific implementation of [`KafkaAbstractSink`](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSink.java). |
| `producerConfigProperties`  | Map    | false    | " " (empty string)                                        | The producer configuration properties to be passed to producers. <br /><br />**Note:  other properties specified in the connector configuration file take precedence over this configuration**.                                                                                     |


# Kafka source
Source: https://docs.streamnative.io/connect/connectors/kafka-source/current/kafka-source

The Kafka source connector pulls messages from Kafka topics and persists the messages to Pulsar topics.

The [Kafka](https://kafka.apache.org/) source connector pulls messages from Kafka topics and persists the messages to Pulsar topics. For more information about connectors, see [Connector Overview](https://docs.streamnative.io/docs/connector-overview).

This document introduces how to get started with creating a Kafka source connector and get it up and running.

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

<img alt="" />

## Quick start

### Prerequisites

The prerequisites for connecting an Kafka source connector to external systems include:

Apache Kafka: Ensure you have a running Kafka instance. You can follow the official Kafka [Quickstart guide](https://kafka.apache.org/quickstart) to set up a Kafka instance if you don't have one already.

### 1. Create a connector

The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector,
you need to replace `--source-type kafka` with `--archive /path/to/pulsar-io-kafka.nar`. You can find the button to download the `nar` package at the beginning of the document.

<Note title="For StreamNative Cloud User">
  If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first.
</Note>

```bash theme={null}
pulsarctl sources create \
  --source-type kafka \
  --name kafka-source \
  --tenant public \
  --namespace default \
  --destination-topic-name "Your topic name" \
  --parallelism 1 \
  --source-config \
  '{
    "bootstrapServers": "localhost:9092",
    "topic": "kafka-topic-name",
    "groupId": "group-id"
  }'
```

The `--source-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.
If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference.

<Note title="Note">
  You can also choose to use a variety of other tools to create a connector:

  * [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [RestAPI](https://pulsar.apache.org/sink-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document.
</Note>

### 2. Send messages to the Kafka topic

You can send the message using the command line:

```sh theme={null}
$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kafka-topic-name
> hello pulsar
```

### 3. Check the data on Pulsar topic

You can consume the data from the Pulsar topic using the command:

```sh theme={null}
$ bin/pulsar-client consume --subscription-name my-subscription pulsar-topic-name -n 0
```

If everything is set up correctly, you should see the message "hello pulsar" in the Pulsar consumer.

## Configuration Properties

This table outlines the properties of a Kafka source connector.

| Name                        | Type    | Required | Default                                                     | Description                                                                                                                                                                                                                                                                            |
| --------------------------- | ------- | -------- | ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `bootstrapServers`          | String  | true     | " " (empty string)                                          | A comma-separated list of host and port pairs for establishing the initial connection to the Kafka cluster.                                                                                                                                                                            |
| `groupId`                   | String  | true     | " " (empty string)                                          | A unique string that identifies the group of consumer processes to which this consumer belongs.                                                                                                                                                                                        |
| `fetchMinBytes`             | long    | false    | 1                                                           | The minimum byte expected for each fetch response.                                                                                                                                                                                                                                     |
| `autoCommitEnabled`         | boolean | false    | true                                                        | If set to true, the consumer's offset is periodically committed in the background.<br /><br /> This committed offset is used when the process fails as the position from which a new consumer begins.                                                                                  |
| `autoCommitIntervalMs`      | long    | false    | 5000                                                        | The frequency in milliseconds that the consumer offsets are auto-committed to Kafka if `autoCommitEnabled` is set to true.                                                                                                                                                             |
| `heartbeatIntervalMs`       | long    | false    | 3000                                                        | The interval between heartbeats to the consumer when using Kafka's group management facilities. <br /><br />**Note: `heartbeatIntervalMs` must be smaller than `sessionTimeoutMs`**.                                                                                                   |
| `sessionTimeoutMs`          | long    | false    | 30000                                                       | The timeout used to detect consumer failures when using Kafka's group management facility.                                                                                                                                                                                             |
| `topic`                     | String  | true     | " " (empty string)                                          | The Kafka topic that sends messages to Pulsar.                                                                                                                                                                                                                                         |
| `consumerConfigProperties`  | Map     | false    | " " (empty string)                                          | The consumer configuration properties to be passed to consumers. <br /><br />**Note: other properties specified in the connector configuration file take precedence over this configuration**.                                                                                         |
| `keyDeserializationClass`   | String  | false    | org.apache.kafka.common.serialization.StringDeserializer    | The deserializer class for Kafka consumers to deserialize keys.<br /> The deserializer is set by a specific implementation of [`KafkaAbstractSource`](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java). |
| `valueDeserializationClass` | String  | false    | org.apache.kafka.common.serialization.ByteArrayDeserializer | The deserializer class for Kafka consumers to deserialize values.                                                                                                                                                                                                                      |
| `autoOffsetReset`           | String  | false    | earliest                                                    | The default offset reset policy.                                                                                                                                                                                                                                                       |


# Kinesis sink
Source: https://docs.streamnative.io/connect/connectors/kinesis-sink/current/kinesis-sink

The Kinesis sink connector pulls data from Pulsar and persists data into Amazon Kinesis.

The AWS Kinesis sink connector pulls data from Pulsar and persists data into Amazon Kinesis. For more information about connectors, see [Connector Overview](https://docs.streamnative.io/docs/connector-overview).

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

<img alt="" />

This document introduces how to get started with creating an AWS Kinesis sink connector and get it up and running.

## Quick start

### Prerequisites

The prerequisites for connecting an AWS Kinesis sink connector to external systems include:

1. Create a Kinesis data stream in AWS.
2. Create an [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and an `AccessKey`(Please record the value of `AccessKey` and its `SecretKey`).
3. Assign the following permissions to the AWS User:

* [AmazonKinesisFullAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonKinesisFullAccess.html)
* [CloudWatch:PutMetricData](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_PutMetricData.html): it is required because AWS Kinesis producer will periodically [send metrics to CloudWatch](https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-cloudwatch.html).

### 1. Create a connector

The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector,
you need to replace `--sink-type kinesis` with `--archive /path/to/pulsar-io-kinesis.nar`. You can find the button to download the `nar` package at the beginning of the document.

<Note title="For StreamNative Cloud User">
  If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first.
</Note>

```bash theme={null}
pulsarctl sinks create \
  --sink-type kinesis \
  --name kinesis-sink \
  --tenant public \
  --namespace default \
  --inputs "Your topic name" \
  --parallelism 1 \
  --sink-config \
  '{
    "awsRegion": "Your aws kinesis region", 
    "awsKinesisStreamName": "Your kinesis stream name",
    "awsCredentialPluginParam": "{\"accessKey\":\"Your AWS access key\",\"secretKey\":\"Your AWS secret access key\"}"
  }'
```

The `--sink-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.
If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference.

<Note title="Note">
  You can also choose to use a variety of other tools to create a connector:

  * [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [RestAPI](https://pulsar.apache.org/sink-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document.
</Note>

### 2. Send messages to the topic

<Note title="Note">
  If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information.
</Note>

```java theme={null}
   PulsarClient client = PulsarClient.builder()
            .serviceUrl("{{Your Pulsar URL}}")
            .build();

   Producer<String> producer = client.newProducer(Schema.STRING)
     .topic("{{Your topic name}}")
     .create();

   String message = "test-message";
   MessageId msgID = producer.send(message);
   System.out.println("Publish " + message + " and message ID " + msgID);

   producer.flush();
   producer.close();
   client.close();
```

### 3. Show data on AWS Kinesis console

You can use the AWS Kinesis `Data Viewer` to view the data. <img alt="" />

## Configuration Properties

This table outlines the properties of an AWS Kinesis sink connector.

| Name                        | Type          | Required | Sensitive | Default            | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| --------------------------- | ------------- | -------- | --------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `awsKinesisStreamName`      | String        | true     | false     | " " (empty string) | The Kinesis stream name.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| `awsRegion`                 | String        | true     | false     | " " (empty string) | The AWS Kinesis [region](https://www.aws-services.info/regions.html). <br /><br />**Example:**<br /> us-west-1, us-west-2.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `awsCredentialPluginName`   | String        | false    | false     | " " (empty string) | The fully-qualified class name of implementation of [AwsCredentialProviderPlugin](https://github.com/apache/pulsar/blob/master/pulsar-io/aws/src/main/java/org/apache/pulsar/io/aws/AwsCredentialProviderPlugin.java). Please refer to \[Configure AwsCredentialProviderPlugin]\(###Configure AwsCredentialProviderPlugin)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `awsCredentialPluginParam`  | String        | false    | true      | " " (empty string) | The JSON parameter to initialize `awsCredentialsProviderPlugin`. Please refer to \[Configure AwsCredentialProviderPlugin]\(###Configure AwsCredentialProviderPlugin)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `awsEndpoint`               | String        | false    | false     | " " (empty string) | A custom Kinesis endpoint. For more information, see [AWS documentation](https://docs.aws.amazon.com/general/latest/gr/rande.html).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `retainOrdering`            | Boolean       | false    | false     | false              | Whether Pulsar connectors retain the ordering when moving messages from Pulsar to Kinesis.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| `messageFormat`             | MessageFormat | false    | false     | ONLY\_RAW\_PAYLOAD | Message format in which Kinesis sink converts Pulsar messages and publishes them to Kinesis streams.<br /><br />Available options include:<br /><br /><li>`ONLY_RAW_PAYLOAD`: Kinesis sink directly publishes Pulsar message payload as a message into the configured Kinesis stream. <br /></li><li>`FULL_MESSAGE_IN_JSON`: Kinesis sink creates a JSON payload with Pulsar message payload, properties, and encryptionCtx, and publishes JSON payload into the configured Kinesis stream.<br /></li><li>`FULL_MESSAGE_IN_FB`: Kinesis sink creates a flatbuffers serialized payload with Pulsar message payload, properties, and encryptionCtx, and publishes flatbuffers payload into the configured Kinesis stream. <br /></li><li>`FULL_MESSAGE_IN_JSON_EXPAND_VALUE`:  Kinesis sink sends a JSON structure containing the record topic name, key, payload, properties, and event time. The record schema is used to convert the value to JSON. </li> |
| `jsonIncludeNonNulls`       | Boolean       | false    | false     | true               | Only the properties with non-null values are included when the message format is `FULL_MESSAGE_IN_JSON_EXPAND_VALUE`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `jsonFlatten`               | Boolean       | false    | false     | false              | When it is set to `true` and the message format is `FULL_MESSAGE_IN_JSON_EXPAND_VALUE`, the output JSON is flattened.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `retryInitialDelayInMillis` | Long          | false    | false     | 100                | The initial delay (in milliseconds) between retries.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `retryMaxDelayInMillis`     | Long          | false    | false     | 60000              | The maximum delay(in milliseconds) between retries.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |

### Configure AwsCredentialProviderPlugin

AWS Kinesis sink connector allows you to use three ways to connect to AWS Kinesis by configuring `awsCredentialPluginName`.

* Leave `awsCredentialPluginName` empty to get the connector authenticated by passing `accessKey` and `secretKey` in `awsCredentialPluginParam`.

```json theme={null}
{"accessKey":"Your access key","secretKey":"Your secret key"}
```

* Set `awsCredentialPluginName` to `org.apache.pulsar.io.aws.AwsDefaultProviderChainPlugin` to use the default AWS provider chain. With this option, you don’t need to configure `awsCredentialPluginParam`. For more information, see [AWS documentation](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default).

* Set `awsCredentialPluginName`to `org.apache.pulsar.io.aws.STSAssumeRoleProviderPlugin` to use the [default AWS provider chain](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default), and you need to configure `roleArn` and `roleSessionNmae` in `awsCredentialPluginParam`. For more information, see [AWS documentation](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html)

  ```json theme={null}
  {"roleArn": "arn...", "roleSessionName": "name"}
  ```


# Kinesis source
Source: https://docs.streamnative.io/connect/connectors/kinesis-source/current/kinesis-source

The Kinesis source connector pulls data from Amazon Kinesis and persists data into Pulsar

The Kinesis source connector pulls data from Amazon Kinesis and persists data into Pulsar. For more information about connectors, see [Connector Overview](https://docs.streamnative.io/docs/connector-overview).

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

<img alt="" />

This connector uses the [Kinesis Consumer Library](https://github.com/awslabs/amazon-kinesis-client) (KCL) to consume messages. The KCL uses [DynamoDB](https://docs.aws.amazon.com/streams/latest/dev/shared-throughput-kcl-consumers.html) to track checkpoints for consumers,
and uses [CloudWatch](https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-cloudwatch.html) to track metrics for consumers.

This document introduces how to get started with creating an AWS Kinesis source connector and get it up and running.

<Note title="Note">
  Currently, the Kinesis source connector only supports raw messages. If you use [AWS Key Management Service (KMS)](https://docs.aws.amazon.com/streams/latest/dev/server-side-encryption.html) encrypted messages, the encrypted messages are sent to Pulsar directly. You need to [manually decrypt](https://aws.amazon.com/blogs/big-data/encrypt-and-decrypt-amazon-kinesis-records-using-aws-kms/) the data on the consumer side of Pulsar.
</Note>

## Quick start

### Prerequisites

The prerequisites for connecting an AWS Kinesis source connector to external systems include:

1. Create a Kinesis data stream in AWS.
2. Create an [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and an `AccessKey`(Please record the value of `AccessKey` and its `SecretKey`).
3. Assign the following permissions to the AWS User:

* [AmazonKinesisFullAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonKinesisFullAccess.html)
* [CloudWatch:PutMetricData](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_PutMetricData.html): it is required because AWS Kinesis client will periodically [send metrics to CloudWatch](https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-cloudwatch.html).
* [AmazonDynamoDBFullAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonDynamoDBFullAccess.html): it is required because AWS Kinesis client will use [DynamoDB store checkpoint status](https://docs.aws.amazon.com/streams/latest/dev/shared-throughput-kcl-consumers.html#shared-throughput-kcl-consumers-what-is-leasetable).

### 1. Create a connector

The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector,
you need to replace `--source-type kinesis` with `--archive /path/to/pulsar-io-kinesis.nar`. You can find the button to download the `nar` package at the beginning of the document.

<Note title="For StreamNative Cloud User">
  If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first.
</Note>

```bash theme={null}
pulsarctl sources create \
  --source-type kinesis \
  --name kinesis-source \
  --tenant public \
  --namespace default \
  --destination-topic-name "Your topic name" \
  --parallelism 1 \
  --source-config \
  '{
    "awsRegion": "Your aws kinesis region", 
    "awsKinesisStreamName": "Your kinesis stream name",
    "awsCredentialPluginParam": "{\"accessKey\":\"Your AWS access key\",\"secretKey\":\"Your AWS secret access key\"}",
    "applicationName": "Your application name, which will be used as the table name for DynamoDB. E.g.: pulsar-io-kinesis"
  }'
```

The `--source-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.
If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference.

<Note title="Note">
  You can also choose to use a variety of other tools to create a connector:

  * [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [RestAPI](https://pulsar.apache.org/source-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document.
</Note>

### 2. Send messages to Kinesis

<Note title="Note">
  The following example uses KPL to send data to Kinesis. For more details, see [Writing to your Kinesis Data Stream Using the KPL](https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-writing.html)
</Note>

```java theme={null}
public static void main(String[] args) throws Exception {

  AWSCredentialsProvider credentialsProvider =
  new AWSStaticCredentialsProvider(new BasicAWSCredentials("Your access key", "Your secret key"));

  KinesisProducerConfiguration kinesisConfig = new KinesisProducerConfiguration();
  kinesisConfig.setRegion("Your aws kinesis region");
  kinesisConfig.setCredentialsProvider(credentialsProvider);
  KinesisProducer kinesis = new KinesisProducer(kinesisConfig);
  // Put some records 
  for (int i = 0; i < 10; ++i) {
    ByteBuffer data = ByteBuffer.wrap("test-kinesis-data".getBytes("UTF-8"));
    // doesn't block       
    kinesis.addUserRecord("Your kinesis stream name", "myPartitionKey", data);
  }
  kinesis.flush();
  Thread.sleep(60000);
}
```

### 3. Show data using Pulsar client

<Note title="Note">
  If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information.
</Note>

```
bin/pulsar-client \
--url "Your Pulsar serviceUrl" \
consume "The topic that you specified when you created the connector" -s "test-sub" -n 10 -p Earliest

----- got message -----
key:[myPartitionKey], properties:[=49643665543143983613442895450427674751028642409795813410], content:test-kinesis-data
sidebarTitle: overview.md
----- got message -----
key:[myPartitionKey], properties:[=49643665543143983613442895450430092602667871668145225762], content:test-kinesis-data
----- got message -----
key:[myPartitionKey], properties:[=4964366554314398361344289545044.0.3528487486297319931938], content:test-kinesis-data
sidebarTitle: overview.md
----- got message -----
key:[myPartitionKey], properties:[=49643665543143983613442895450432510454.0.300926494638114], content:test-kinesis-data
----- got message -----
key:[myPartitionKey], properties:[=49643665543143983613442895450433719380126715555669344290], content:test-kinesis-data
sidebarTitle: overview.md
----- got message -----
key:[myPartitionKey], properties:[=4964366554314398361344289545043492830594633018484.0.3466], content:test-kinesis-data
----- got message -----
key:[myPartitionKey], properties:[=4964366554314398361344289545043614.0.3765944814018756642], content:test-kinesis-data
sidebarTitle: overview.md
----- got message -----
key:[myPartitionKey], properties:[=49643665543143983613442895450437346157585559443193462818], content:test-kinesis-data
----- got message -----
key:[myPartitionKey], properties:[=49643665543143983613442895450438555083405174072368168994], content:test-kinesis-data
sidebarTitle: overview.md
----- got message -----
key:[myPartitionKey], properties:[=49643665543143983613442895450439764009224788701542875170], content:test-kinesis-data

```

## Configuration Properties

This table outlines the properties of an AWS Kinesis source connector.

| Name                       | Type                    | Required | Sensitive | Default                                                                                           | Description                                                                                                                                                                                                                                                                                                                                |   |
| -------------------------- | ----------------------- | -------- | --------- | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | - |
| `awsKinesisStreamName`     | String                  | true     | false     | " " (empty string)                                                                                | The Kinesis stream name.                                                                                                                                                                                                                                                                                                                   |   |
| `awsRegion`                | String                  | false    | false     | " " (empty string)                                                                                | The AWS region. <br /><br />**Example**<br /> us-west-1, us-west-2.                                                                                                                                                                                                                                                                        |   |
| `awsCredentialPluginName`  | String                  | false    | false     | " " (empty string)                                                                                | The fully-qualified class name of implementation of [AwsCredentialProviderPlugin](https://github.com/apache/pulsar/blob/master/pulsar-io/aws/src/main/java/org/apache/pulsar/io/aws/AwsCredentialProviderPlugin.java). For more information, see \[Configure AwsCredentialProviderPlugin]\(###Configure AwsCredentialProviderPlugin).      |   |
| `awsCredentialPluginParam` | String                  | false    | true      | " " (empty string)                                                                                | The JSON parameter to initialize `awsCredentialsProviderPlugin`. For more information, see \[Configure AwsCredentialProviderPlugin]\(###Configure AwsCredentialProviderPlugin).                                                                                                                                                            |   |
| `awsEndpoint`              | String                  | false    | false     | " " (empty string)                                                                                | The Kinesis end-point URL, which can be found at [here](https://docs.aws.amazon.com/general/latest/gr/rande.html).                                                                                                                                                                                                                         |   |
| `dynamoEndpoint`           | String                  | false    | false     | " " (empty string)                                                                                | The Dynamo end-point URL, which can be found at [here](https://docs.aws.amazon.com/general/latest/gr/rande.html).                                                                                                                                                                                                                          |   |
| `cloudwatchEndpoint`       | String                  | false    | false     | " " (empty string)                                                                                | The Cloudwatch end-point URL. For more information, see[Amazon documentation](https://docs.aws.amazon.com/general/latest/gr/rande.html).                                                                                                                                                                                                   |   |
| `applicationName`          | String                  | false    | false     | Pulsar IO connector                                                                               | The name of the Amazon Kinesis application, which will be used as the table name for DynamoDB.                                                                                                                                                                                                                                             |   |
| `initialPositionInStream`  | InitialPositionInStream | false    | false     | LATEST                                                                                            | The position where the connector starts from.<br /><br />Below are the available options:<br /><br /><li>`AT_TIMESTAMP`: start from the record at or after the specified timestamp.<br /></li><li>`LATEST`: start after the most recent data record.<br /></li><li>`TRIM_HORIZON`: start from the oldest available data record.      </li> |   |
| `startAtTime`              | Date                    | false    | false     | " " (empty string)                                                                                | If set to `AT_TIMESTAMP`, it specifies the time point to start consumption.                                                                                                                                                                                                                                                                |   |
| `checkpointInterval`       | Long                    | false    | false     | 60000                                                                                             | The frequency of the Kinesis stream checkpoint in milliseconds.                                                                                                                                                                                                                                                                            |   |
| `backoffTime`              | Long                    | false    | false     | 3000                                                                                              | The amount of time to delay between requests when the connector encounters a throttling exception from AWS Kinesis in milliseconds.                                                                                                                                                                                                        |   |
| `numRetries`               | int                     | false    | false     | 3                                                                                                 | The number of re-attempts when the connector encounters an exception while trying to set a checkpoint.                                                                                                                                                                                                                                     |   |
| `receiveQueueSize`         | int                     | false    | false     | 1000                                                                                              | The maximum number of AWS records that can be buffered inside the connector. <br /><br />Once the `receiveQueueSize` is reached, the connector does not consume any messages from Kinesis until some messages in the queue are successfully consumed.                                                                                      |   |
| `useEnhancedFanOut`        | boolean                 | false    | false     | true                                                                                              | If set to true, it uses Kinesis enhanced fan-out.<br /><br />If set to false, it uses polling.                                                                                                                                                                                                                                             |   |
| `kinesisRecordProperties`  | String                  | false    | false     | "kinesis.arrival.timestamp,kinesis.encryption.type,kinesis.partition.key,kinesis.sequence.number" | A comma-separated list of Kinesis metadata properties to include in the Pulsar message properties. The supported properties are: `kinesis.arrival.timestamp, kinesis.encryption.type, kinesis.partition.key, kinesis.sequence.number, kinesis.shard.id, kinesis.millis.behind.latest`                                                      |   |

### Configure AwsCredentialProviderPlugin

AWS Kinesis source connector allows you to use three ways to connect to AWS Kinesis by configuring `awsCredentialPluginName`.

* Leave `awsCredentialPluginName` empty to get the connector authenticated by passing `accessKey` and `secretKey` in `awsCredentialPluginParam`.

  ```json theme={null}
  {"accessKey":"Your access key","secretKey":"Your secret key"}
  ```

* Set `awsCredentialPluginName` to `org.apache.pulsar.io.aws.AwsDefaultProviderChainPlugin` to use the default AWS provider chain. With this option, you don’t need to configure `awsCredentialPluginParam`. For more information, see [AWS documentation](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default).

* Set `awsCredentialPluginName`to `org.apache.pulsar.io.aws.STSAssumeRoleProviderPlugin` to use the [default AWS provider chain](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default), and you need to configure `roleArn` and `roleSessionNmae` in `awsCredentialPluginParam`. For more information, see [AWS documentation](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html)

  ```json theme={null}
  {"roleArn": "arn...", "roleSessionName": "name"}
  ```


# Lakehouse sink
Source: https://docs.streamnative.io/connect/connectors/lakehouse-sink/current/lakehouse-sink

pulsar lakehouse connector

The Lakehouse sink connector (including the [Hudi](https://hudi.apache.org), [Iceberg](https://iceberg.apache.org/), and [Delta Lake](https://delta.io/) sink connectors) fetches data from a Pulsar topic and saves data to the Lakehouse tables.

![](https://raw.githubusercontent.com/streamnative/pulsar-hub/refs/heads/master/images/connectors/sync/lakehouse-lakehouse-sink.png)

# How to get

This section describes how to build the Lakehouse sink connector.

You can get the Lakehouse sink connector using one of the following methods:

* Download the NAR package from [the download page](https://github.com/streamnative/pulsar-io-lakehouse/releases).
* Build it from the source code.

To build the Lakehouse sink connector from the source code, follow these steps.

1. Clone the source code to your machine.

   ```bash theme={null}
   git clone https://github.com/streamnative/pulsar-io-lakehouse.git
   ```

2. Build the connector in the `pulsar-io-lakehouse` directory.

   * Build the NAR package for your local file system.

     ```bash theme={null}
     mvn clean install -DskipTests
     ```

   * Build the NAR package for your cloud storage (Including AWS, GCS and Azure related package dependency).

     ```bash theme={null}
     mvn clean install -P cloud -DskipTests
     ```

   After the connector is successfully built, a NAR package is generated under the target directory.

   ```bash theme={null}
   ls target
   pulsar-io-lakehouse-4.0.3.1.nar
   ```

# How to configure

Before using the Lakehouse sink connector, you need to configure it. This table lists the properties and the descriptions.

<Tabs>
  <Tab title="Hudi">
    For a list of Hudi configurations, see [Write Client Configs](https://hudi.apache.org/docs/configurations#WRITE_CLIENT).

    | Name                                          | Type           | Required | Default                 | Description                                                                                                                                                                                                                                                                    |   |
    | --------------------------------------------- | -------------- | -------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | - |
    | `type`                                        | String         | true     | N/A                     | The type of the Lakehouse source connector. Available values: `hudi`, `iceberg`, and `delta`.                                                                                                                                                                                  |   |
    | `maxCommitInterval`                           | Integer        | false    | 120                     | The maximum flush interval (in units of seconds) for each batch. By default, it is set to 120s.                                                                                                                                                                                |   |
    | `maxRecordsPerCommit`                         | Integer        | false    | 10\_000\_000            | The maximum number of records for each batch to commit. By default, it is set to `10_000_000`.                                                                                                                                                                                 |   |
    | `maxCommitFailedTimes`                        | Integer        | false    | 5                       | The maximum commit failure times until failing the process. By default, it is set to `5`.                                                                                                                                                                                      |   |
    | `sinkConnectorQueueSize`                      | Integer        | false    | 10\_000                 | The maximum queue size of the Lakehouse sink connector to buffer records before writing to Lakehouse tables.                                                                                                                                                                   |   |
    | `partitionColumns`                            | `List<String>` | false    | Collections.empytList() | The partition columns for Lakehouse tables.                                                                                                                                                                                                                                    |   |
    | `processingGuarantees`                        | Int            | true     | " " (empty string)      | The processing guarantees. Currently the Lakehouse connector only supports `EFFECTIVELY_ONCE`.                                                                                                                                                                                 |   |
    | `hudi.table.name`                             | String         | true     | N/A                     | The name of the Hudi table that Pulsar topic sinks data to.                                                                                                                                                                                                                    |   |
    | `hoodie.table.type`                           | String         | false    | COPY\_ON\_WRITE         | The type of the Hudi table of the underlying data for one write. It cannot be changed between writes.                                                                                                                                                                          |   |
    | `hoodie.base.path`                            | String         | true     | N/A                     | The base path of the lake storage where all table data is stored. It always has a specific prefix with the storage scheme (for example, hdfs\://, s3:// etc). Hudi stores all the main metadata about commits, savepoints, cleaning audit logs etc in the `.hoodie` directory. |   |
    | `hoodie.datasource.write.recordkey.field`     | String         | false    | UUID                    | The record key field. It is used as the `recordKey` component of `HoodieKey`. You can obtain the value by invoking `.toString()` on the field value. You can use the dot notation for nested fields such as a.b.c.                                                             |   |
    | `hoodie.datasource.write.partitionpath.field` | String         | true     | N/A                     | The partition path field. It is used as the `partitionPath` component of the `HoodieKey`.  You can obtain the value by invoking `.toString()`.                                                                                                                                 |   |
  </Tab>

  <Tab title="Iceberg">
    | Name                     | Type                  | Required | Default                 | Description                                                                                                                                                                                                                                                                                              |   |
    | ------------------------ | --------------------- | -------- | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | - |
    | `type`                   | String                | true     | N/A                     | The type of the Lakehouse source connector. Available values: `hudi`, `iceberg`, and `delta`.                                                                                                                                                                                                            |   |
    | `maxCommitInterval`      | Integer               | false    | 120                     | The maximum flush interval (in units of seconds) for each batch. By default, it is set to 120s.                                                                                                                                                                                                          |   |
    | `maxRecordsPerCommit`    | Integer               | false    | 10\_000\_000            | The maximum number of records for each batch to commit. By default, it is set to `10_000_000`.                                                                                                                                                                                                           |   |
    | `maxCommitFailedTimes`   | Integer               | false    | 5                       | The maximum commit failure times until failing the process. By default, it is set to `5`.                                                                                                                                                                                                                |   |
    | `sinkConnectorQueueSize` | Integer               | false    | 10\_000                 | The maximum queue size of the Lakehouse sink connector to buffer records before writing to Lakehouse tables.                                                                                                                                                                                             |   |
    | `partitionColumns`       | `List<String>`        | false    | Collections.empytList() | The partition columns for Lakehouse tables.                                                                                                                                                                                                                                                              |   |
    | `processingGuarantees`   | Int                   | true     | " " (empty string)      | The processing guarantees. Currently the Lakehouse connector only supports `EFFECTIVELY_ONCE`.                                                                                                                                                                                                           |   |
    | `catalogProperties`      | `Map<String, String>` | true     | N/A                     | The properties of the Iceberg catalog. For details, see  [Iceberg catalog properties](https://iceberg.apache.org/docs/latest/configuration/#catalog-properties). `catalog-impl` and `warehouse` configurations are required. Currently, Iceberg catalogs only support `hadoopCatalog` and `hiveCatalog`. |   |
    | `tableProperties`        | `Map<String, String>` | false    | N/A                     | The properties of the Iceberg table. For details, see [Iceberg  table properties](https://iceberg.apache.org/docs/latest/configuration/#table-properties).                                                                                                                                               |   |
    | `catalogName`            | String                | false    | icebergSinkConnector    | The name of the Iceberg catalog.                                                                                                                                                                                                                                                                         |   |
    | `tableNamespace`         | String                | true     | N/A                     | The namespace of the Iceberg table.                                                                                                                                                                                                                                                                      |   |
    | `tableName`              | String                | true     | N/A                     | The name of the Iceberg table.                                                                                                                                                                                                                                                                           |   |
  </Tab>

  <Tab title="Delta Lake">
    | Name                     | Type           | Required | Default                     | Description                                                                                                  |   |
    | ------------------------ | -------------- | -------- | --------------------------- | ------------------------------------------------------------------------------------------------------------ | - |
    | `type`                   | String         | true     | N/A                         | The type of the Lakehouse source connector. Available values: `hudi`, `iceberg`, and `delta`.                |   |
    | `maxCommitInterval`      | Integer        | false    | 120                         | The maximum flush interval (in units of seconds) for each batch. By default, it is set to 120s.              |   |
    | `maxRecordsPerCommit`    | Integer        | false    | 10\_000\_000                | The maximum number of records for each batch to commit. By default, it is set to `10_000_000`.               |   |
    | `maxCommitFailedTimes`   | Integer        | false    | 5                           | The maximum commit failure times until failing the process. By default, it is set to `5`.                    |   |
    | `sinkConnectorQueueSize` | Integer        | false    | 10\_000                     | The maximum queue size of the Lakehouse sink connector to buffer records before writing to Lakehouse tables. |   |
    | `partitionColumns`       | `List<String>` | false    | Collections.empytList()     | The partition columns for Lakehouse tables.                                                                  |   |
    | `processingGuarantees`   | Int            | true     | " " (empty string)          | The processing guarantees. Currently the Lakehouse connector only supports `EFFECTIVELY_ONCE`.               |   |
    | `tablePath`              | String         | true     | N/A                         | The path of the Delta table.                                                                                 |   |
    | `compression`            | String         | false    | SNAPPY                      | The compression type of the Delta Parquet file. compression type. By default, it is set to `SNAPPY`.         |   |
    | `deltaFileType`          | String         | false    | parquet                     | The type of the Delta file. By default, it is set to `parquet`.                                              |   |
    | `appId`                  | String         | false    | pulsar-delta-sink-connector | The Delta APP ID. By default, it is set to `pulsar-delta-sink-connector`.                                    |   |
  </Tab>
</Tabs>

<Note title="Note">
  The Lakehouse sink connector uses the Hadoop file system to read and write data to and from cloud objects, such as AWS, GCS, and Azure. If you want to configure Hadoop related properties, you should use the prefix `hadoop.`.
</Note>

## Examples

You can create a configuration file (JSON or YAML) to set the properties if you use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run connectors in a cluster.

<Tabs>
  <Tab title="Hudi">
    * The Hudi table that is stored in the file system

      ```json theme={null}
       {
           "tenant": "public",
           "namespace": "default",
           "name": "hudi-sink",
           "inputs": [
             "test-hudi-pulsar"
           ],
           "archive": "connectors/pulsar-io-hudi-4.0.3.1.nar",
           "processingGuarantees": "EFFECTIVELY_ONCE",
           "parallelism": 1,
           "configs":   {
               "type": "hudi",
               "hoodie.table.name": "hudi-connector-test",
               "hoodie.table.type": "COPY_ON_WRITE",
               "hoodie.base.path": "file:///tmp/data/hudi-sink",
               "hoodie.datasource.write.recordkey.field": "id",
               "hoodie.datasource.write.partitionpath.field": "id",
           }
       }
      ```

    * The Hudi table that is stored in the AWS S3

      ```json theme={null}
       {
           "tenant": "public",
           "namespace": "default",
           "name": "hudi-sink",
           "inputs": [
             "test-hudi-pulsar"
           ],
           "archive": "connectors/pulsar-io-hudi-4.0.3.1-cloud.nar",
           "parallelism": 1,
           "processingGuarantees": "EFFECTIVELY_ONCE",
           "configs":   {
               "type": "hudi",
               "hoodie.table.name": "hudi-connector-test",
               "hoodie.table.type": "COPY_ON_WRITE",
               "hoodie.base.path": "s3a://bucket/path/to/hudi",
               "hoodie.datasource.write.recordkey.field": "id",
               "hoodie.datasource.write.partitionpath.field": "id",
               "hadoop.fs.s3a.aws.credentials.provider": "com.amazonaws.auth.DefaultAWSCredentialsProviderChain"
           }
       }
      ```
  </Tab>

  <Tab title="Iceberg">
    * The Iceberg table that is stored in the file system

      ```json theme={null}
      {
          "tenant":"public",
          "namespace":"default",
          "name":"iceberg_sink",
          "parallelism":2,
          "inputs": [
            "test-iceberg-pulsar"
          ],
          "archive": "connectors/pulsar-io-lakehouse-4.0.3.1.nar",
          "processingGuarantees":"EFFECTIVELY_ONCE",
          "configs":{
              "type":"iceberg",
              "maxCommitInterval":120,
              "maxRecordsPerCommit":10000000,
              "catalogName":"test_v1",
              "tableNamespace":"iceberg_sink_test",
              "tableName":"ice_sink_person",
              "catalogProperties":{
                  "warehouse":"file:///tmp/data/iceberg-sink",
                  "catalog-impl":"hadoopCatalog"
              }
          }
      }
      ```

    * The Iceberg table that is stored in cloud storage (AWS S3, GCS, or Azure)

      ```json theme={null}
      {
          "tenant":"public",
          "namespace":"default",
          "name":"iceberg_sink",
          "parallelism":2,
          "inputs": [
            "test-iceberg-pulsar"
          ],
          "archive": "connectors/pulsar-io-lakehouse-4.0.3.1-cloud.nar",
          "processingGuarantees":"EFFECTIVELY_ONCE",
          "configs":{
              "type":"iceberg",
              "maxCommitInterval":120,
              "maxRecordsPerCommit":10000000,
              "catalogName":"test_v1",
              "tableNamespace":"iceberg_sink_test",
              "tableName":"ice_sink_person",
              "hadoop.fs.s3a.aws.credentials.provider": "com.amazonaws.auth.DefaultAWSCredentialsProviderChain",
              "catalogProperties":{
                  "warehouse":"s3a://test-dev-us-west-2/lakehouse/iceberg_sink",
                  "catalog-impl":"hadoopCatalog"
              }
          }
      }
      ```
  </Tab>

  <Tab title="Delta Lake">
    * The Delta table that is stored in the file system

      ```json theme={null}
      {
          "tenant":"public",
          "namespace":"default",
          "name":"delta_sink",
          "parallelism":1,
          "inputs": [
            "test-delta-pulsar"
          ],
          "archive": "connectors/pulsar-io-lakehouse-4.0.3.1.nar",
          "processingGuarantees":"EFFECTIVELY_ONCE",
          "configs":{
              "type":"delta",
              "maxCommitInterval":120,
              "maxRecordsPerCommit":10000000,
              "tablePath": "file:///tmp/data/delta-sink"
          }
      }
      ```

    * The Delta table that is stored in cloud storage (AWS S3, GCS, or Azure)

      ```json theme={null}
      {
          "tenant":"public",
          "namespace":"default",
          "name":"delta_sink",
          "parallelism":1,
          "inputs": [
            "test-delta-pulsar"
          ],
          "archive": "connectors/pulsar-io-lakehouse-4.0.3.1-cloud.nar",
          "processingGuarantees":"EFFECTIVELY_ONCE",
          "configs":{
              "type":"delta",
              "maxCommitInterval":120,
              "maxRecordsPerCommit":10000000,
              "tablePath": "s3a://test-dev-us-west-2/lakehouse/delta_sink",
              "hadoop.fs.s3a.aws.credentials.provider": "com.amazonaws.auth.DefaultAWSCredentialsProviderChain"
          }
      }
      ```
  </Tab>
</Tabs>

## Data format types

The Lakehouse sink connector provides multiple output format options, including Avro and Parquet. The default format is Parquet.
With the current implementation, there are some limitations for different formats:

This table lists the Pulsar Schema types supported by the writers.

| Pulsar Schema     | Writer: Avro | Writer: Parquet |
| ----------------- | ------------ | --------------- |
| Primitive         | ✗            | ✗               |
| Avro              | ✔            | ✔               |
| Json              | ✔            | ✔               |
| Protobuf \*       | ✗            | ✗               |
| ProtobufNative \* | ✗            | ✗               |

> \*: The Protobuf schema is based on the Avro schema. It uses Avro as an intermediate format, so it may not provide the best effort conversion.
>
> \*: The ProtobufNative record holds the Protobuf descriptor and the message. When writing to Avro format, the connector uses [avro-protobuf](https://github.com/apache/avro/tree/master/lang/java/protobuf) to do the conversion.

# How to use

You can use the Lakehouse sink connector with Function Worker. You can use the Lakehouse sink connector as a non built-in connector or a built-in connector.

<Tabs>
  <Tab title="Use it as a non built-in connector">
    If you already have a Pulsar cluster, you can use the Lakehouse sink connector as a non built-in connector directly.

    This example shows how to create a Lakehouse sink connector on a Pulsar cluster using the [`pulsar-admin sinks create`](http://pulsar.apache.org/tools/pulsar-admin/2.8.0-SNAPSHOT/#-em-create-em--24) command.

    ```
    PULSAR_HOME/bin/pulsar-admin sinks create \
    --sink-config-file <lakehouse-sink-config.yaml>
    ```
  </Tab>

  <Tab title="Use it as a built-in connector">
    You can make the Lakehouse sink connector as a built-in connector and use it on a standalone cluster or an on-premises cluster.

    ## Standalone cluster

    This example describes how to use the Lakehouse sink connector to fetch data from Pulsar topics and save data to Lakehouse tables in standalone mode.

    ### Prerequisites

    * Install Pulsar locally. For details, see [set up a standalone Pulsar locally](https://pulsar.apache.org/docs/en/standalone/#install-pulsar-using-binary-release).

    ### Steps

    1. Copy the NAR package to the Pulsar connectors directory.

       ```
       cp pulsar-io-lakehouse-4.0.3.1.nar PULSAR_HOME/connectors/pulsar-io-lakehouse-4.0.3.1.nar
       ```

    2. Start Pulsar in standalone mode.

       ```
       PULSAR_HOME/bin/pulsar standalone
       ```

    3. Run the lakehouse sink connector locally.

       ```
       PULSAR_HOME/bin/pulsar-admin sink localrun \
       --sink-config-file <lakehouse-sink-config.yaml>
       ```

    4. Send messages to Pulsar topics.

       This example sends ten “hello” messages to the `test-lakehouse-pulsar` topic in the `default` namespace of the `public` tenant.

       ```
       PULSAR_HOME/bin/pulsar-client produce public/default/test-lakehouse-pulsar --messages hello -n 10
       ```

    5. Query the data from the Lakehouse table. For details, see [Hudi Quickstart guide](https://hudi.apache.org/docs/quick-start-guide), [Iceberg Quickstart guide](https://iceberg.apache.org/docs/latest/getting-started/), and [Delta Quickstart guide](https://delta.io/learn/getting-started).

    ## On-premises cluster

    This example explains how to create a Lakehouse sink connector in an on-premises cluster.

    1. Copy the NAR package of the Lakehouse sink connector to the Pulsar connectors directory.

       ```bash theme={null}
       cp pulsar-io-lakehouse-4.0.3.1.nar $PULSAR_HOME/connectors/pulsar-io-lakehouse-4.0.3.1.nar
       ```

    2. Reload all [built-in connectors](https://pulsar.apache.org/docs/en/next/io-connectors/).

       ```bash theme={null}
       PULSAR_HOME/bin/pulsar-admin sinks reload
       ```

    3. Check whether the Lakehouse sink connector is available on the list or not.

       ```bash theme={null}
       PULSAR_HOME/bin/pulsar-admin sinks available-sinks
       ```

    4. Create a Lakehouse sink connector on a Pulsar cluster using the [`pulsar-admin sinks create`](http://pulsar.apache.org/tools/pulsar-admin/2.8.0-SNAPSHOT/#-em-create-em--24) command.

       ```bash theme={null}
       PULSAR_HOME/bin/pulsar-admin sinks create \
       --sink-config-file <lakehouse-sink-config.yaml>
       ```
  </Tab>
</Tabs>

# Demos

This table lists demos that show how to run the [Delta Lake](https://delta.io/), [Hudi](https://hudi.apache.org), and [Iceberg](https://iceberg.apache.org/) sink connectors with other external systems.

Currently, only the demo on the Delta Lake sink connector is available.

| Connector  | Link                                                                                                                             |
| ---------- | -------------------------------------------------------------------------------------------------------------------------------- |
| Delta Lake | For details, see the [Delta Lake demo](https://github.com/streamnative/pulsar-io-lakehouse/blob/master/docs/delta-lake-demo.md). |
| Hudi       |                                                                                                                                  |
| Iceberg    |                                                                                                                                  |


# Lakehouse source
Source: https://docs.streamnative.io/connect/connectors/lakehouse-source/current/lakehouse-source

pulsar lakehouse connector

The Lakehouse source connector (currently only including the [Delta Lake](https://delta.io/) source connector) fetches the Lakehouse table's changelog and saves changelogs into a Pulsar topic.

![](https://raw.githubusercontent.com/streamnative/pulsar-hub/refs/heads/master/images/connectors/sync/lakehouse-lakehouse-source.png)

# How to get

This section describes how to build the Lakehouse source connector.

You can get the Lakehouse source connector using one of the following methods:

* Download the NAR package from [the download page](https://github.com/streamnative/pulsar-io-lakehouse/releases).
* Build it from the source code.

To build the Lakehouse source connector from the source code, follow these steps.◊

1. Clone the source code to your machine.

   ```bash theme={null}
   git clone https://github.com/streamnative/pulsar-io-lakehouse.git
   ```

2. Build the connector in the `pulsar-io-lakehouse` directory.

   * Build the NAR package for your local file system.

     ```bash theme={null}
     mvn clean install -DskipTests
     ```

   * Build the NAR package for your cloud storage (Including AWS, GCS, and Azure-related package dependency).

     ```bash theme={null}
     mvn clean install -P cloud -DskipTests
     ```

   After the connector is successfully built, a NAR package is generated under the target directory.

   ```bash theme={null}
   ls target
   pulsar-io-lakehouse-4.0.3.1.nar
   ```

# How to configure

Before using the Lakehouse source connector, you need to configure it. This table lists the properties and the descriptions.

<Tabs>
  <Tab title="Delta Lake">
    | Name                       | Type   | Required | Default                                    | Description                                                                                                                                                                       |
    | -------------------------- | ------ | -------- | ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
    | `type`                     | String | true     | N/A                                        | The type of the Lakehouse source connector. Available values: `delta`.                                                                                                            |
    | `checkpointInterval`       | int    | false    | 30                                         | The checkpoint interval (in units of seconds). By default, it is set to 30s.                                                                                                      |
    | `queueSize`                | int    | false    | 10\_000                                    | The buffer queue size of the Lakehouse source connector. The buffer queue is used for store records before they are sent to Pulsar topics. By default, it is set to `10_000`.     |
    | `fetchHistoryData`         | bool   | false    | false                                      | Configure whether to fetch the history data of the table. By default, it is set to `false`.                                                                                       |
    | `startSnapshotVersion`     | long   | false    | -1                                         | The Delta snapshot version to start capturing data change. Available values: \[-1: LATEST, -2: EARLIEST]. The `startSnapshotVersion` and `startTimestamp` are mutually exclusive. |
    | `startTimestamp`           | long   | false    | N/A                                        | The Delta snapshot timestamp (in units of seconds) to start capturing data change. The `startSnapshotVersion` and `startTimestamp` are mutually exclusive.                        |
    | `tablePath`                | String | true     | N/A                                        | The path of the Delta table.                                                                                                                                                      |
    | `parquetParseThreads`      | int    | false    | Runtime.getRuntime().availableProcessors() | The parallelism of paring Delta Parquet files. By default, it is set to `Runtime.getRuntime().availableProcessors()`.                                                             |
    | `maxReadBytesSizeOneRound` | long   | false    | Total memory \* 0.2                        | The maximum read bytes size from Parquet files in one fetch round. By default, it is set to 20% of the heap memory.                                                               |
    | `maxReadRowCountOneRound`  | int    | false    | 100\_000                                   | The maximum read number of rows processed in one round. By default, it is set to `1_000_000`.                                                                                     |
  </Tab>
</Tabs>

<Note title="Note">
  The Lakehouse source connector uses the Hadoop file system to read and write data to and from cloud objects, such as AWS, GCS, and Azure. If you want to configure Hadoop related properties, you should use the prefix `hadoop.`.
</Note>

## Examples

You can create a configuration file (JSON or YAML) to set the properties if you use [Pulsar Function Worker](https://pulsar.apache.org/docs/en/functions-worker/) to run connectors in a cluster.

<Tabs>
  <Tab title="Delta Lake">
    * The Delta table that is stored in the file system

      ```json theme={null}
      {
          "tenant":"public",
          "namespace":"default",
          "name":"delta_source",
          "parallelism":1,
          "topicName": "delta_source",
          "processingGuarantees":"ATLEAST_ONCE",
          "archive": "connectors/pulsar-io-lakehouse-4.0.3.1.nar",
          "configs":{
              "type":"delta",
              "checkpointInterval": 180,
              "queueSize": 10000,
              "fatchHistoryData": false,
              "startSnapshotVersion": -1,
              "tablePath": "file:///tmp/data/delta-source",
              "parquetParseThreads": 3,
              "maxReadBytesSizeOneRound": 134217728,
              "maxReadRowCountOneRound": 100000
          }
      }
      ```

    * The Delta table that is stored in cloud storage (AWS S3, GCS, or Azure)

      ```json theme={null}
      {
          "tenant":"public",
          "namespace":"default",
          "name":"delta_source",
          "parallelism":1,
          "topicName": "delta_source",
          "processingGuarantees":"ATLEAST_ONCE",
          "archive": "connectors/pulsar-io-lakehouse-4.0.3.1-cloud.nar",
          "configs":{
              "type":"delta",
              "checkpointInterval": 180,
              "queueSize": 10000,
              "fatchHistoryData": false,
              "startSnapshotVersion": -1,
              "tablePath": "s3a://test-dev-us-west-2/lakehouse/delta_source",
              "hadoop.fs.s3a.aws.credentials.provider": "com.amazonaws.auth.DefaultAWSCredentialsProviderChain",
              "parquetParseThreads": 3,
              "maxReadBytesSizeOneRound": 134217728,
              "maxReadRowCountOneRound": 100000
          }
      }
      ```
  </Tab>
</Tabs>

## Data format types

Currently, The Lakehouse source connector only supports reading Delta table changelogs, which adopt a `parquet` storage format.

# How to use

You can use the Lakehouse source connector with Function Worker. You can use the Lakehouse source connector as a non built-in connector or a built-in connector.

<Tabs>
  <Tab title="Use it as a non built-in connector">
    If you already have a Pulsar cluster, you can use the Lakehouse source connector as a non built-in connector directly.

    This example shows how to create a Lakehouse source connector on a Pulsar cluster using the [`pulsar-admin sources create`](https://pulsar.apache.org/tools/pulsar-admin/2.8.0-SNAPSHOT/#-em-create-em--14) command.

    ```
    PULSAR_HOME/bin/pulsar-admin sources create \
    --source-config-file <lakehouse-source-config.yaml>
    ```
  </Tab>

  <Tab title="Use it as a built-in connector">
    You can make the Lakehouse source connector as a built-in connector and use it on a standalone cluster or an on-premises cluster.

    ## Standalone cluster

    This example describes how to use the Lakehouse source connector to fetch data from Lakehouse tables and save data to Pulsar topics in standalone mode.

    ### Prerequisites

    * Install Pulsar locally. For details, see [set up a standalone Pulsar locally](https://pulsar.apache.org/docs/en/standalone/#install-pulsar-using-binary-release).

    ### Steps

    1. Copy the NAR package to the Pulsar connectors directory.

       ```
       cp pulsar-io-lakehouse-4.0.3.1.nar PULSAR_HOME/connectors/pulsar-io-lakehouse-4.0.3.1.nar
       ```

    2. Start Pulsar in standalone mode.

       ```
       PULSAR_HOME/bin/pulsar standalone
       ```

    3. Run the lakehouse source connector locally.

       ```bash theme={null}
       PULSAR_HOME/bin/pulsar-admin sources localrun \
       --source-config-file <lakehouse-source-config.yaml>
       ```

    4. Write rows into the Lakehouse table. For details, see [Getting Started with Delta Lake](https://delta.io/learn/getting-started).

    5. Consume Pulsar topics to get changelogs.

       ```bash theme={null}
       PULSAR_HOME/bin/pulsar-client consume -s test-sub -n 0 <topic-name>
       ```

    ## On-premises cluster

    This example explains how to create a Lakehouse source connector in an on-premises cluster.

    1. Copy the NAR package of the Lakehouse source connector to the Pulsar connectors directory.

       ```
       cp pulsar-io-lakehouse-4.0.3.1.nar $PULSAR_HOME/connectors/pulsar-io-lakehouse-4.0.3.1.nar
       ```

    2. Reload all [built-in connectors](https://pulsar.apache.org/docs/en/next/io-connectors/).

       ```
       PULSAR_HOME/bin/pulsar-admin sources reload
       ```

    3. Check whether the Lakehouse source connector is available on the list or not.

       ```
       PULSAR_HOME/bin/pulsar-admin sources available-sources
       ```

    4. Create a Lakehouse source connector on a Pulsar cluster using the [`pulsar-admin sources create`](https://pulsar.apache.org/tools/pulsar-admin/2.8.0-SNAPSHOT/#-em-create-em--14) command.

       ```
       PULSAR_HOME/bin/pulsar-admin sources create \
       --source-config-file <lakehouse-source-config.yaml>
       ```
  </Tab>
</Tabs>

# Demos

This table lists demos that show how to run the [Delta Lake](https://delta.io/), [Hudi](https://hudi.apache.org), and [Iceberg](https://iceberg.apache.org/) source connectors with other external systems.

Currently, only the demo on the Delta Lake source connector is available.

| Connector  | Link                                                                                                                             |
| ---------- | -------------------------------------------------------------------------------------------------------------------------------- |
| Delta Lake | For details, see the [Delta Lake demo](https://github.com/streamnative/pulsar-io-lakehouse/blob/master/docs/delta-lake-demo.md). |
| Hudi       |                                                                                                                                  |
| Iceberg    |                                                                                                                                  |


# Mongodb sink
Source: https://docs.streamnative.io/connect/connectors/mongodb-sink/current/mongodb-sink

The MongoDB sink connector pulls messages from Pulsar topics and persists the messages to collections.

The MongoDB sink connector pulls messages from Pulsar topics and persists the messages to collections.

# Configuration

The configuration of the MongoDB sink connector has the following properties.

## Property

| Name          | Type   | Required | Sensitive | Default            | Description                                                                                                                                                                          |
| ------------- | ------ | -------- | --------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `mongoUri`    | String | true     | true      | " " (empty string) | The MongoDB URI to which the connector connects. <br /><br />For more information, see [connection string URI format](https://docs.mongodb.com/manual/reference/connection-string/). |
| `database`    | String | true     | false     | " " (empty string) | The database name to which the collection belongs.                                                                                                                                   |
| `collection`  | String | true     | false     | " " (empty string) | The collection name to which the connector writes messages.                                                                                                                          |
| `batchSize`   | int    | false    | false     | 100                | The batch size of writing messages to collections.                                                                                                                                   |
| `batchTimeMs` | long   | false    | false     | 1000               | The batch operation interval in milliseconds.                                                                                                                                        |

## Example

Before using the Mongo sink connector, you need to create a configuration file through one of the following methods.

* JSON

  ```json theme={null}
  {
      "mongoUri": "mongodb://localhost:27017",
      "database": "pulsar",
      "collection": "messages",
      "batchSize": "2",
      "batchTimeMs": "500"
  }
  ```

* YAML

  ```yaml theme={null}
  {
      mongoUri: "mongodb://localhost:27017"
      database: "pulsar"
      collection: "messages"
      batchSize: 2
      batchTimeMs: 500
  }
  ```


# Netty source
Source: https://docs.streamnative.io/connect/connectors/netty-source/current/netty-source

The Netty source connector opens a port that accepts incoming data via the configured network protocol and publish it to user-defined Pulsar topics

The Netty source connector opens a port that accepts incoming data via the configured network protocol and publish it to user-defined Pulsar topics.

This connector can be used in a containerized (for example, k8s) deployment. Otherwise, if the connector is running in process or thread mode, the instance may be conflicting on listening to ports.

# Configuration

The configuration of the Netty source connector has the following properties.

## Property

| Name              | Type   | Required | Default   | Description                                                                                                                                           |
| ----------------- | ------ | -------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `type`            | String | true     | tcp       | The network protocol over which data is transmitted to netty. <br /><br />Below are the available options:<br /><li>tcp</li><li>http</li><li>udp</li> |
| `host`            | String | true     | 127.0.0.1 | The host name or address on which the source instance listen.                                                                                         |
| `port`            | int    | true     | 10999     | The port on which the source instance listen.                                                                                                         |
| `numberOfThreads` | int    | true     | 1         | The number of threads of Netty TCP server to accept incoming connections and handle the traffic of accepted connections.                              |

## Example

Before using the Netty source connector, you need to create a configuration file through one of the following methods.

* JSON

  ```json theme={null}
  {
      "type": "tcp",
      "host": "127.0.0.1",
      "port": "10911",
      "numberOfThreads": "1"
  }
  ```

* YAML

  ```yaml theme={null}
  configs:
      type: "tcp"
      host: "127.0.0.1"
      port: 10999
      numberOfThreads: 1
  ```

# Usage

The following examples show how to use the Netty source connector with TCP and HTTP.

## TCP

1. Start Pulsar standalone.

   ```bash theme={null}
   $ docker pull apachepulsar/pulsar:{version}

   $ docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-netty-standalone apachepulsar/pulsar:{version} bin/pulsar standalone
   ```

2. Create a configuration file *netty-source-config.yaml*.

   ```yaml theme={null}
   configs:
       type: "tcp"
       host: "127.0.0.1"
       port: 10999
       numberOfThreads: 1
   ```

3. Copy the configuration file *netty-source-config.yaml* to Pulsar server.

   ```bash theme={null}
   $ docker cp netty-source-config.yaml pulsar-netty-standalone:/pulsar/conf/
   ```

4. Download the Netty source connector.

   ```bash theme={null}
   $ docker exec -it pulsar-netty-standalone /bin/bash
   curl -O http://mirror-hk.koddos.net/apache/pulsar/pulsar-{version}/connectors/pulsar-io-netty-{version}.nar
   ```

5. Start the Netty source connector.

   ```bash theme={null}
   $ ./bin/pulsar-admin sources localrun \
   --archive pulsar-io-{{pulsar:version}}.nar \
   --tenant public \
   --namespace default \
   --name netty \
   --destination-topic-name netty-topic \
   --source-config-file netty-source-config.yaml \
   --parallelism 1
   ```

6. Consume data.

   ```bash theme={null}
   $ docker exec -it pulsar-netty-standalone /bin/bash

   $ ./bin/pulsar-client consume -t Exclusive -s netty-sub netty-topic -n 0
   ```

7. Open another terminal window to send data to the Netty source.

   ```bash theme={null}
   $ docker exec -it pulsar-netty-standalone /bin/bash

   $ apt-get update

   $ apt-get -y install telnet

   $ root@1d19327b2c67:/pulsar# telnet 127.0.0.1 10999
   Trying 127.0.0.1...
   Connected to 127.0.0.1.
   Escape character is '^]'.
   hello
   world
   ```

8. The following information appears on the consumer terminal window.

   ```bash theme={null}
   ----- got message -----
   hello

   ----- got message -----
   world
   ```

## HTTP

1. Start Pulsar standalone.

   ```bash theme={null}
   $ docker pull apachepulsar/pulsar:{version}

   $ docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-netty-standalone apachepulsar/pulsar:{version} bin/pulsar standalone
   ```

2. Create a configuration file *netty-source-config.yaml*.

   ```yaml theme={null}
   configs:
       type: "http"
       host: "127.0.0.1"
       port: 10999
       numberOfThreads: 1
   ```

3. Copy the configuration file *netty-source-config.yaml* to Pulsar server.

   ```bash theme={null}
   $ docker cp netty-source-config.yaml pulsar-netty-standalone:/pulsar/conf/
   ```

4. Download the Netty source connector.

   ```bash theme={null}
   $ docker exec -it pulsar-netty-standalone /bin/bash
   curl -O http://mirror-hk.koddos.net/apache/pulsar/pulsar-{version}/connectors/pulsar-io-netty-{version}.nar
   ```

5. Start the Netty source connector.

   ```bash theme={null}
   $ ./bin/pulsar-admin sources localrun \
   --archive pulsar-io-{{pulsar:version}}.nar \
   --tenant public \
   --namespace default \
   --name netty \
   --destination-topic-name netty-topic \
   --source-config-file netty-source-config.yaml \
   --parallelism 1
   ```

6. Consume data.

   ```bash theme={null}
   $ docker exec -it pulsar-netty-standalone /bin/bash

   $ ./bin/pulsar-client consume -t Exclusive -s netty-sub netty-topic -n 0
   ```

7. Open another terminal window to send data to the Netty source.

   ```bash theme={null}
   $ docker exec -it pulsar-netty-standalone /bin/bash

   $ curl -X POST --data 'hello, world!' http://127.0.0.1:10999/
   ```

8. The following information appears on the consumer terminal window.

   ```bash theme={null}
   ----- got message -----
   hello, world!
   ```


# Pinecone sink
Source: https://docs.streamnative.io/connect/connectors/pinecone-sink/current/pinecone-sink

A connector to pinecone.io

<Note title="✅ Available on StreamNative Cloud">
  This connector is available as a built-in connector on StreamNative Cloud.
</Note>

# Pinecone Sink Connector

This connector allows access to pinecone.io with a pulsar topic. The sink connector
takes in messages and writes them if they are in a proper format to a Pinecone
index.

<img alt="" />

## Quick start

1. Pay for a license.
2. Create an index on pinecone.io

Do one of the following.

Either

* Download the image (from streamnative/pulsar-io-pinecone).

or

* Run the connector directly on StreamNative Cloud.

And finally

* Provide the configuration below and start the connector.

### Prerequisites

The prerequisites for connecting a Pinecone sink connector to external systems include:

1. A pinecone.io api key
2. A index name
3. A namespace name

See conf/pulsar-io-template.yaml for more information.

### 1. Create a connector

The following command shows how to use [pulsarctl](https://github.com/streamnative/pulsarctl) to create a `builtin` connector. If you want to create a `non-builtin` connector,
you need to replace `--sink-type pinecone` with `--archive /path/to/pulsar-io-pinecone.nar`. You can find the button to download the `nar` package at the beginning of the document.

<Note title="For StreamNative Cloud User">
  If you are a StreamNative Cloud user, you need [set up your environment](https://docs.streamnative.io/docs/connector-setup) first.
</Note>

```bash theme={null}
pulsarctl sink create \
  --sink-type pinecone \
  --name pinecone \
  --tenant public \
  --namespace default \
  --inputs "Your topic name" \
  --parallelism 1 \
  --sink-config \
  '{ "apiKey": "abcd-123","indexName": "test", "namespace": "test" }'
```

The `--sink-config` is the minimum necessary configuration for starting this connector, and it is a JSON string. You need to substitute the relevant parameters with your own.
If you want to configure more parameters, see [Configuration Properties](#configuration-properties) for reference.

<Note title="Note">
  You can also choose to use a variety of other tools to create a connector:

  * [pulsar-admin](https://pulsar.apache.org/docs/3.1.x/io-use/): The command arguments for `pulsar-admin` are similar to those of `pulsarctl`. You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [RestAPI](https://pulsar.apache.org/sink-rest-api/?version=3.1.1): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Terraform](https://github.com/hashicorp/terraform): You can find an example for [StreamNative Cloud Doc](https://docs.streamnative.io/docs/connector-create#create-a-built-in-connector).
  * [Function Mesh](https://functionmesh.io/docs/connectors/run-connector): The docker image can be found at the beginning of the document.
</Note>

### 2. Send messages to the topic

<Note title="Note">
  If your connector is created on StreamNative Cloud, you need to authenticate your clients. See [Build applications using Pulsar clients](https://docs.streamnative.io/docs/qs-connect#jumpstart-for-beginners) for more information.
</Note>

```java theme={null}
@Data
@ToString
public class TestMessage {
    public static void main(String[] args) {
        PulsarClient client = PulsarClient.builder()
        .serviceUrl("{{Your Pulsar URL}}")
        .build();

        Producer<String> producer = client.newProducer(Schema.STRING)
              .topic("my-topic")
              .create();

        String testMessage = '{ "id": "v1", "values": [1.0]}';

        MessageId msgID = producer.send(testMessage);
        System.out.println("Publish " + testMessage + " and message ID " + msgID);

        producer.flush();
        producer.close();
        client.close();
    }
}

```

### 3. Querying Data From Index

You can look in the query UI from Pinecone or you can run a raw Pinecone
query yourself using a client. There are several on the Pinecone website
which are listed including Python, Node, and cURL.

```python theme={null}
# Taken from https://www.pinecone.io/
# Mock vectorized search query (vectorize with LLM of choice)
query = [0.1]  # len(query) = 1, same as the indexed vectors

# Send query with (optional) filter to index and get back 1 result (top_k=1)
index.query(
  vector=query,
  top_k=1
)
```

## Configuration Properties

Before using the Pinecone sink connector, you need to configure it. This table outlines the properties and the descriptions.

| Name          | Type    | Required | Sensitive | Default | Description                                                                                                                                                                                                                                                                                                                      |
| ------------- | ------- | -------- | --------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| apiKey        | string  | True     | True      | None    | The API key for the Pinecone service. Find this in the Pinecone dashboard.                                                                                                                                                                                                                                                       |
| indexName     | string  | True     | False     | None    | The name of the Pinecone index to which you want to write data. Find this in the Pinecone dashboard.                                                                                                                                                                                                                             |
| namespace     | string  | True     | False     | None    | The name of the Pinecone namespace to which you want to write data. Find this in the Pinecone dashboard.                                                                                                                                                                                                                         |
| dimensions    | integer | False    | False     | None    | The number of dimensions required by the index. If a request is made to upsert data into an index with a different number of dimensions, the request will fail. If not provided the connector will make it's best attempt to upsert the data and if the connection fails due to a mismatch the message will eventually be DLQ'd. |
| queryMetadata | JSON    | False    | False     | None    | The metadata to be associated with the request to the index.This should be a JSON object in the form `{"key": "value", "key2": "value2" }`.                                                                                                                                                                                      |

## Advanced features

### Monitoring

Currently we provide several metrics for monitoring.

* `pinecone-upsert-successful`
* `pinecone-upsert-failed`
* `pinecone-connector-active`
* `pinecone-upsert-failed-no-config`
* `pinecone-upsert-failed-no-client`
* `pinecone-upsert-failed-no-index-connection`
* `pinecone-upsert-failed-parsing-error`
* `pinecone-upsert-failed-dimension-error`

These can all be used to manage the connectors status.

### Troubleshooting

If you get a failed upsert problem the most likely candidate is the formatting
of your messages. These are required to be in a format like the following.

```
`{ "id": "string", "values": [float, float, ...]}`
```

or the form

```
`{ "metadata": { "key": "value", "key2": "value2", ... }`, id: "string", "values": [float, float, ...]}
```

Other likely candidates are problems with your connection to Pinecone. Check your
configuration values and any exceptions that are ocurring from the connector.

Some example commands for debugging locally are as follows.

Produce a sample message.

```
pulsar-client produce persistent://public/default/pinecone-source -m '{"id":"v1", "values": [3.0]}' -s '\n'
```

Clear a backlog of messages.

```
pulsar-admin --admin-url http://localhost:8080 topics clear-backlog --subscription public/default/pinecone persistent://public/default/pinecone-source
```

Delete a topic subscription.

```
pulsar-admin --admin-url http://localhost:8080 topics unsubscribe \
    --subscription public/default/pinecone \
    persistent://public/default/pinecone-source
```

Consume a group of messages.

```
pulsar-client consume -n 1 persistent://public/default/pinecone-source -s public/default/pinecone
```

If you need to add a maven shell using jenv you can do this with a
helpful script.

```
mvn dependency:build-classpath -DincludeTypes=jar -Dmdep.outputFile=.cp.txt
jshell --class-path `cat .cp.txt`:target/classes
```

And remember if you have maven problems on install that you need to
use JDK 8 with this project.

```
mvn --version # should be java 8
jenv exec mvn # if using jenv you can exec the local version using
              # this
```

### Delivery guarantees

The Pulsar IO connector framework provides three [delivery guarantees](https://pulsar.apache.org/docs/next/functions-concepts#processing-guarantees-and-subscription-types): `at-most-once`, `at-least-once`, and `effectively-once`.

Currently, the Pinecone sink connector provides the at-least-once delivery guarantee.

### Examples

With the source connector you can connect to Pinecone with a valid configuration
and then write messages to it. An example using localrun is shown below.

```
pulsar-admin --admin-url http://localhost:8080/ sinks localrun --broker-service-url pulsar://localhost:6650/ --archive "file:///Users/your-user/src/pulsar-io-pinecone/pinecone-connector/target/pulsar-io-pinecone-0.2.0.nar" --classname "org.streamnative.pulsar.io.pinecone.PineconeConnectorSink" --name "pinecone" --sink-config '{ "apiKey": "abcd-123","indexName": "test", "namespace": "test", "dimensions": 1 }' --inputs persistent://public/default/pinecone-source
```

This can be used when building the JAR of the project from scratch using
`mvn clean install`.

Similar configuration can be setup when using an image mounted with a config file
defining environment varia