streamnative.io
Help Center
Login
  1. StreamNative Managed Cloud
  2. Reference

Cloud release notes

July 2023

What's New

  • Documentation Site Refresh

    StreamNative’s documentation site has been redesigned. There are now two separate sub-sites for StreamNative Managed Cloud and StreamNative Platform, and each section is organized by the type of work that you are trying to do:

    • Build Applications: This section is for Pulsar users within companies that are using Pulsar to build applications, such as application development teams. (They do not need to know anything about managing or architecting/administering Pulsar.)
    • Administer: This section is for Pulsar power users and core platform teams that manage Pulsar components. This section includes information on creating tenants, namespaces, and topics, as well as managing organizations and instances.
    • Manage Infrastructure: This section is primarily for customers who are managing clusters and infrastructure. (That is, SRE folks - cluster, metrics, logs.)

    new-doc-site-demo.gif

Other Enhancements

  • Show memory and CPU when configuring Brokers and Bookies

    The Advanced tab for configuring a Pulsar cluster now surfaces both memory and CPU for Brokers and Bookies, making it easier to understand your cluster’s compute and storage capabilities.

    Broker and bookie usage

  • Pulsar version 2.11.1.2 for newly deployed Hosted clusters

    Updates from version 2.10.3.7 include the following:

    • Upgrade Pulsar server Java build version to 17.
    • Support chunking with shared subscriptions.
    • Support the pluggable topic factory.
    • Support configuring the compression type for ManagedCursorInfo to balance storage resource usage.

    To learn more about updates included in this version, see What's New in Apache Pulsar 2.11.

    You can see what version your Pulsar cluster is on by checking the Details tab on the cluster configuration page. If you want to upgrade your existing clusters, please submit a ticket.

  • Pulsar Admin Go version 0.1.1

    Pulsar Admin Go library v0.1.1 has been released, and includes the following updates:

    - Add two new parameters (BrokerClientTrustCertsFilePath and BrokerClientTlsEnabled) to support TLS when creating a cluster. - Add GetMessageID to the API, which returns the message ID by timestamp(ms) of a topic.

    To learn more about Pulsar Admin Go, check out the blog post here.

  • Functions Usage Chart

    The Organization Usage page on StreamNative Cloud Console now includes a chart showing an organization’s function usage in Compute Units (CUs), for organizations that have functions enabled.

    Function Usage

Bug Fixes

Fixed a bug that caused the same metrics to show up for multiple Pulsar clusters within an instance.

June 2023

What's New

  • New Connector: AWS EventBridge Sink Connector

  • StreamNative Hub

    The StreamNative Hub has a new design and is quickly accessible from the StreamNative Docs site, making it easier to find connectors and understand which connectors are supported on StreamNative Cloud. StreamNative Hub

  • Pulsar 3.0 available on StreamNative Cloud

    Pulsar 3.0, the next evolution of Pulsar, is now available on StreamNative Cloud. Pulsar 3.0 allows teams to run even bigger workloads, and is easier for developers to work with on their local machines. Updates include LTS support, an improved load balancer, several performance improvements, and Docker images for M1/M2 Macs. You can try Pulsar 3.0 on a new cluster or reach out to the StreamNative Support Team about upgrading existing clusters.

Other Enhancements

  • Additional Version Details available on StreamNative Cloud Console

    Pulsar, BookKeeper, and ZooKeeper versions are now available on the cluster details page, giving you greater insight into your cluster’s configuration details. Cluster details page

  • Improved experience on the Organization Usage page

    It is now easier to understand how many resources your clusters consume. You can view usage metrics (such as compute, storage and data) for a specific instance and then compare it to the organization’s usage across all instances. Organization usage selector.gif

  • AWS Marketplace listing update

    We have updated our product listing on AWS Marketplace. With the more streamlined listing, you can view usage data via the Organization Usage page.

May 2023

What's New

  • New Setup Guide. A new setup guide has been added to the Dashboard page and contains everything that you need to get up and running with a hosted cluster. On the Dashboard page, you will see step-by-step instructions on adding a payment method, creating and configuring Pulsar clusters, creating topics, setting up clients, and sending messages. The setup guide can be viewed by logging in to the StreamNative Cloud Console and clicking Dashboard on the left navigation pane. new setup guide

  • Organization Usage Page. Customers with StreamNative Managed Clusters (Hosted + BYOC) can now view both real-time and historical cluster usage data on the new Organization Usage page. This page features charts visualizing the usage dimensions (Compute Units, Storage Units, Throughput, and Storage Size) for all clusters within an organization. You can view usage by instance or in aggregate across your organization and narrow the date range to hone in on usage over a specific time period. You can view up to the past 60 days of usage. org usage view

  • Improved Usability for Kafka Clients. Customers with clusters that have Kafka protocol enabled will include a new Kafka Clients page that walks you through using Kafka clients to connect to your clusters. The Kafka client setup wizard enables application developers using Kafka clients to easily connect to their organization's Pulsar cluster. This page includes client libraries (Java, Go, Python, and Node.js), Apache Kafka CLI tools, KStream applications, and KSQL. In addition, the Kafka protocol is also enabled by default on all new managed Pulsar clusters (Hosted + BYOC). kafka client setup wizard

Other Enhancements

  • Refreshed Dashboard for managing instances and user accounts on StreamNative Cloud Console. You will see two new cards on the Dashboard page, surfacing additional information about your organization's instances and user accounts, allowing you to manage instances and user accounts directly from the Dashboard page without going to a separate screen.

  • New hosted clusters on Pulsar 2.10.3.7. New hosted Pulsar clusters (clusters hosted on StreamNative's public cloud account) will now be on version 2.10.3.7, where they were previously on version 2.9.3.21. As a result, you will benefit from the following enhancements:

    • Pulsar provides automatic failure recovery between the primary and backup clusters.
    • With the lazy-loading feature added to PartitionedProducer, fewer producers are needed and the broker memory utilization is improved.
    • Topic map support added with a new TableView type using key values in received messages.

    To learn more about features offered in this version, see What's New in Apache Pulsar 2.10. If you would like to upgrade your cluster, please submit a ticket.

  • Audit log now includes produce and consume events. For clusters on Pulsar 2.10.3.7 and later, Produce and Consume events can now be tracked in the audit log. audit log

  • Streamlined instance deletion experience. You can now delete an instance and all Pulsar clusters within it in one step, rather than having to individually delete all associated clusters before deleting the instance. streamlined instance deletion

  • Snowflake Sink Connector upgrade. The Snowflake Sink Connector on StreamNative Cloud Console has been upgraded from 2.9.4.1 to 2.10.3.4. In addition to smaller bug fixes, this connector now supports concurrency where it previously did not, meaning the replicas can now be set to more than 1 when configuring a connector. Snowflake Sink Connector supports concurrency

Bug Fixes

  • Fixed an issue preventing the creation of a Debezium-Postgres connector via the StreamNative Cloud Console.
  • Fixed an issue where namespaces with lots of topics would sometimes show a spurious error on the Namespace page of the StreamNative Cloud Console.
  • Fixed an issue causing configuration details on the Functions page to display without formatting.

March + April 2023

What's New

  • New Clients Page. The Clients page makes it easier to connect to a Pulsar cluster with your client of choice. You'll find comprehensive example snippets with better syntax highlighting, more programming languages (Java, Go, Python, C++, Node.js, Rust, Spring, C#), and more REST API and CLI tool examples.

    You can download authentication tokens (either an OAuth2 credential file or a JWT token) directly from the Clients page.

    To view the new page, select a programming language, CLI tool, or REST API. Then, you can see installation and connection instructions. Watch the video below for an example of connecting to a cluster using the Java client. Client Setup page in the Cloud Console

  • Pulsar Transactions on StreamNative Cloud. You can now configure Pulsar Transactions using the StreamNative Cloud Console.

    Pulsar Transactions (in Beta on StreamNative Cloud) enable event-streaming applications to consume, process, and produce messages in one atomic operation, including:

    • Atomic writes across multiple topic partitions
    • Atomic acknowledgments across multiple topic partitions
    • All the operations made within one transaction either all succeed, or all fail
    • Consumers are ONLY allowed to read committed messages Transactions

Improvements

  • Navigation bar refresh. The new design for the navigation bar on the StreamNative Cloud Console includes a lighter theme and updated icons, making it easier to navigate the console. To colocate all Admin tasks, the Clients tab was moved to the Admin section. New Cloud navigation bar

  • Pulsar cluster names. With this release, the maximum number of characters for Pulsar cluster names is 10 characters. (Note: Existing Pulsar clusters with names longer than 10 characters are not affected by this change.)

  • Request a region. If you are creating a new cluster using StreamNative Cloud Console and a region is not currently supported, you can request support for that region. Support for requesting a new region

Bug Fixes

  • Previously, it was possible to delete instances with active clusters. Now, before you can delete an instance, you must first delete all clusters within that instance.

  • Previously, you could not disable the audit log once it was turned on (enabled). With this release, it is now possible to toggle the audit log on and off.

Breaking Changes

  • Deprecation of free cloud. New customers can no longer sign up for free clusters. If you are interested in requesting access to StreamNative Cloud, please Contact us.

February 2023

New provisioning changes for hosted clusters

We have improved the customer experience for self-service provisioning, making it easier for you to provision StreamNative hosted clusters.

  • You can provision new clusters through StreamNative Cloud console, with a prompt to update payment information.
  • New changes will not impact existing clusters. To adjust the sizing of existing clusters, please submit a ticket, and our team will assist with this one-time migration.

Audit Log

Audit Log allows you to monitor which user accounts and applications are accessing data, and what they do with it. Audit Log supports various audit event types, including creating, updating, and deleting clusters, tenants, namespaces, and topics.

New connectors

New connectors are available on StreamNative Cloud:

If you are interested in using these connectors, submit a ticket with this request to the support team.

Cluster Metrics in Beta

StreamNative Cloud offers a new endpoint and dashboard for Cluster Metrics to help provide visibility into your Pulsar clusters and track performance over time. Collect and monitor cluster health and performance metrics in real time, analyze trends, and proactively maintain your applications to ensure optimal performance.

screenshot of new cluster metrics

Pulsar Functions on Cloud in Beta

Pulsar Functions are now available on StreamNative Cloud. Pulsar Functions on StreamNative Cloud enable you to build real-time data pipelines for ETL jobs, event-driven applications, and simple data analytics applications.

Check out this demo of Pulsar Functions in action:

https://www.youtube.com/watch?v=ycfwzlKhI5A

REST API

New REST API for StreamNative Cloud allows the following:

  • Produce, consume, and acknowledge messages
  • Monitor the state of your Pulsar clusters
  • Perform a wide range of administrative actions

Check out this demo of the StreamNative Rest API in action:

https://www.youtube.com/watch?v=pds3gizWEyw

Improvements for new clusters

We have made several improvements to new Pulsar clusters on StreamNative Cloud. These changes will not impact existing clusters, but new clusters will benefit from the following enhancements:

Cluster configurations added to the cluster Overview tab

You can view a new configuration section on the Overview tab on the Clusters page in the StreamNative Cloud Console for better visibility and accessibility. This allows you to quickly access and manage information on the backlog quota, backlog retention policy, subscription throttling rate, and producer/consumer limits per topic.

screenshot of new cluster overview section

Cloud Console optimizations

To facilitate easier debugging and troubleshooting of data issues with a topic, a new Stats tab has been added to the Topic page in StreamNative Cloud. This tab displays the output of the commands topic-stats and topic-stats-internal, making it easier for you to access and analyze topic statistics, streamlining the debugging process.

Instance selection made clearer

The StreamNative Cloud Console dashboard now clearly indicates the currently selected instance through the addition of a checkmark. This enhancement eliminates confusion and allows you to easily identify the active instance.

screenshot of updated instance selection

Documentation updates

In January 2023, the StreamNative Cloud Documentation Team rolled out the first version of a new documentation experience. We implemented several new features and sections for the documentation site, including the following:

  • A card-style landing page.
  • Enhanced navigation with breadcrumbs and back/next links on every page.
  • A new search engine for faster, more accurate search results.
  • New About section for Apache Pulsar.
  • New Coming Soon section that lists upcoming and current beta features. For details, see the Coming Soon overview.

November 2022

This update includes changes to the StreamNative Hosted Cloud documentation.

  • The procedure for creating a StreamNative Cloud account through the AWS Marketplace was updated and the help topic now reflects the correct workflow. For details, see the AWS billing page.

  • The Google BigQuery sink was added as a new connector and the documentation page was updated accordingly. For details, see the work with sink connectors page.

  • The name StreamNative Cloud Manager was updated to StreamNative Cloud Console throughout the documentation site. This change was made to be consistent with the app.

October 2022

This update includes enhancements and updates to the StreamNative Cloud Manager interface.

  • In StreamNative Cloud Manager Release 2.2.2, a new sign in flow was added for first-time users. For details, see the quick start.

  • The Flink SQL UI option has been deprecated. However, you can still work with Flink SQL through the CLI.

  • A new source connector, Debezium Microsoft SQL Server, was added. For details, see work with source connectors.

May 2022

This update reflects several important changes to the onboarding flow for StreamNative Cloud Manager including support for Single Sign-On (SSO), automatic organization deployment, and more choices in cloud regions for clusters.

  • For StreamNative Cloud Manager Release 2.1.7, support for SSO was added to the login process. For details, see the Quick start.
  • A default organization is now created for you when you sign up for StreamNative Cloud, saving you time in the set up process. For details, see Work with organizations.
  • Clusters include more choices in cloud provider regions. For details, see Work with clusters.

March 2022

StreamNative Cloud is designed to make it easy for teams to leverage the power of Apache Pulsar. We are working hard on improving the StreamNative Cloud Console to make it simpler for teams to provision and understand their Pulsar clusters.

Today, we are pleased to announce the release of an updated Cloud Console that brings some new features and refinements to better meet your needs. This release features a new Cloud Dashboard and Navigation to provide better visibility into your clusters and to provide more intuitive navigation. This is the first in many releases, and over the coming weeks and months. Expect to see more improvements that add additional features and refinements in the future.

If you have any questions or issues about this update, don't hesitate to reach out through the support portal or give us feedback through the chat app.

Highlights

  • New Dashboard page that serves as a quick health check of your instances with your most important tasks close at hand.
  • Updated navigation menu to simplify and streamline the management of StreamNative and Pulsar resources.
  • Significant internal re-architecture to make the UI faster and more responsive, without some of the previous delays of API changes not being shown in the UI.

What's New

  • Cloud Console Dashboard and login changes
    • This new home page is designed to make it easy for you to see the status of your instances and to quickly get started with StreamNative Cloud.
    • The Cloud Console now remembers your most recent organization and instance.
  • New Navigation
    • The navigation menu has been significantly changed to make navigation more intuitive.
    • A new quick switcher for tenants and namespaces has been added to the navigation bar.

Changes

  • Update your bookmarks! The links have changed. The previous links did not allow for deep linking.
  • Certain metrics, such as number of topics in a namespace, traffic, and others have been removed. These metrics were a point-in-time snapshot that was often out-of-date, potentially leading to confusion.
  • Numerous bug fixes and performance improvements have been implemented. These bug fixes include issues with changes via the Pulsar REST API not being visible within the UI and issues with large numbers of topics or namespaces.

Future plans

This update is just the beginning. More features are coming soon, including:

  • Additional dashboard cards with contextual actions to simplify managing your Pulsar infrastructure.
  • Richer, historical metrics to see how your instance and applications are performing.
  • Additional improvements to the provisioning workflow to make it easier to deploy and manage instances and clusters.

Documentation improvements

February 2022

This release covers several documentation updates.

  • A new topic on StreamNative Cloud Secrets was added to the Managed Access section. StreamNative Cloud Secrets allow you to store and manage sensitive data such as passwords, tokens, and private keys. You can create Secrets and refer to them for computing purposes (Pulsar connectors, Pulsar Functions, Flink SQL and general Flink jobs). For details, see work with Secrets.
  • Per-topic throughput limit and storage limit for clusters was updated in the Clusters topic.
  • Billing information was removed from the Billing topic, as up-to-date billing information is on the StreamNative website.
  • The snctl reference topic was updated with an improved order of information.

January 2022

Going forward, you will continue to see changes and improvements to the documentation site, in addition to keeping up with product changes. Updates will include the following:

  • New layout and formatting
  • Clear, consistent language for faster reading
  • Improved cross-referencing for easier navigation
  • Streamlined procedures
  • Updated organization
  • New sections for specific use cases

We welcome your feedback! Email us with your documentation comments: [email protected].

Updated topics in this Release

  • Overview
  • Quick Start with StreamNative Cloud Manager
  • snctl reference

December 2021

StreamNative Cloud 1.11 is now available.

CLI updates

Added a new experimental command that configures pulsarctl CLI tool to connect to a given Pulsar cluster.

In release 1.11, we added a new command snctl x update-pulsar-config --cluster-name <cluster_name> -n <organization_name> that appends a specific Pulsar cluster to the Pulsar configuration file for the pulsarctl CLI tool. With this command, you can connect to the specified cluster directly without manually setting up some options, including the token, the cluster Service URL, and so on. For details, see connect to Pulsar clusters using pulsarctl.

September 2021

StreamNative Cloud 1.10 is now available.

  • Added support for enabling WebSocket service at the cluster level. The Pulsar WebSocket API provides a simple way to interact with Pulsar using languages that do not have an official client library. For details, see connect to cluster through WebSocket API.

  • Added support for customizing some cluster configurations. You can customize their clusters either by defining spec.config custom field in the cluster YAML file or specifying cluster configurations using StreamNative Cloud Manager. For details, see work with clusters.

Feature Enhancements

Three Debezium source connectors are now available in StreamNative Cloud. They provide a convenient way to move data into the cloud without writing any code.

  • Debezium MongoDB: the Debezium MongoDB source connector pulls messages from MongoDB and persists the messages to Pulsar topics.
  • Debezium MySQL: the Debezium MySQL source connector pulls messages from MySQL and persists the messages to Pulsar topics.
  • Debezium PostgreSQL: the Debezium PostgreSQL source connector pulls messages from PostgreSQL and persists the messages to Pulsar topics.

For details about how to work with these connectors using StreamNative Cloud Manager, see work with connectors.

August 2021

StreamNative Cloud 1.9 is now available.

AWS Marketplace

Added support for subscribing to StreamNative cloud service on AWS Marketplace. If you already have an AWS account, you can subscribe to the StreamNative cloud-hosted product in AWS Marketplace. You can create an organization associated with that AWS subscription. Usage for resources created under that organization, such as Pulsar cluster and Flink cluster are billed through your AWS account. For details, see here.

Feature enhancements

StreamNative Cloud Console

  • Added more fine-grained and easy-to-use configuration of Pulsar connectors. You can specify each configuration from the StreamNative Cloud console.
  • Added monitoring support for connectors. You can view the status of a connector including when a connector fails to be created, updated, or cloned. For details, see here.

snctl CLI tool

  • Added support for installing snctl CLI tools on Linux and MAC OS using Homebrew.

June 2021

StreamNative Cloud 1.8 is now available.

General changes

  • StreamNative Cloud supports Pulsar 2.8.0.
  • Connectors are now available in StreamNative Cloud. They provide a convenient way to move data in and out of the cloud without writing any code.
    • This release supports pre-approved connectors for SQS, Kinesis, AWS S3, and Lambda functions on AWS and Google Cloud cloud-hosted platforms.
    • You can configure and manage connectors using the pulsarctl CLI tool or the StreamNative Cloud Manager. For details, see here.

Bug fixes

Previous issues with Flink SQL statement queries were fixed to support more types of Flink SQL statements, including:

  • Basic (select, insert, filter, aggregate)
  • TimeSeries Aggregation (watermark, event time, session time, processing time)
  • Joins

March 2021

StreamNative Cloud introduces Flink SQL, which enables you to execute interactive SQL queries within StreamNative Cloud. The pricing has also been defined for Flink SQL on Google Cloud and AWS cloud platforms.

  • You can create a Flink Cluster through either snctl or StreamNative Cloud Manager.

  • You can interact with Flink SQL through StreamNative Cloud Manager to perform the following operations:

    • Run streaming queries on data stored in Pulsar topics using Flink SQL.
    • Explore query results.

January 2021

  • StreamNative Cloud now enables to deploy clusters on AWS cloud platform. The pricing has also been defined for newly created clusters on AWS cloud platform.

  • Support different locations for clusters deployed on the Google Cloud and AWS cloud platforms.

  • Add an Instance Setting option on the left navigation pane of the StreamNative Cloud Manager, where you can check detailed information about created instances and delete the target instance.

October 2020

  • StreamNative Cloud provides a promotion code for users who want to use the StreamNative Cloud service. Contact us to get a promotion code. For details about how to apply the promotion code, see apply promotion code.

  • Updated the layout of the Sign up page.

August 2020

This is the first official release of StreamNative Cloud. The StreamNative Cloud provides an easy-to-use and fully-managed Pulsar service in the public cloud. You can sign up on StreamNative Cloud through the StreamNative website to create and manage Pulsar clusters. After signing up for the cloud service, you can use the StreamNative Cloud CLI tool (snctl) for managing the clusters or use the StreamNative Cloud UI (StreamNative Cloud Manager) for monitoring and managing the created cluster.

This release includes the following major highlights:

Security

  • Secure clusters based on OAuth 2.0 and TLS.

Functionality

You can work with the following StreamNative Cloud resources and Pulsar components through StreamNative Cloud:

Previous
snctl Reference
Next
StreamNative Cloud Objects