Documentation Site Refresh
StreamNative’s documentation site has been redesigned. There are now two separate sub-sites for StreamNative Managed Cloud and StreamNative Platform, and each section is organized by the type of work that you are trying to do:
- Build Applications: This section is for Pulsar users within companies that are using Pulsar to build applications, such as application development teams. (They do not need to know anything about managing or architecting/administering Pulsar.)
- Administer: This section is for Pulsar power users and core platform teams that manage Pulsar components. This section includes information on creating tenants, namespaces, and topics, as well as managing organizations and instances.
- Manage Infrastructure: This section is primarily for customers who are managing clusters and infrastructure. (That is, SRE folks - cluster, metrics, logs.)
Show memory and CPU when configuring Brokers and Bookies
The Advanced tab for configuring a Pulsar cluster now surfaces both memory and CPU for Brokers and Bookies, making it easier to understand your cluster’s compute and storage capabilities.
Pulsar version 22.214.171.124 for newly deployed Hosted clusters
Updates from version 126.96.36.199 include the following:
- Upgrade Pulsar server Java build version to 17.
- Support chunking with shared subscriptions.
- Support the pluggable topic factory.
- Support configuring the compression type for
ManagedCursorInfoto balance storage resource usage.
To learn more about updates included in this version, see What's New in Apache Pulsar 2.11.
Pulsar Admin Go version 0.1.1
Pulsar Admin Go library v0.1.1 has been released, and includes the following updates:
- Add two new parameters (
BrokerClientTlsEnabled) to support TLS when creating a cluster. - Add
GetMessageIDto the API, which returns the message ID by timestamp(ms) of a topic.
To learn more about Pulsar Admin Go, check out the blog post here.
Functions Usage Chart
Fixed a bug that caused the same metrics to show up for multiple Pulsar clusters within an instance.
New Connector: AWS EventBridge Sink Connector
- The StreamNative team has built a new connector for open source Pulsar - the AWS EventBridge Sink Connector! With it, you can pull data from a Pulsar cluster and persist it to AWS EventBridge, making it easier to build scalable event-driven applications. To learn more, check out our recent blog post.
- Additionally, the AWS Eventbridge Sink Connector is also available on StreamNative Cloud! For a detailed setup tutorial, check out our blog post announcing the integration. If you would like to use it, please reach out to the StreamNative Support Team.
The StreamNative Hub has a new design and is quickly accessible from the StreamNative Docs site, making it easier to find connectors and understand which connectors are supported on StreamNative Cloud.
Pulsar 3.0 available on StreamNative Cloud
Pulsar 3.0, the next evolution of Pulsar, is now available on StreamNative Cloud. Pulsar 3.0 allows teams to run even bigger workloads, and is easier for developers to work with on their local machines. Updates include LTS support, an improved load balancer, several performance improvements, and Docker images for M1/M2 Macs. You can try Pulsar 3.0 on a new cluster or reach out to the StreamNative Support Team about upgrading existing clusters.
Additional Version Details available on StreamNative Cloud Console
Pulsar, BookKeeper, and ZooKeeper versions are now available on the cluster details page, giving you greater insight into your cluster’s configuration details.
Improved experience on the Organization Usage page
It is now easier to understand how many resources your clusters consume. You can view usage metrics (such as compute, storage and data) for a specific instance and then compare it to the organization’s usage across all instances.
AWS Marketplace listing update
New Setup Guide. A new setup guide has been added to the Dashboard page and contains everything that you need to get up and running with a hosted cluster. On the Dashboard page, you will see step-by-step instructions on adding a payment method, creating and configuring Pulsar clusters, creating topics, setting up clients, and sending messages. The setup guide can be viewed by logging in to the StreamNative Cloud Console and clicking Dashboard on the left navigation pane.
Organization Usage Page. Customers with StreamNative Managed Clusters (Hosted + BYOC) can now view both real-time and historical cluster usage data on the new Organization Usage page. This page features charts visualizing the usage dimensions (Compute Units, Storage Units, Throughput, and Storage Size) for all clusters within an organization. You can view usage by instance or in aggregate across your organization and narrow the date range to hone in on usage over a specific time period. You can view up to the past 60 days of usage.
Improved Usability for Kafka Clients. Customers with clusters that have Kafka protocol enabled will include a new Kafka Clients page that walks you through using Kafka clients to connect to your clusters. The Kafka client setup wizard enables application developers using Kafka clients to easily connect to their organization's Pulsar cluster. This page includes client libraries (Java, Go, Python, and Node.js), Apache Kafka CLI tools, KStream applications, and KSQL. In addition, the Kafka protocol is also enabled by default on all new managed Pulsar clusters (Hosted + BYOC).
Refreshed Dashboard for managing instances and user accounts on StreamNative Cloud Console. You will see two new cards on the Dashboard page, surfacing additional information about your organization's instances and user accounts, allowing you to manage instances and user accounts directly from the Dashboard page without going to a separate screen.
New hosted clusters on Pulsar 188.8.131.52. New hosted Pulsar clusters (clusters hosted on StreamNative's public cloud account) will now be on version 184.108.40.206, where they were previously on version 220.127.116.11. As a result, you will benefit from the following enhancements:
- Pulsar provides automatic failure recovery between the primary and backup clusters.
- With the lazy-loading feature added to
PartitionedProducer, fewer producers are needed and the broker memory utilization is improved.
- Topic map support added with a new
TableViewtype using key values in received messages.
Audit log now includes produce and consume events. For clusters on Pulsar 18.104.22.168 and later, Produce and Consume events can now be tracked in the audit log.
Streamlined instance deletion experience. You can now delete an instance and all Pulsar clusters within it in one step, rather than having to individually delete all associated clusters before deleting the instance.
Snowflake Sink Connector upgrade. The Snowflake Sink Connector on StreamNative Cloud Console has been upgraded from 22.214.171.124 to 126.96.36.199. In addition to smaller bug fixes, this connector now supports concurrency where it previously did not, meaning the replicas can now be set to more than 1 when configuring a connector.
- Fixed an issue preventing the creation of a Debezium-Postgres connector via the StreamNative Cloud Console.
- Fixed an issue where namespaces with lots of topics would sometimes show a spurious error on the Namespace page of the StreamNative Cloud Console.
- Fixed an issue causing configuration details on the Functions page to display without formatting.
March + April 2023
New Clients Page. The Clients page makes it easier to connect to a Pulsar cluster with your client of choice. You'll find comprehensive example snippets with better syntax highlighting, more programming languages (Java, Go, Python, C++, Node.js, Rust, Spring, C#), and more REST API and CLI tool examples.
You can download authentication tokens (either an OAuth2 credential file or a JWT token) directly from the Clients page.
To view the new page, select a programming language, CLI tool, or REST API. Then, you can see installation and connection instructions. Watch the video below for an example of connecting to a cluster using the Java client.
Pulsar Transactions on StreamNative Cloud. You can now configure Pulsar Transactions using the StreamNative Cloud Console.
Pulsar Transactions (in Beta on StreamNative Cloud) enable event-streaming applications to consume, process, and produce messages in one atomic operation, including:
- Atomic writes across multiple topic partitions
- Atomic acknowledgments across multiple topic partitions
- All the operations made within one transaction either all succeed, or all fail
- Consumers are ONLY allowed to read committed messages
Navigation bar refresh. The new design for the navigation bar on the StreamNative Cloud Console includes a lighter theme and updated icons, making it easier to navigate the console. To colocate all Admin tasks, the Clients tab was moved to the Admin section.
Pulsar cluster names. With this release, the maximum number of characters for Pulsar cluster names is 10 characters. (Note: Existing Pulsar clusters with names longer than 10 characters are not affected by this change.)
Request a region. If you are creating a new cluster using StreamNative Cloud Console and a region is not currently supported, you can request support for that region.
Previously, you could not disable the audit log once it was turned on (enabled). With this release, it is now possible to toggle the audit log on and off.
- Deprecation of free cloud. New customers can no longer sign up for free clusters. If you are interested in requesting access to StreamNative Cloud, please Contact us.
New provisioning changes for hosted clusters
We have improved the customer experience for self-service provisioning, making it easier for you to provision StreamNative hosted clusters.
- You can provision new clusters through StreamNative Cloud console, with a prompt to update payment information.
- New changes will not impact existing clusters. To adjust the sizing of existing clusters, please submit a ticket, and our team will assist with this one-time migration.
Audit Log allows you to monitor which user accounts and applications are accessing data, and what they do with it. Audit Log supports various audit event types, including creating, updating, and deleting clusters, tenants, namespaces, and topics.
New connectors are available on StreamNative Cloud:
- Debezium / Microsoft SQL Server Source - Pulls messages from the SQL Server and persists the messages to Pulsar topics.
- Google BigQuery Source - Feeds data from Google Cloud BigQuery tables and writes data to Pulsar topics.
- Google BigQuery Sink - Pulls data from Pulsar topics and persists data to Google Cloud.
- Snowflake Sink (available through the CLI) - Loads data from Pulsar topics to Snowflake in real time.
If you are interested in using these connectors, submit a ticket with this request to the support team.
Cluster Metrics in Beta
StreamNative Cloud offers a new endpoint and dashboard for Cluster Metrics to help provide visibility into your Pulsar clusters and track performance over time. Collect and monitor cluster health and performance metrics in real time, analyze trends, and proactively maintain your applications to ensure optimal performance.
Pulsar Functions on Cloud in Beta
Pulsar Functions are now available on StreamNative Cloud. Pulsar Functions on StreamNative Cloud enable you to build real-time data pipelines for ETL jobs, event-driven applications, and simple data analytics applications.
Check out this demo of Pulsar Functions in action:
New REST API for StreamNative Cloud allows the following:
- Produce, consume, and acknowledge messages
- Monitor the state of your Pulsar clusters
- Perform a wide range of administrative actions
Check out this demo of the StreamNative Rest API in action:
Improvements for new clusters
We have made several improvements to new Pulsar clusters on StreamNative Cloud. These changes will not impact existing clusters, but new clusters will benefit from the following enhancements:
- All new clusters are based on Pulsar v188.8.131.52.
Cluster configurations added to the cluster Overview tab
You can view a new configuration section on the Overview tab on the Clusters page in the StreamNative Cloud Console for better visibility and accessibility. This allows you to quickly access and manage information on the backlog quota, backlog retention policy, subscription throttling rate, and producer/consumer limits per topic.
Cloud Console optimizations
To facilitate easier debugging and troubleshooting of data issues with a topic, a new Stats tab has been added to the Topic page in StreamNative Cloud. This tab displays the output of the commands
topic-stats-internal, making it easier for you to access and analyze topic statistics, streamlining the debugging process.
Instance selection made clearer
The StreamNative Cloud Console dashboard now clearly indicates the currently selected instance through the addition of a checkmark. This enhancement eliminates confusion and allows you to easily identify the active instance.
In January 2023, the StreamNative Cloud Documentation Team rolled out the first version of a new documentation experience. We implemented several new features and sections for the documentation site, including the following:
- A card-style landing page.
- Enhanced navigation with breadcrumbs and back/next links on every page.
- A new search engine for faster, more accurate search results.
- New About section for Apache Pulsar.
- New Coming Soon section that lists upcoming and current beta features. For details, see the Coming Soon overview.
This update includes changes to the StreamNative Hosted Cloud documentation.
The procedure for creating a StreamNative Cloud account through the AWS Marketplace was updated and the help topic now reflects the correct workflow. For details, see the AWS billing page.
The Google BigQuery sink was added as a new connector and the documentation page was updated accordingly. For details, see the work with sink connectors page.
The name StreamNative Cloud Manager was updated to StreamNative Cloud Console throughout the documentation site. This change was made to be consistent with the app.
This update includes enhancements and updates to the StreamNative Cloud Manager interface.
In StreamNative Cloud Manager Release 2.2.2, a new sign in flow was added for first-time users. For details, see the quick start.
The Flink SQL UI option has been deprecated. However, you can still work with Flink SQL through the CLI.
A new source connector, Debezium Microsoft SQL Server, was added. For details, see work with source connectors.
This update reflects several important changes to the onboarding flow for StreamNative Cloud Manager including support for Single Sign-On (SSO), automatic organization deployment, and more choices in cloud regions for clusters.
- For StreamNative Cloud Manager Release 2.1.7, support for SSO was added to the login process. For details, see the Quick start.
- A default organization is now created for you when you sign up for StreamNative Cloud, saving you time in the set up process. For details, see Work with organizations.
- Clusters include more choices in cloud provider regions. For details, see Work with clusters.
StreamNative Cloud is designed to make it easy for teams to leverage the power of Apache Pulsar. We are working hard on improving the StreamNative Cloud Console to make it simpler for teams to provision and understand their Pulsar clusters.
Today, we are pleased to announce the release of an updated Cloud Console that brings some new features and refinements to better meet your needs. This release features a new Cloud Dashboard and Navigation to provide better visibility into your clusters and to provide more intuitive navigation. This is the first in many releases, and over the coming weeks and months. Expect to see more improvements that add additional features and refinements in the future.
If you have any questions or issues about this update, don't hesitate to reach out through the support portal or give us feedback through the chat app.
- New Dashboard page that serves as a quick health check of your instances with your most important tasks close at hand.
- Updated navigation menu to simplify and streamline the management of StreamNative and Pulsar resources.
- Significant internal re-architecture to make the UI faster and more responsive, without some of the previous delays of API changes not being shown in the UI.
- Cloud Console Dashboard and login changes
- This new home page is designed to make it easy for you to see the status of your instances and to quickly get started with StreamNative Cloud.
- The Cloud Console now remembers your most recent organization and instance.
- New Navigation
- The navigation menu has been significantly changed to make navigation more intuitive.
- A new quick switcher for tenants and namespaces has been added to the navigation bar.
- Update your bookmarks! The links have changed. The previous links did not allow for deep linking.
- Certain metrics, such as number of topics in a namespace, traffic, and others have been removed. These metrics were a point-in-time snapshot that was often out-of-date, potentially leading to confusion.
- Numerous bug fixes and performance improvements have been implemented. These bug fixes include issues with changes via the Pulsar REST API not being visible within the UI and issues with large numbers of topics or namespaces.
This update is just the beginning. More features are coming soon, including:
- Additional dashboard cards with contextual actions to simplify managing your Pulsar infrastructure.
- Richer, historical metrics to see how your instance and applications are performing.
- Additional improvements to the provisioning workflow to make it easier to deploy and manage instances and clusters.
- Several topics were updated to reflect the updated StreamNative Cloud Manager UI. This was a major update to the UI, reflecting StreamNative's commitment to continuous improvement.
- The Node.js client topic was updated to reflect support for OAuth2 authentication. For details, see connect to Pulsar clusters through the Node.js client.
- The AWS Lambda sink connector is onboarding to StreamNative Cloud. For details, see work with sink connectors.
This release covers several documentation updates.
- A new topic on StreamNative Cloud Secrets was added to the Managed Access section. StreamNative Cloud Secrets allow you to store and manage sensitive data such as passwords, tokens, and private keys. You can create Secrets and refer to them for computing purposes (Pulsar connectors, Pulsar Functions, Flink SQL and general Flink jobs). For details, see work with Secrets.
- Per-topic throughput limit and storage limit for clusters was updated in the Clusters topic.
- Billing information was removed from the Billing topic, as up-to-date billing information is on the StreamNative website.
- The snctl reference topic was updated with an improved order of information.
Going forward, you will continue to see changes and improvements to the documentation site, in addition to keeping up with product changes. Updates will include the following:
- New layout and formatting
- Clear, consistent language for faster reading
- Improved cross-referencing for easier navigation
- Streamlined procedures
- Updated organization
- New sections for specific use cases
We welcome your feedback! Email us with your documentation comments: [email protected].
Updated topics in this Release
- Quick Start with StreamNative Cloud Manager
- snctl reference
StreamNative Cloud 1.11 is now available.
Added a new experimental command that configures
pulsarctl CLI tool to connect to a given Pulsar cluster.
In release 1.11, we added a new command
snctl x update-pulsar-config --cluster-name <cluster_name> -n <organization_name> that appends a specific Pulsar cluster to the Pulsar configuration file for the
pulsarctl CLI tool. With this command, you can connect to the specified cluster directly without manually setting up some options, including the token, the cluster Service URL, and so on. For details, see connect to Pulsar clusters using pulsarctl.
StreamNative Cloud 1.10 is now available.
Added support for enabling WebSocket service at the cluster level. The Pulsar WebSocket API provides a simple way to interact with Pulsar using languages that do not have an official client library. For details, see connect to cluster through WebSocket API.
Added support for customizing some cluster configurations. You can customize their clusters either by defining
spec.config customfield in the cluster YAML file or specifying cluster configurations using StreamNative Cloud Manager. For details, see work with clusters.
Three Debezium source connectors are now available in StreamNative Cloud. They provide a convenient way to move data into the cloud without writing any code.
- Debezium MongoDB: the Debezium MongoDB source connector pulls messages from MongoDB and persists the messages to Pulsar topics.
- Debezium MySQL: the Debezium MySQL source connector pulls messages from MySQL and persists the messages to Pulsar topics.
- Debezium PostgreSQL: the Debezium PostgreSQL source connector pulls messages from PostgreSQL and persists the messages to Pulsar topics.
For details about how to work with these connectors using StreamNative Cloud Manager, see work with connectors.
StreamNative Cloud 1.9 is now available.
Added support for subscribing to StreamNative cloud service on AWS Marketplace. If you already have an AWS account, you can subscribe to the StreamNative cloud-hosted product in AWS Marketplace. You can create an organization associated with that AWS subscription. Usage for resources created under that organization, such as Pulsar cluster and Flink cluster are billed through your AWS account. For details, see here.
StreamNative Cloud Console
- Added more fine-grained and easy-to-use configuration of Pulsar connectors. You can specify each configuration from the StreamNative Cloud console.
- Added monitoring support for connectors. You can view the status of a connector including when a connector fails to be created, updated, or cloned. For details, see here.
snctl CLI tool
- Added support for installing snctl CLI tools on Linux and MAC OS using Homebrew.
StreamNative Cloud 1.8 is now available.
- StreamNative Cloud supports Pulsar 2.8.0.
- Connectors are now available in StreamNative Cloud. They provide a convenient way to move data in and out of the cloud without writing any code.
- This release supports pre-approved connectors for SQS, Kinesis, AWS S3, and Lambda functions on AWS and Google Cloud cloud-hosted platforms.
- You can configure and manage connectors using the pulsarctl CLI tool or the StreamNative Cloud Manager. For details, see here.
Previous issues with Flink SQL statement queries were fixed to support more types of Flink SQL statements, including:
- Basic (select, insert, filter, aggregate)
- TimeSeries Aggregation (watermark, event time, session time, processing time)
StreamNative Cloud introduces Flink SQL, which enables you to execute interactive SQL queries within StreamNative Cloud. The pricing has also been defined for Flink SQL on Google Cloud and AWS cloud platforms.
You can create a Flink Cluster through either snctl or StreamNative Cloud Manager.
You can interact with Flink SQL through StreamNative Cloud Manager to perform the following operations:
- Run streaming queries on data stored in Pulsar topics using Flink SQL.
- Explore query results.
StreamNative Cloud now enables to deploy clusters on AWS cloud platform. The pricing has also been defined for newly created clusters on AWS cloud platform.
Support different locations for clusters deployed on the Google Cloud and AWS cloud platforms.
Add an Instance Setting option on the left navigation pane of the StreamNative Cloud Manager, where you can check detailed information about created instances and delete the target instance.
StreamNative Cloud provides a promotion code for users who want to use the StreamNative Cloud service. Contact us to get a promotion code. For details about how to apply the promotion code, see apply promotion code.
Updated the layout of the Sign up page.
This is the first official release of StreamNative Cloud. The StreamNative Cloud provides an easy-to-use and fully-managed Pulsar service in the public cloud. You can sign up on StreamNative Cloud through the StreamNative website to create and manage Pulsar clusters. After signing up for the cloud service, you can use the StreamNative Cloud CLI tool (snctl) for managing the clusters or use the StreamNative Cloud UI (StreamNative Cloud Manager) for monitoring and managing the created cluster.
This release includes the following major highlights:
- Secure clusters based on OAuth 2.0 and TLS.
You can work with the following StreamNative Cloud resources and Pulsar components through StreamNative Cloud: