SQL Workspaces Architecture (Private Preview)

SQL Workspaces provide a managed architecture for processing streams from StreamNative Pulsar and Kafka clusters with PostgreSQL-compatible SQL.

SQL Workspaces are available by invitation only. During Private Preview, manage SQLWorkspace and SQLCatalog resources through the StreamNative Cloud Console. You can query a workspace through SQL Studio or its native PostgreSQL endpoint.

Get started

Create a workspace, add a catalog, and run a query.

Understand access control

Learn how cloud resource and SQLWorkspace engine permissions work.

Understand SQLWorkspace

Learn about the engine model, placement, lifecycle, and query endpoints.

Understand SQLCatalog

Learn how Pulsar and Kafka clusters map to databases and topic sources.

Architecture

StreamNative Cloud control plane
+----------------------------------------------------------------+
| Cloud Console                                                  |
|   |                                                            |
|   +-- manages --> SQLWorkspace (engine and placement)          |
|   +-- manages --> SQLCatalog(s) (cluster connections)          |
+-------------------------------+--------------------------------+
                                | reconciles
                                v
SQLWorkspace data plane
+----------------------------------------------------------------+
| SQLWorkspace engine                                            |
|   RisingWave -- metadata --> Amazon RDS for PostgreSQL         |
|              +-- state ----> Amazon S3                         |
|                                                                |
| Pulsar/Kafka -- periodic discovery --> SQLCatalog database(s)  |
|                                             |                  |
|                                             +--> topic sources |
|                                                                |
| SQL Studio -------- WebSocket query path ----+                 |
| PostgreSQL clients -- native TLS/pgwire -----+--> engine       |
+----------------------------------------------------------------+

Components

Component	Responsibility
SQLWorkspace	Defines the placement and lifecycle of a managed SQLWorkspace engine and its query endpoints.
SQLWorkspace engine	Executes SQL and maintains streaming computations. The engine model is pluggable, but RisingWave is the only supported engine during Private Preview. On AWS, the engine uses managed Amazon RDS for PostgreSQL for metadata and Amazon S3 for durable state.
SQLCatalog	Connects one supported StreamNative cluster to a workspace and represents it as a database in the SQLWorkspace engine. During Private Preview, SQLCatalog supports only Pulsar and Kafka clusters.

Control flow

A user creates a SQLWorkspace in an eligible hosted location or ready BYOC Cloud Environment through the Cloud Console.
StreamNative provisions the selected SQLWorkspace engine and its managed metadata and object storage. During Private Preview, the engine is RisingWave and SQLWorkspace engine infrastructure is supported only on AWS.
The user adds a Pulsar or Kafka cluster as a SQLCatalog.
StreamNative creates a database for the catalog and periodically discovers eligible topics to create missing source objects.
The user creates tables or materialized views based on the managed topic sources for subsequent processing.

Query flow

Pulsar or Kafka delivers new events to the connected topics.
The topic sources make those events available to the SQLWorkspace engine.
The SQLWorkspace engine evaluates ad hoc queries or continuously maintains materialized views.
Users submit SQL and retrieve results through SQL Studio or a PostgreSQL-compatible client.

Overview

Engine

Get Started

Operations

Access Control

SQL Workspaces Architecture (Private Preview)

Get started

Understand access control

Understand SQLWorkspace

Understand SQLCatalog

Architecture

Components

Control flow

Query flow

Get started

Understand access control

Understand SQLWorkspace

Understand SQLCatalog

​Architecture

​Components

​Control flow

​Query flow

Architecture

Components

Control flow

Query flow