Skip to main content
Source discovery turns eligible topics from a SQL Catalog cluster into managed source objects in the SQL Workspace engine.

​
How discovery works

  1. A SQL Catalog connects a ready SQL Workspace to one Pulsar or Kafka cluster.
  2. After the catalog becomes ready, StreamNative periodically checks the cluster for eligible topics.
  3. StreamNative creates a source object for each eligible topic that is missing from the catalog database.
  4. The source makes the topic’s events available to queries in the SQL Workspace engine.
Discovery is periodic rather than immediate. If an eligible source does not appear as soon as the SQL Catalog becomes ready, allow time for the next discovery cycle. A logical partitioned Pulsar topic is represented by one managed source. Its individual -partition-N topics are not created as separate sources. For the complete schema compatibility matrix, see Schema Support. For the database, schema, and source names created from these topics, see Naming Rules.

​
Process discovered sources

A managed source exposes incoming topic events. StreamNative does not automatically create tables or materialized views from that source. Create the SQL objects required for filtering, joining, aggregating, or maintaining results:
This example uses a Pulsar topic in the public/default namespace. Kafka sources use the public schema. For joins, windows, watermarks, and other processing patterns, see Data processing in RisingWave.

​
Catalog removal

Removing a SQL Catalog stops topic discovery and stops its managed sources from consuming newly published records. Data that the SQL Workspace engine already ingested or processed is retained, and the source Pulsar or Kafka cluster is unchanged. See Data Retention.