Skip to main content
Source discovery turns eligible topics from a SQLCatalog cluster into managed source objects in the SQLWorkspace engine.

​
How discovery works

  1. A SQLCatalog connects a ready SQLWorkspace to one Pulsar or Kafka cluster.
  2. After the catalog becomes ready, StreamNative periodically checks the cluster for eligible topics.
  3. StreamNative creates a source object for each eligible topic that is missing from the catalog database.
  4. The source makes the topic’s events available to queries in the SQLWorkspace engine.
Discovery is periodic rather than immediate. If an eligible source does not appear as soon as the SQLCatalog becomes ready, allow time for the next discovery cycle. A logical partitioned Pulsar topic is represented by one managed source. Its individual -partition-N topics are not created as separate sources. For the complete schema compatibility matrix, see Schema Support. For the database, schema, and source names created from these topics, see Naming Rules.

​
Process discovered sources

A managed source exposes incoming topic events. StreamNative does not automatically create tables or materialized views from that source. Create the SQL objects required for filtering, joining, aggregating, or maintaining results:
This example uses a Pulsar topic in the public/default namespace. Kafka sources use the public schema. For joins, windows, watermarks, and other processing patterns, see RisingWave.

​
Catalog removal

Removing a SQLCatalog stops topic discovery and stops its managed sources from consuming newly published records. Data that the SQLWorkspace engine already ingested or processed is retained, and the source Pulsar or Kafka cluster is unchanged. See Data Retention.