Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.streamnative.io/llms.txt

Use this file to discover all available pages before exploring further.

StreamNative Lakehouse Tables StreamNative Lakehouse Tables provide a unified way to expose streaming topics as open table format objects—such as Apache Iceberg and Delta Lake—directly within StreamNative Cloud. With Lakehouse Tables, data produced to Pulsar or Kafka-compatible topics can be automatically stored in object storage in a transactional, analytics-ready table format. This enables seamless integration between real-time data streams and downstream analytics, AI/ML, and governance systems. Overview StreamNative Lakehouse Tables bridge the gap between streaming and batch systems by converting message data from topics into table-backed datasets. Each Lakehouse Table maintains:
  • Metadata (schema, manifest lists, snapshots)
  • Data files (Parquet/columnar format)
  • Transaction logs (for table evolution)
These components reside in user-controlled object storage, ensuring low cost, high durability, and interoperability with a broad ecosystem of tools. Key Capabilities 1. Open Table Format Support StreamNative Lakehouse Tables support industry-standard formats:
  • Apache Iceberg
  • Delta Lake (Delta 2.0 and above)
This ensures compatibility with engines such as Databricks, Snowflake, Spark, Trino, Flink, BigQuery, StarTree, and more. 2. Native Topic-to-Table Mapping Each table is backed by one or more StreamNative topics. Data is automatically:
  • Ingested from the streaming topic
  • Serialized into Parquet files
  • Committed into the table as immutable snapshots
  • Made available for SQL queries and analytical engines
This provides a streaming-first lakehouse architecture without external ingestion pipelines. 3. Object Storage as the Source of Truth All table artifacts are stored in object storage such as:
  • Amazon S3
  • Google Cloud Storage
  • Azure Blob Storage
This enables low-cost storage, independent scaling, and easy interoperability. 4. Schema-Aware and Schema-Safe Lakehouse Tables use StreamNative’s schema registry to ensure:
  • Schema inference from topics
  • Backward/forward compatible evolution
  • Safe writes with schema enforcement
  • Automatic mapping to Iceberg/Delta schemas
  • Users retain full control over table evolution policies.
5. Transactional Guarantees Using open table format guarantees, Lakehouse Tables support:
  • ACID transactions
  • Snapshot isolation
  • Time travel (via historical snapshots)
  • Incremental reads
This brings reliability and consistency to streaming data workflows.
  1. Full Interoperability with Data and AI Platforms
Once a table is materialized in Iceberg or Delta Lake format, it is fully queryable by:
  • Databricks
  • Snowflake
  • BigQuery Managed Tables
  • Apache Spark, Flink, and Trino
  • StarTree and Pinot
  • DuckDB
  • pandas & PyArrow
No connectors or intermediate ETL is required.