streamnative.io
Help Center
Login
  1. StreamNative Managed Cloud
  2. Connect to External Systems

Connector Overview

Introduction to Pulsar IO Connectors

Concept

Pulsar IO connectors allow you to integrate Pulsar with various external systems and data sources for both inbound and outbound data movement. For example, you can use Pulsar IO Connectors to ingest data from external sources into Pulsar topics or export data from Pulsar topics to other external systems.

Pulsar IO Connectors can be further categorized as source connectors or sink connectors:

  • A source connector ingests data from an external system into Pulsar.
  • A sink connector egresses data from Pulsar to an external system.

The following diagram illustrates the data movement among source connectors, Pulsar, sink connectors, and external systems.

Pulsar IO connector

Benefits

  • Seamless & Simplified Data Integration: Pulsar IO Connectors provide a unified interface for connecting Pulsar to various external systems and data sources, allowing developers to easily integrate Pulsar with their existing infrastructure without the need for custom integration code.

  • Extensibility: Pulsar IO Connectors are designed to be easily extensible, allowing developers to create custom connectors for specific use cases and data sources not covered by the built-in connectors.

  • Reduced Development Effort: By leveraging pre-built connectors, developers can save time and effort. They don't have to write and maintain complex integration code from scratch for each external system.

  • Reliable and Scalable: Pulsar IO Connectors are built to be reliable and scalable, ensuring the data transfers between Pulsar and external systems are efficient and fault-tolerant.

Use cases

  • Data Ingestion to Pulsar: If you have data coming from various external sources, such as Apache Kafka, databases, message queues, or cloud storage systems like Amazon S3, you can use Pulsar IO Connectors to ingest that data into Pulsar topics.

  • Data Export from Pulsar: Pulsar IO Connectors also enable you to export data from Pulsar topics to other systems or storage solutions. This helps you to synchronize data across different environments, replicate data, or stream data to external services or databases.

  • Real-time Data Processing: Pulsar IO Connectors facilitate real-time data processing by enabling the seamless movement of data between Pulsar and other systems. This is particularly useful in event-driven architectures, streaming applications, and microservices-based solutions.

  • Extending Pulsar's Functionality: If you have specific use cases or data sources not directly supported by Pulsar, you can deploy custom Pulsar IO Connectors to extend Pulsar's functionality and integrate with those systems.

Built-in connectors on StreamNative Cloud

To further reduce the development overhead and time, StreamNative has pre-built a variety of Pulsar IO connectors on StreamNative Cloud. With proper configurations, you can integrate the data between your Pulsar cluster on StreamNative Cloud and your data systems effortlessly.

Built-in source connectors

Currently, StreamNative Cloud supports the following source connectors:

Built-in sink connectors

Currently, StreamNative Cloud supports the following sink connectors.

What’s next?

Previous
Connect to External Systems
Next
Deploy Connectors