streamnative.io
Help Center
Login
  1. StreamNative Cloud
  2. Connect to External Systems
  3. Pulsar IO

Connector Overview

Introduction to Pulsar IO Connectors

Concept

Pulsar IO connectors allow you to integrate Pulsar with various external systems and data sources for both inbound and outbound data movement. For example, you can use Pulsar IO Connectors to ingest data from external sources into Pulsar topics or export data from Pulsar topics to other external systems.

Pulsar IO Connectors can be further categorized as source connectors or sink connectors:

  • A source connector ingests data from an external system into Pulsar.
  • A sink connector egresses data from Pulsar to an external system.

The following diagram illustrates the data movement among source connectors, Pulsar, sink connectors, and external systems.

Pulsar IO connector

Benefits

  • Seamless & Simplified Data Integration: Pulsar IO Connectors provide a unified interface for connecting Pulsar to various external systems and data sources, allowing developers to easily integrate Pulsar with their existing infrastructure without the need for custom integration code.

  • Extensibility: Pulsar IO Connectors are designed to be easily extensible, allowing developers to create custom connectors for specific use cases and data sources not covered by the built-in connectors.

  • Reduced Development Effort: By leveraging pre-built connectors, developers can save time and effort. They don't have to write and maintain complex integration code from scratch for each external system.

  • Reliable and Scalable: Pulsar IO Connectors are built to be reliable and scalable, ensuring the data transfers between Pulsar and external systems are efficient and fault-tolerant.

Use cases

  • Data Ingestion to Pulsar: If you have data coming from various external sources, such as Apache Kafka, databases, message queues, or cloud storage systems like Amazon S3, you can use Pulsar IO Connectors to ingest that data into Pulsar topics.

  • Data Export from Pulsar: Pulsar IO Connectors also enable you to export data from Pulsar topics to other systems or storage solutions. This helps you to synchronize data across different environments, replicate data, or stream data to external services or databases.

  • Real-time Data Processing: Pulsar IO Connectors facilitate real-time data processing by enabling the seamless movement of data between Pulsar and other systems. This is particularly useful in event-driven architectures, streaming applications, and microservices-based solutions.

  • Extending Pulsar's Functionality: If you have specific use cases or data sources not directly supported by Pulsar, you can deploy custom Pulsar IO Connectors to extend Pulsar's functionality and integrate with those systems.

Connectors Shared Responsibility

StreamNative and our customers have a shared responsibility for maintaining and keeping connectors properly functioning. Outlined below, StreamNative has a responsibility for custom connectors that are maintained by uploading to StreamNative Cloud to maintain connectivity to the Pulsar cluster and for logging and monitoring. Customers who upload their connectors are responsible for all other operations such as configurations, updates, development, support and plugin installation. Partner connectors have similar support but the partners with StreamNative will be responsible for development and connector support. For built in connectors, StreamNative is responsible for everything except connector configuration.

For more details, see below.

Connectors Shared Responsibility Model

Built-in connectors on StreamNative Cloud

To further reduce the development overhead and time, StreamNative has pre-built a variety of Pulsar IO connectors on StreamNative Cloud. With proper configurations, you can integrate the data between your Pulsar cluster on StreamNative Cloud and your data systems effortlessly.

Built-in source connectors

Currently, StreamNative Cloud supports the following source connectors:

Built-in sink connectors

Currently, StreamNative Cloud supports the following sink connectors.

What’s next?

Previous
Overview
Next
Deploy Connectors