Connector Overview
Introduction to Pulsar IO Connectors
Concept
Pulsar IO connectors allow you to integrate Pulsar with various external systems and data sources for both inbound and outbound data movement. For example, you can use Pulsar IO Connectors to ingest data from external sources into Pulsar topics or export data from Pulsar topics to other external systems.
Pulsar IO Connectors can be further categorized as source connectors or sink connectors:
- A source connector ingests data from an external system into Pulsar.
- A sink connector egresses data from Pulsar to an external system.
The following diagram illustrates the data movement among source connectors, Pulsar, sink connectors, and external systems.
Benefits
Seamless & Simplified Data Integration: Pulsar IO Connectors provide a unified interface for connecting Pulsar to various external systems and data sources, allowing developers to easily integrate Pulsar with their existing infrastructure without the need for custom integration code.
Extensibility: Pulsar IO Connectors are designed to be easily extensible, allowing developers to create custom connectors for specific use cases and data sources not covered by the built-in connectors.
Reduced Development Effort: By leveraging pre-built connectors, developers can save time and effort. They don't have to write and maintain complex integration code from scratch for each external system.
Reliable and Scalable: Pulsar IO Connectors are built to be reliable and scalable, ensuring the data transfers between Pulsar and external systems are efficient and fault-tolerant.
Use cases
Data Ingestion to Pulsar: If you have data coming from various external sources, such as Apache Kafka, databases, message queues, or cloud storage systems like Amazon S3, you can use Pulsar IO Connectors to ingest that data into Pulsar topics.
Data Export from Pulsar: Pulsar IO Connectors also enable you to export data from Pulsar topics to other systems or storage solutions. This helps you to synchronize data across different environments, replicate data, or stream data to external services or databases.
Real-time Data Processing: Pulsar IO Connectors facilitate real-time data processing by enabling the seamless movement of data between Pulsar and other systems. This is particularly useful in event-driven architectures, streaming applications, and microservices-based solutions.
Extending Pulsar's Functionality: If you have specific use cases or data sources not directly supported by Pulsar, you can deploy custom Pulsar IO Connectors to extend Pulsar's functionality and integrate with those systems.
Connectors Shared Responsibility
StreamNative and our customers have a shared responsibility for maintaining and keeping connectors properly functioning. Outlined below, StreamNative has a responsibility for custom connectors that are maintained by uploading to StreamNative Cloud to maintain connectivity to the Pulsar cluster and for logging and monitoring. Customers who upload their connectors are responsible for all other operations such as configurations, updates, development, support and plugin installation. Partner connectors have similar support but the partners with StreamNative will be responsible for development and connector support. For built in connectors, StreamNative is responsible for everything except connector configuration.
For more details, see below.
Built-in connectors on StreamNative Cloud
To further reduce the development overhead and time, StreamNative has pre-built a variety of Pulsar IO connectors on StreamNative Cloud. With proper configurations, you can integrate the data between your Pulsar cluster on StreamNative Cloud and your data systems effortlessly.
Built-in source connectors
Currently, StreamNative Cloud supports the following source connectors:
- AMQP1_0 source connector
- AWS SQS source connector
- Google BigQuery source connector
- Debezium Microsoft SQL Server source connector
- Debezium MongoDB source connector
- Debezium MySQL source connector
- Debezium PostgreSQL source connector
- Kafka source connector
- Kinesis source connector
- Data generator source connector (for internal testing purposes only)
Built-in sink connectors
Currently, StreamNative Cloud supports the following sink connectors.
- AMQP1_0 sink connector
- AWS EventBridge sink connector
- AWS Lambda sink connector
- AWS SQS sink connector
- AWS S3 Sink Connector
- Google Cloud Storage Sink Connector
- Azure Blob Storage Sink Connector
- Elasticsearch sink connector
- Google BigQuery sink connector
- Kinesis sink connector
- Snowflake sink connector
- Data generator sink connector (for internal testing purposes only)