Introduction to Pulsar IO Connectors
Pulsar IO connectors allow you to integrate Pulsar with various external systems and data sources for both inbound and outbound data movement. For example, you can use Pulsar IO Connectors to ingest data from external sources into Pulsar topics or export data from Pulsar topics to other external systems.
Pulsar IO Connectors can be further categorized as source connectors or sink connectors:
- A source connector ingests data from an external system into Pulsar.
- A sink connector egresses data from Pulsar to an external system.
The following diagram illustrates the data movement among source connectors, Pulsar, sink connectors, and external systems.
Seamless & Simplified Data Integration: Pulsar IO Connectors provide a unified interface for connecting Pulsar to various external systems and data sources, allowing developers to easily integrate Pulsar with their existing infrastructure without the need for custom integration code.
Extensibility: Pulsar IO Connectors are designed to be easily extensible, allowing developers to create custom connectors for specific use cases and data sources not covered by the built-in connectors.
Reduced Development Effort: By leveraging pre-built connectors, developers can save time and effort. They don't have to write and maintain complex integration code from scratch for each external system.
Reliable and Scalable: Pulsar IO Connectors are built to be reliable and scalable, ensuring the data transfers between Pulsar and external systems are efficient and fault-tolerant.
Data Ingestion to Pulsar: If you have data coming from various external sources, such as Apache Kafka, databases, message queues, or cloud storage systems like Amazon S3, you can use Pulsar IO Connectors to ingest that data into Pulsar topics.
Data Export from Pulsar: Pulsar IO Connectors also enable you to export data from Pulsar topics to other systems or storage solutions. This helps you to synchronize data across different environments, replicate data, or stream data to external services or databases.
Real-time Data Processing: Pulsar IO Connectors facilitate real-time data processing by enabling the seamless movement of data between Pulsar and other systems. This is particularly useful in event-driven architectures, streaming applications, and microservices-based solutions.
Extending Pulsar's Functionality: If you have specific use cases or data sources not directly supported by Pulsar, you can deploy custom Pulsar IO Connectors to extend Pulsar's functionality and integrate with those systems.
Built-in connectors on StreamNative Cloud
To further reduce the development overhead and time, StreamNative has pre-built a variety of Pulsar IO connectors on StreamNative Cloud. With proper configurations, you can integrate the data between your Pulsar cluster on StreamNative Cloud and your data systems effortlessly.
Built-in source connectors
Currently, StreamNative Cloud supports the following source connectors:
- AMQP1_0 source connector
- AWS SQS source connector
- Google BigQuery source connector
- Debezium Microsoft SQL Server source connector
- Debezium MongoDB source connector
- Debezium MySQL source connector
- Debezium PostgreSQL source connector
- Kafka source connector
- Kinesis source connector
- Data generator source connector (for internal testing purposes only)
Built-in sink connectors
Currently, StreamNative Cloud supports the following sink connectors.
- AMQP1_0 sink connector
- AWS EventBridge sink connector
- AWS Lambda sink connector
- AWS SQS sink connector
- AWS S3 Sink Connector
- Google Cloud Storage Sink Connector
- Azure Blob Storage Sink Connector
- Elasticsearch sink connector
- Google BigQuery sink connector
- Kinesis sink connector
- Snowflake sink connector
- Data generator sink connector (for internal testing purposes only)