Understand Pulsar Operators

An Operator is a controller that manages an application in Kubernetes. It provides a full management life-cycle for the application, including deployment, upgrades, scaling, and configuration changes. With the Pulsar Operator, Pulsar can run seamlessly in the Kubernetes clusters deployed on a public or private cloud. You can then manage the Pulsar cluster using the Kubernetes Application Programming Interface (API) and the kubectl tool.

Why use Operators?

Operators take human operational knowledge and encode it into the software that is more easily shared with consumers.

Pulsar Operators provide:

Horizontal Scaling
The Pulsar Operators enable Pulsar clusters to scale horizontally in the cloud.
Rolling updates
Gracefully perform rolling updates of the Pulsar cluster with zero downtime.
Multi-tenant support
Users can deploy and manage multiple Pulsar clusters on a single Kubernetes cluster easily.
Automatic monitoring
Automatically deploy Prometheus or Grafana to monitor the Pulsar cluster.

What are Operators?

An Operator builds upon the basic Kubernetes resource and controller concepts, but includes domain or application-specific knowledge to automate the entire life cycle of the Pulsar instances it manages.

An Operator is a custom Kubernetes controller that uses Custom Resources (CRs) to manage components. The user provides high-level configuration and settings within a CR. Operators translate these high-level directives into the low level actions, based on best practices embedded within the Operator’s logic.

Concepts

This section provides some basic concepts about the Operators.

Custom Resource Definitions

A Custom Resource Definition (CRD) defines a CR and lists out all of the configurations available. The Operator introduces new object types through CRDs. The Kubernetes API handles CRDs just like built-in objects, including interaction via kubectl and inclusion in Role-Based Access Control (RBAC) policies.

When a cluster administrator adds a new CRD to the cluster, the Kubernetes API server creates a new RESTful resource path that the entire cluster or a single project (namespace) can access and begin serving the specified CR.

While only cluster administrators can create CRDs, developers can create the CR from an existing CRD if they have read and write permissions to it.

Custom Resources

A CR is the Application Programming Interface (API) extension mechanism in Kubernetes.When a cluster administrator adds CRDs to a cluster, CR objects are created, allowing all cluster users to add the new resource type into projects. The Operator watches a CR type and takes application-specific actions to make the current state match the desired state in that resource.

Supported Operators

StreamNative provides the following Operators:

Pulsar Operator
The Pulsar Operator is a controller that manages Pulsar clusters in Kubernetes. It provides a full management life-cycle for Pulsar, including deployment, upgrades, scaling, and configuration changes. With the Pulsar Operator, Pulsar can run seamlessly in the Kubernetes clusters deployed on a public or private cloud. You can then manage Pulsar clusters using the Kubernetes Application Programming Interface (API) and the kubectl tool.
BookKeeper Operator
The BookKeeper Operator manages BookKeeper clusters deployed to Kubernetes and automates tasks related to operating a Bookkeeper cluster. It provides a full management life-cycle for BookKeeper, including deployment, upgrades, scaling, and configuration changes. You can manage BookKeeper clusters using the Kubernetes Application Programming Interface (API) and the kubectl tool.
ZooKeeper Operator
The ZooKeeper Operator manages ZooKeeper clusters deployed to Kubernetes and automates tasks related to operating a ZooKeeper cluster. It provides a full management life-cycle for ZooKeeper, including deployment, upgrades, scaling, and configuration changes. You can manage ZooKeeper clusters using the Kubernetes Application Programming Interface (API) and the kubectl tool.
Function Mesh Operator
The Function Mesh Operator enables users to run Pulsar Functions and Pulsar connectors natively on Kubernetes, unlocking the full power of Kubernetes’ features and resources. By providing a serverless framework that enables users to organize a collection of Pulsar Functions and connectors, the Function Mesh Operator simplifies the process of creating complex streaming jobs.

How Operators work?

Basically, an Operator is a Kubernetes controller. First, the user provides a manifest file. Then, the controller creates the corresponding resource in the Kubernetes cluster according to the information in the manifest file, and adjusts the resource to ensure their states meet the user's specifications.

The diagram above illustrates the process for using an Operator.

The user submits a manifest file directly to the Kubernetes cluster using the kubectl CLI.
The Operator (installed inside Kubernetes) deploys the resource.
The Kubernetes API server creates, updates, or deletes the corresponding containers and services based on the manifest file.

Naming conventions

StreamNative names operators in a format of operator_name-version, such as pulsar_operator-v0.8.10.

You can find the latest images of the following operators via the Docker Hub.