- Operating StreamNative Platform
- Maintenance
Upgrade StreamNative Platform
This document introduces how to migrate StreamNative Platform from previous releases to StreamNative Platform 1.8.0 and later.
Note
- This document only applies to a situation where you install StreamNative Platform using a custom
values.yaml
file instead of the defaultvalues.yaml
file that is brought by thesn-platform
chart. - It is recommended to upgrade Operators first and then upgrade StreamNative Platform components. Otherwise, some unexpected errors might occur.
- It is recommended that you perform this upgrade separately from making other changes to your clusters, such as scaling up or changing configuration.
Upgrade Operators
StreamNative Platform uses different Operators for deploying Vault, cert-manager, Pulsar cluster, and Function Mesh.
To upgrade Operators used by StreamNative Platform, follow these steps.
Get the latest version of the Vault Operator, cert-manager Operator, Pulsar Operator, and Function Mesh operator.
helm repo add streamnative https://charts.streamnative.io helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com helm repo add jetstack https://charts.jetstack.io helm repo add function-mesh http://charts.functionmesh.io/ helm repo update
Download the latest charts.
This example shows how to download the latest chart for the Pulsar Operator. If you want to download a specific version, you can use the
--version
option to specify the chart version.helm pull streamnative/pulsar-operator --untar
Upgrade StreamNative Platform Custom Resource Definitions (CRDs).
Note
Helm does not support upgrading or deleting CRDs using Helm. For more information, see the Helm documentation.
Upgrade the CRD for the Pulsar Operator.
kubectl apply -f /path/to/your/pulsar-operator/crds/
Upgrade the CRD for the Function Mesh operator.
kubectl apply -f /path/to/your/functionmesh-operator/crds/
Use the
helm upgrade
command to upgrade these Operators one by one.By default, the latest versions of Operators are installed. If you want to install a specific version, you can use the
--version
option to specify the Operator version.Vault Operator
helm upgrade --install vault-operator banzaicloud-stable/vault-operator -n <k8s_namespace>
cert-manager Operator
helm upgrade --install cert-manager jetstack/cert-manager -n <k8s_namespace>
Pulsar Operator
helm upgrade --install pulsar-operator streamnative/pulsar-operator -n <k8s_namespace>
Function Mesh operator
helm upgrade --install function-mesh function-mesh/function-mesh-operator -n <k8s_namespace>
Confirm that all Kubernetes Pods are running.
kubectl get pods -n <kubernetes_namespace>
Upgrade StreamNative Platform components
StreamNative Platform 1.8.0 brought some breaking changes that deprecated several APIs. Some StreamNative Platform components are affected due to these deprecated APIs. Therefore, in addition to other custom configurations, when you upgrade StreamNative Platform components from previous releases to StreamNative Platform 1.8.0 or later, pay special attention to these components.
Kafka client
In previous charts, you can enable the Kafka client in the toolset
Pod by specifying the components.kop
option in the values.yaml
YAML file.
Before upgrade
components:
kop: false
However, the components.kop
option has no relationship with the KoP protocol plugin that is configured through the broker.kop
option and often causes confusion. Therefore, in StreamNative Platform 1.8.0, we deprecated the components.kop
option and used the toolset.kafka
option to enable the Kafka client in the toolset
Pod.
After upgrade
# kafka client deprecated 1.8
components:
kop: false
# kafka client 1.8, replacing components.kop
toolset:
kafka:
enabled: false
ZooKeeper
In previous releases, you can use the zookeeper.advanced.staticServerList
option to specify the ZooKeeper servers to be observed or joined.
Before upgrade
zookeeper:
reconfig:
enabled: true
advanced:
staticServerList:
- id: 1
listeners:
- hostname: 'pulsar-zookeeper-0.pulsar-zookeeper.snp.svc.cluster.local'
- id: 2
listeners:
- hostname: 'pulsar-zookeeper-1.pulsar-zookeeper.snp.svc.cluster.local'
- id: 3
listeners:
- hostname: 'pulsar-zookeeper-2.pulsar-zookeeper.snp.svc.cluster.local'
In StreamNative Platform 1.8.0, you can use the zookeeper.reconfig.staticServerList
option to specify the ZooKeeper servers.
After upgrade
zookeeper:
reconfig:
enabled: true
staticServerList:
- id: 1
listeners:
- hostname: 'pulsar-zookeeper-0.pulsar-zookeeper.snp.svc.cluster.local'
- id: 2
listeners:
- hostname: 'pulsar-zookeeper-1.pulsar-zookeeper.snp.svc.cluster.local'
- id: 3
listeners:
- hostname: 'pulsar-zookeeper-2.pulsar-zookeeper.snp.svc.cluster.local'
Local storage
The local_storage
option is used to enable the local storage provisioner that is installed through the streamnative/local-storage-provisioner
chart. It is not a generic way to use local storage.
Before upgrade
volumes:
local_storage: true
zookeeper:
volumes:
data:
local_storage: true
dataLog:
local_storage: true
bookkeeper:
volumes:
journal:
local_storage: true
ledgers:
local_storage: true
<component>: # --- [1]
volumes:
data:
local_storage: true
- [1]
<component>
: represent Prometheus, Grafana, StreamNative Console, and Vault.
The more generic way is to specify the StorageClass
and configure the storageClassName
option for StreamNative Platform components. If you do not specify the storageClassName
for StreamNative Platform components but set the default storage class in the values.yaml
YAML, the Pulsar Operator uses the default storage class.
After upgrade
volumes:
local_storage: false
zookeeper:
volumes:
data:
storageClassName: '<Your-Installed-storageClass>'
dataLog:
storageClassName: '<Your-Installed-storageClass>'
bookkeeper:
volumes:
journal:
storageClassName: '<Your-Installed-storageClass>'
ledgers:
storageClassName: '<Your-Installed-storageClass>'
<component>:
volumes:
data:
storageClassName: '<Your-Installed-storageClass>'
Datadog
In StreamNative Platform 1.8.0, you can install the Datadog in the same Kubernetes namespace where your Pulsar cluster is deployed. Therefore, we deprecated the datadog.namespace
option and now refer to the pulsar.namespace
.
Before upgrade
monitoring:
datadog: true
datadog:
component: datadog
namespace: pulsar # --- [1]
components:
zookeeper:
enabled: false
metrics: ['"_*"']
bookkeeper:
enabled: false
metrics: ['"_*"']
broker:
enabled: false
metrics: ['"_*"']
---
# other configs
- [1]
namespace
: specify the Kubernetes namespace where Datadog is installed.
After upgrade
monitoring:
datadog: true
datadog:
component: datadog
components:
zookeeper:
enabled: false
metrics: ['"_*"']
bookkeeper:
enabled: false
metrics: ['"_*"']
broker:
enabled: false
metrics: ['"_*"']
---
# other configs
Pulsar resource CRs
In previous releases, legacy Pulsar resource CRs are bundled with the sn-platform
chart. StreamNative has released the Pulsar Resources Operator. The Pulsar Resources Operator is an independent controller that automatically manages Pulsar resources (such as tenants, namespace, topics, and permissions) on Kubernetes through manifest files. In StreamNative Platform 1.8.0, we remove legacy Pulsar resource CRs and create new Pulsar resource CRs using the Pulsar Resources Operator. This helps simplify the StreamNative Platform installation process and eliminate the confusion. Follow the instructions below to install the new Pulsar Resource Operator and create Pulsar resource CRs.
Note
Before creating Pulsar resource CRs using the Pulsar Resources Operator, ensure that the lifecyclePolicy
option of your existing Pulsar resource CRs is not set to CleanUpAfterDeletion
.
Install the Pulsar Resources Operator. For details, see the installation guide.
Create the PulsarConnection.
The PulsarConnection covers the address of the Pulsar cluster and the authentication information. The Pulsar Resources Operator uses PulsarConnection to access the Pulsar cluster to create other resources.
This example shows how to create a PulsarConnection (named
pulsar-connection
) using the Pulsar Resources Operator.a. Define a PulsarConnection in a YAML file that contains the fields shown in the file below.
apiVersion: resource.streamnative.io/v1alpha1 kind: PulsarConnection metadata: name: pulsar-connection namespace: <k8s-namespace> spec: adminServiceURL: http://pulsar-sn-platform-broker.test.svc.cluster.local:8080 # --- [1]
- [1]
adminServiceURL
: the service URL of the Pulsar cluster.
b. Apply the YAML file to create the PulsarConnection and check the status.
kubectl apply -f connection.yaml
c. Check the PulsarConnection status.
kubectl get pulsarconnections.resource.streamnative.io/v1alpha1 -n <k8s-namespace> NAME ADMIN_SERVICE_URL GENERATION OBSERVED_GENERATION READY pulsar-connection http://pulsar-xxxx:8080 1 1 True
- [1]
Create Pulsar resources using the Pulsar Resources Operator.
The newly created Pulsar resources are using the new API with the
resource.streamnative.io/v1alpha1
API version.This example shows how to create a tenant (named
pulsar-tenant
) using the Pulsar Resources Operator.a. Define a tenant in a YAML file that contains the fields shown in the file below.
apiVersion: resource.streamnative.io/v1alpha1 kind: PulsarTenant metadata: name: pulsar-tenant namespace: <k8s-namespace> spec: name: pulsar-tenant connectionRef: name: pulsar-connection # --- [1] adminRoles: - admin - ops
- [1]
name
: the name of the PulsarConnection that is created in the previous step.
b. Apply the YAML file to create the tenant and check the status.
kubectl apply -f tenant.yaml
c. Check Pulsar tenant status.
kubectl get pulsartenants.resource.streamnative.io -n <k8s-namespace> NAME RESOURCE_NAME GENERATION OBSERVED_GENERATION READY pulsar-tenant pulsar-tenant 1 1 True
- [1]
Delete legacy Pulsar resource CRs that are created by the legacy API with the
pulsar.streamnative.io/v1alpha1
API version.