Configure Oxia

Oxia support is experimental in StreamNative Private Cloud. Use it only when your deployment plan, Private Cloud release, and support guidance require it.

Oxia is a metadata store and coordination system that can back Pulsar metadata and related Private Cloud workloads when your release supports the integration. In Private Cloud V2, configure Oxia with these Kubernetes custom resources:

OxiaCluster configures the Oxia server cluster, including the image, replicas, storage behavior, monitoring, and scheduling options.
OxiaNamespace configures logical namespaces inside an Oxia cluster. Use namespaces to separate broker metadata, schema metadata, Lakehouse storage metadata, or other supported workloads.

Example manifest

For a runnable Oxia-backed Pulsar cluster, apply the Private Cloud example manifest:

kubectl apply -f https://raw.githubusercontent.com/streamnative/private-cloud/main/oxia/pulsar-oxia.yaml

The example manifest creates the OxiaCluster, the required OxiaNamespace resources, the Private Cloud metadata wiring, and the Pulsar components that use Oxia. The following excerpt shows the core Oxia resources:

apiVersion: k8s.streamnative.io/v1alpha1
kind: OxiaCluster
metadata:
  name: private-cloud
  namespace: pulsar
  labels:
    k8s.streamnative.io/coordinator-name: private-cloud
spec:
  monitoringEnabled: false
  image: oxia/oxia:<version>
  imagePullPolicy: IfNotPresent
  server:
    replicas: 3
---
apiVersion: k8s.streamnative.io/v1alpha1
kind: OxiaNamespace
metadata:
  name: private-cloud-broker
  namespace: pulsar
spec:
  clusterRef:
    name: private-cloud
    namespace: pulsar
  namespaceConfig:
    name: broker
    initialShardCount: 1
    replicationFactor: 3
    notificationsEnabled: true

When you use the example manifest, you do not need to configure Oxia URLs separately.

Function package management is not supported yet with Oxia-backed Private Cloud clusters. The example manifest disables broker package management and Function Mesh package uploads.

Configure OxiaCluster

OxiaCluster manages the Oxia server cluster. Create it in the same namespace as the Pulsar cluster unless your deployment plan uses a different namespace.

apiVersion: k8s.streamnative.io/v1alpha1
kind: OxiaCluster
metadata:
  name: private-cloud
  namespace: pulsar
  labels:
    k8s.streamnative.io/coordinator-name: private-cloud
spec:
  monitoringEnabled: false
  image: oxia/oxia:<version>
  imagePullPolicy: IfNotPresent
  server:
    replicas: 3
    persistentVolumeClaimRetentionPolicy:
      whenDeleted: Retain

Supported OxiaCluster fields:

Field	Description
`metadata.name`	Name of the Oxia cluster resource. `OxiaNamespace.spec.clusterRef.name` must match this value. Generated server resources use the `<name>-oxia` naming pattern.
`metadata.namespace`	Kubernetes namespace for the Oxia resources. Use the Pulsar namespace unless your deployment plan separates components.
`metadata.labels["k8s.streamnative.io/coordinator-name"]`	Associates the Oxia resource with the Private Cloud coordinator.
`spec.image`	Oxia image and version. Choose a version supported by your Private Cloud release.
`spec.imagePullPolicy`	Kubernetes image pull policy for the Oxia image. If omitted, the operator defaults it to `IfNotPresent`.
`spec.labels`	Labels propagated to generated Oxia workloads.
`spec.monitoringEnabled`	Controls whether the operator creates Prometheus Operator monitoring resources for Oxia. If omitted, monitoring is enabled. The local/test example sets this to `false` because clusters without Prometheus Operator CRDs cannot create the generated `PodMonitor`.
`spec.pprofEnabled`	Controls whether pprof endpoints are enabled. If omitted, pprof is enabled.
`spec.serviceURLGenerationPolicy`	Service URL generation policy. The CRD currently supports `FQDN` only.

Supported spec.coordinator fields:

Field	Description
`spec.coordinator.config`	Arbitrary Oxia coordinator configuration. The operator preserves unknown fields and merges this configuration with operator-managed defaults.
`spec.coordinator.logLevel`	Oxia coordinator log level. If omitted, the operator defaults it to `INFO`.
`spec.coordinator.resources`	Resource requests and limits for the Oxia coordinator. If requests are omitted, the operator defaults requests to `cpu: 0.5` and `memory: 400Mi`.

Supported spec.istio fields:

Field	Description
`spec.istio.enabled`	Enables Istio integration for Oxia traffic.
`spec.istio.dataplaneMode`	Istio dataplane mode. Supported values are `sidecar` and `ambient`. The default is `sidecar`.
`spec.istio.revision`	Istio control-plane revision used for sidecar injection. The default is `default`.
`spec.istio.mtls.mode`	Istio mTLS mode. The default is `strict`.
`spec.istio.trustDomain`	Istio trust domain used when generating workload principals for authorization policies.
`spec.istio.authRules`	Authorization policy rules. Each rule can define source principals and destination ports.
`spec.istio.gateway`	Single Istio gateway configuration. The operator uses this field when `spec.istio.gateways` is empty.
`spec.istio.gateways`	Multiple Istio gateway configurations. The CRD supports up to 10 entries and uses `key` as the list map key. When this list is not empty, it takes precedence over `spec.istio.gateway`.

Supported spec.server fields:

Field	Description
`spec.server.replicas`	Number of Oxia server replicas. The CRD default is `3`, and the minimum allowed value is `0`. Use a value that satisfies your availability and quorum requirements.
`spec.server.authOptions.enabled`	Enables Oxia server authentication.
`spec.server.authOptions.authProviderName`	Authentication provider name. The operator source includes `oidc` as the OIDC provider name.
`spec.server.authOptions.authProviderParams`	Authentication provider parameters. For OIDC, parameters include allowed issuer URLs, allowed audiences, and the username claim.
`spec.server.blockCacheSize`	Oxia shared DB block cache size as a Kubernetes quantity, such as `512Mi` or `1Gi`. If omitted and a server memory limit is set, the operator uses 50% of the memory limit.
`spec.server.blockCacheSizeMB`	Deprecated block cache size in MB. Use `spec.server.blockCacheSize` instead. If both are set, `blockCacheSize` takes precedence.
`spec.server.config`	Arbitrary Oxia server configuration. The operator preserves unknown fields and merges this configuration with operator-managed defaults.
`spec.server.logLevel`	Oxia server log level. If omitted, the operator defaults it to `INFO`.
`spec.server.resources`	Resource requests and limits for Oxia server pods. If requests are omitted, the operator defaults requests to `cpu: 0.5` and `memory: 400Mi`.
`spec.server.persistentVolumeClaimRetentionPolicy.whenDeleted`	Controls whether PVCs created from the Oxia StatefulSet volume claim template are retained or deleted when the StatefulSet is deleted. Kubernetes defaults to `Retain`.
`spec.server.persistentVolumeClaimRetentionPolicy.whenScaled`	Controls whether PVCs are retained or deleted when replicas are scaled down. Kubernetes defaults to `Retain`.
`spec.server.volumeClaimSpec`	Persistent volume claim template for Oxia data storage. If omitted, the operator defaults to `ReadWriteOnce` with a `4Gi` storage request.

Supported spec.server.pod fields:

Field	Description
`spec.server.pod.affinity`	Pod affinity, anti-affinity, and node affinity. If omitted, the operator adds preferred anti-affinity across hostnames and topology zones for Oxia server pods.
`spec.server.pod.annotations`	Additional annotations for generated Oxia server pods.
`spec.server.pod.debug`	Enables pod debug mode.
`spec.server.pod.imagePullSecrets`	Image pull secrets for Oxia server pods.
`spec.server.pod.initContainers`	Additional init containers.
`spec.server.pod.labels`	Additional labels for generated Oxia server pods.
`spec.server.pod.minReadySeconds`	Minimum ready seconds for the generated StatefulSet.
`spec.server.pod.nodeSelector`	Node selector for scheduling Oxia server pods.
`spec.server.pod.resources`	Pod-level resource requests and limits from the shared pod policy. Prefer `spec.server.resources` for the Oxia server container.
`spec.server.pod.secretRefs`	Secrets mounted into generated containers. Each entry defines a secret name and mount path.
`spec.server.pod.securityContext`	Pod security context.
`spec.server.pod.serviceAccountName`	Service account used by Oxia server pods. If omitted, the operator defaults it to `<metadata.name>-oxia`.
`spec.server.pod.sidecars`	Additional sidecar containers.
`spec.server.pod.terminationGracePeriodSeconds`	Pod termination grace period.
`spec.server.pod.tolerations`	Pod tolerations.
`spec.server.pod.topologySpreadConstraints`	Topology spread constraints for scheduling.
`spec.server.pod.updateStrategy`	StatefulSet update strategy for Oxia server pods.
`spec.server.pod.vars`	Additional variables for generated pod configuration.
`spec.server.pod.volumes`	Additional pod volumes. The CRD supports `configMap` and `secret` volume sources.

Review the Oxia image tag, replica count, persistent volume retention policy, storage class, monitoring mode, authentication, and networking settings before applying the resource.

Configure OxiaNamespace

OxiaNamespace creates isolated Oxia namespaces for Pulsar metadata, schema storage, state storage, or other supported workloads. Create one resource for each namespace your workload requires.

apiVersion: k8s.streamnative.io/v1alpha1
kind: OxiaNamespace
metadata:
  name: broker
  namespace: pulsar
spec:
  clusterRef:
    name: private-cloud
    namespace: pulsar
  namespaceConfig:
    name: broker
    initialShardCount: 1
    replicationFactor: 3
    notificationsEnabled: true

Key fields:

Field	Description
`metadata.name`	Kubernetes resource name for the Oxia namespace.
`metadata.namespace`	Kubernetes namespace where the `OxiaNamespace` resource is created.
`spec.clusterRef.name`	Required. Name of the target `OxiaCluster`.
`spec.clusterRef.namespace`	Required. Namespace of the target `OxiaCluster`.
`spec.ownerRef.name`	Optional owner resource name. The operator can use this reference to set owner relationships for generated Oxia namespace resources.
`spec.ownerRef.namespace`	Optional owner resource namespace.
`spec.namespaceConfig.name`	Required. Logical namespace name inside Oxia. Use this value in Oxia-backed metadata paths.
`spec.namespaceConfig.initialShardCount`	Initial shard count for the logical namespace. The CRD default is `1`, and the minimum allowed value is `0`. Tune this for expected metadata scale and workload behavior.
`spec.namespaceConfig.replicationFactor`	Replication factor for the namespace. The CRD default is `3`, and the minimum allowed value is `0`. Tune this for durability and availability requirements.
`spec.namespaceConfig.notificationsEnabled`	Controls Oxia namespace notifications. The CRD default is `true`.

The pulsar-oxia.yaml example creates the following Oxia namespaces for a Pulsar cluster:

Namespace	Shards	Purpose
`broker`	1	Broker metadata and coordination.
`bookkeeper`	1	BookKeeper metadata.
`pulsar-schema`	1	Pulsar schema metadata.
`kafka-schema`	1	Kafka schema registry metadata.
`function`	1	Function metadata.

Adjust metadata.name, namespaceConfig.name, initialShardCount, and replicationFactor for each namespace. When Oxia metadata is enabled through the Oxia-backed Private Cloud manifest, the metadata store and schema storage URLs are configured by that manifest. You do not need to configure the Oxia URLs separately unless you are building a custom manifest.

Check status

Use status fields to confirm that the operator reconciled the Oxia resources:

Resource	Status field	Description
`OxiaCluster`	`status.conditions`	Current reconciliation conditions.
`OxiaCluster`	`status.serverStatus.replica` and `status.serverStatus.readyReplicas`	Desired and ready server replica counts observed from the generated StatefulSet.
`OxiaCluster`	`status.serverStatus.onMeshReplicas`	Number of Oxia server pods detected on the Istio mesh.
`OxiaCluster`	`status.coordinatorStatus.replica` and `status.coordinatorStatus.readyReplicas`	Desired and ready coordinator counts. For Oxia versions that use sidecar coordinator mode, coordinator status mirrors server status.
`OxiaCluster`	`status.namespaces`	Namespace readiness and shard status for namespaces associated with the cluster.
`OxiaCluster`	`status.pendingChanges`	Changes skipped while rollout is paused.
`OxiaNamespace`	`status.conditions`	Current namespace reconciliation conditions.
`OxiaNamespace`	`status.shards`	Oxia shard IDs and states for the namespace.

Overview

Quick Start

Configure Private Cloud

Operate Private Cloud

References

Example manifest

Configure OxiaCluster

Configure OxiaNamespace

Check status

​Example manifest

​Configure OxiaCluster

​Configure OxiaNamespace

​Check status

Example manifest

Configure OxiaCluster

Configure OxiaNamespace

Check status