Develop Pulsar Functions in WASM(Private Preview)

This section introduces how to develop and pacakge WASM Pulsar functions to use on StreamNative cloud. The WASM runtime is still in private preview stage, If you want to try it out or have any questions, please submit a ticket to the support team.

Develop

The WASM runtime is using WasmEdge, theoretically, you can use any languages which can be compiled to a WASM module to write your functions, below is an example using Rust:

cargo.toml

[package]
name = "excla"
version = "0.1.0"
edition = "2021"

[lib]
crate-type = ["cdylib"]

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
apache-avro = { version = "0.15.0", features = ["bzip", "xz", "snappy", "zstandard"] }
serde = "^1.0"
serde_json = "^1.0"
wasmedge-bindgen = "0.4.1"
wasmedge-bindgen-macro = "0.4.1"

src/lib.rs

#[allow(unused_imports)]
use wasmedge_bindgen::*;
use wasmedge_bindgen_macro::*;

use serde::{Deserialize, Serialize};

#[derive(Debug, Deserialize,  Serialize)]
pub struct Student {
    pub name: Option<String>,
    pub age: Option<i32>,
    pub grade: Option<i32>
}

// use `process_json` as the class name when creating functions
#[wasmedge_bindgen]
pub fn process_json(s: Vec<u8>) -> Vec<u8> {
    let stu = serde_json::from_slice::<Student>(&s[..]).unwrap();
    let stu = Student {
        grade: stu.grade.map(|grade| grade + 1),
        ..stu
    };
    let res = serde_json::to_vec(&stu).unwrap();
    return res
}

Feature Matrix

The WASM runtime doesn’t support full features comparing to Java runtime, and it’s still in developing, below is the matrix:

Input Arguments

Input	Java	Go(Pulsar)	Python	WASM
Custom SerDe	✅	❌	✅	?
Schema - Avro	✅	❌	✅	?
Schema - JSON	✅	❌	✅	?
Schema - Protobuf	✅	❌	❌	?
Schema - KeyValue	✅	❌	❌	?
Schema - AutoSchema	✅	❌	❌	?
Scehma - Protobuf Native	✅	❌	❌	?
e-2-e encryption	✅	❌	✅	✅
maxMessageRetries	✅	❌	❌	✅
dead-letter policy	✅	❌	❌	✅
SubscriptionName	✅	✅	✅	✅
SubscriptionType	✅	✅	✅	✅
SubscriptionInitialPosition	✅	❌	✅	✅
AutoAck	✅	✅	✅	✅

Users can implement the Schema themselves since we are passing and expecting []byte to/from the users’ function, so leave ? here.

Output Arguments

Output	Java	Go(Pulsar)	Python	WASM
Custom SerDe	✅	❌	✅	?
Schema - Avro	✅	❌	✅	?
Schema - JSON	✅	❌	✅	?
Schema - Protobuf	✅	❌	❌	?
Schema - KeyValue	✅	❌	❌	?
Schema - AutoSchema	✅	❌	❌	?
Schema - Protobuf Native	✅	❌	❌	?
useThreadLocalProducers	✅	❌	❌	✅
Key-based Batcher	✅	✅	✅	✅
e-2-e encryption	✅	❌	✅	✅
Compression	✅	✅	✅	✅

Context

WASM runtime doesn’t support the Context features at all for now.

Other

Other	Java	Go(Pulsar)	Python	WASM
Resources	✅	✅	✅	✅
At-most-once	✅	✅	✅	✅
At-least-once	✅	✅	✅	✅
Effectively-once	✅	❌	✅	❌

Package

You need to compile the function to a .wasm module first before creating Pulsar Functions.

cargo build --target wasm32-wasi --release

Deploy

After creating a cluster, set up your environment and develop&package your function, you can use the snctl, pulsarctl, pulsar-admin command, the REST API, or terraform to deploy a Pulsar function to your cluster. You can create a WASM Pulsar function by using a local .wasm file or an uploaded Pulsar functions package(recommend).

(Optional) Upload your function file to Pulsar

It’s recommended to upload your function file to Pulsar before you create a function. Since you can add a version suffix to the package.

snctl
Pulsarctl
Pulsar-admin

Upload packages

snctl pulsar admin packages upload function://${tenant}/${namespace}/${package_name} \
--path ${file_path} \
--description "${description}" \
--properties fileName=${file_name}

You should see the following output:

The package 'function://${tenant}/${namespace}/${package_name}' uploaded from path '${file_path}' successfully

You need to set the context for Pulsarctl first:

# create a context
pulsarctl context set ${context-name}  \
--admin-service-url ${admin-service-url} \
--issuer-endpoint ${issuerUrl} \
--audience urn:sn:pulsar:${orgName}:${instanceName} \
--key-file ${privateKey}

# activate oauth2
pulsarctl oauth2 activate

Replace the placeholder variables with the actual values that you can get when setting up client tools.

context-name: any name you want
admin-service-url: the HTTP service URL of your Pulsar cluster.
privateKey: the path to the downloaded OAuth2 key file.
issuerUrl: the URL of the OAuth2 issuer.
audience: the Uniform Resource Name (URN), which is a combination of the urn:sn:pulsar, your organization name, and your Pulsar instance name.
- ${orgName}: the name of your organization.
- ${instanceName}: the name of your instance.

Upload packages

pulsarctl packages upload function://${tenant}/${namespace}/${package_name} \
--path ${file_path} \
--description "${description}" \
--properties fileName=${file_name}

You should see the following output:

The package 'function://${tenant}/${namespace}/${package_name}' uploaded from path '${file_path}' successfully

./bin/pulsar-admin \
    --admin-url "${WEB_SERVICE_URL}" \
    --auth-plugin org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 \
    --auth-params '{"privateKey":"file://${privateKey}","issuerUrl":"${issuerUrl}","audience":"urn:sn:pulsar:${orgName}:${instanceName}}' \
    packages upload function://${tenant}/${namespace}/${package_name} \
    --path ${file_path} \
    --description "${description}" \
    --properties fileName=${file_name}

Replace the placeholder variables with the actual values that you can get when setting up client tools.

admin-url: the HTTP service URL of your Pulsar cluster.
privateKey: the path to the downloaded OAuth2 key file.
issuerUrl: the URL of the OAuth2 issuer.
audience: the Uniform Resource Name (URN), which is a combination of the urn:sn:pulsar, your organization name, and your Pulsar instance name.
- ${orgName}: the name of your organization.
- ${instanceName}: the name of your instance.

You should see the following output:

The package 'function://${tenant}/${namespace}/${package_name}' uploaded from path '${file_path}' successfully

Create

snctl
Pulsarctl
Pulsar-admin
Terraform
REST API

snctl pulsar admin functions create \
--tenant public \
--namespace default \
--name function1 \
--inputs persistent://public/default/test-wasm-input \
--output persistent://public/default/test-wasm-output \
--classname exclamation \
--py function://public/default/wasm-exclamation@v0.1 \
--custom-runtime-options '{"genericKind": "wasm"}' \
--sn-service-account $SERVICE_ACCOUNT

Only Pulsar 4.0.x clusters running version 4.0.10.6 or later, or Pulsar 4.2.x clusters running version 4.2.1.4 or later, support the --sn-service-account flag.You can use --as-service-account instead of --sn-service-account if you are using other versions of Pulsar clusters.

Since Pulsar doesn’t support WASM runtime, we need to use --py to specify the function file and specify the --custom-runtime-options '{"genericKind": "wasm"}' to make it work.

You should see something like this:

Created function1 successfully

pulsarctl functions create \
--tenant public \
--namespace default \
--name function1 \
--inputs persistent://public/default/test-wasm-input \
--output persistent://public/default/test-wasm-output \
--classname exclamation \
--py function://public/default/wasm-exclamation@v0.1 \
--custom-runtime-options '{"genericKind": "wasm"}'

Since Pulsar doesn’t support WASM runtime, we need to use --py to specify the function file and specify the --custom-runtime-options '{"genericKind": "wasm"}' to make it work.

You should see something like this:

Created function1 successfully

./bin/pulsar-admin \
--admin-url "${WEB_SERVICE_URL}" \
--auth-plugin org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 \
--auth-params '{"privateKey":"file:///YOUR-KEY-FILE-PATH","issuerUrl":"https://auth.streamnative.cloud/","audience":"urn:sn:pulsar:${orgName}:${instanceName}}' \
functions create \
--tenant public \
--namespace default \
--name function1 \
--inputs persistent://public/default/test-wasm-input \
--output persistent://public/default/test-wasm-output \
--classname exclamation \
--py function://public/default/wasm-exclamation@v0.1 \
--custom-runtime-options '{"genericKind": "wasm"}'

Since Pulsar doesn’t support WASM runtime, we need to use --py to specify the function file and specify the --custom-runtime-options '{"genericKind": "wasm"}' to make it work.

You should see something like this:

Created successfully

Create your terraform yaml file:

terraform {
  required_providers {
    pulsar = {
      version = "0.2.0"
      source = "registry.terraform.io/streamnative/pulsar"
    }
  }
}

provider "pulsar" {
  web_service_url = "{$admin-url}"
  api_version     = "3"
  audience = "urn:sn:pulsar:${orgName}:${instanceName}}"
  issuer_url = "${issuerUrl}"
  key_file_path = "${privateKey}"
}

// Note: function resource requires v3 api.
resource "pulsar_function" "function-1" {
  provider = pulsar

  name = "function1"
  tenant = "public"
  namespace = "default"
  parallelism = 1

  processing_guarantees = "ATLEAST_ONCE"

  py = "function://public/default/wasm-exclamation@v0.1"
  classname = "exclamation.ExclamationFunction"

  inputs = ["persistent://public/default/test-wasm-input"]

  output = "persistent://public/default/test-wasm-output"

  subscription_name = "test-sub"
  subscription_position = "Latest"
  cleanup_subscription = true
  skip_to_latest = true
  forward_source_message_property = true
  retain_key_ordering = true
  auto_ack = true
  max_message_retries = 100
  dead_letter_topic = "public/default/dlt"
  log_topic = "public/default/lt"
  timeout_ms = 6666

  secrets = jsonencode(
  {
    "SECRET1": {
       "path": "sectest",
       "key": "hello"
    }
  })
  custom_runtime_options = jsonencode(
  {
     "genericKind": "wasm",
      "env": {
          "HELLO": "WORLD"
      },
      "snServiceAccount": "${SERVICE_ACCOUNT}"
  })
}

Since Pulsar doesn’t support WASM runtime, we need to use --py to specify the function file and specify the --custom-runtime-options '{"genericKind": "wasm"}' to make it work.

Init the terraform provider in the same dir of your .tf file if you haven’t done it:

terraform init

You should see something like this:

Initializing the backend...

Initializing provider plugins...
- Finding streamnative/pulsar versions matching "0.2.0"...
- Installing streamnative/pulsar v0.2.0...
- Installed streamnative/pulsar v0.2.0 (self-signed, key ID 3105E1011F3C3671)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Create the function:

terraform apply

You should see something like:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # pulsar_function.function-1 will be created
  + resource "pulsar_function" "function-1" {
      + auto_ack                        = true
      + classname                       = "exclamation.ExclamationFunction"
      + cleanup_subscription            = true
      + cpu                             = 0.5
      + custom_runtime_options          = jsonencode(
            {
              + genericKind = "wasm",
              + env = {
                  + HELLO = "WORLD"
                },
              + snServiceAccount = "${SERVICE_ACCOUNT}"
            }
        )
      + dead_letter_topic               = "public/default/dlt"
      + disk_mb                         = 128
      + forward_source_message_property = true
      + id                              = (known after apply)
      + inputs                          = [
          + "persistent://public/default/test-wasm-input",
        ]
      + py                              = "function://public/default/wasm-exclamation@v0.1"
      + log_topic                       = "public/default/lt"
      + max_message_retries             = 100
      + name                            = "function1"
      + namespace                       = "default"
      + output                          = "persistent://public/default/test-wasm-output"
      + parallelism                     = 1
      + processing_guarantees           = "ATLEAST_ONCE"
      + ram_mb                          = 128
      + retain_key_ordering             = true
      + secrets                         = jsonencode(
            {
              + SECRET1 = {
                  + key  = "hello"
                  + path = "sectest"
                }
            }
        )
      + skip_to_latest                  = true
      + subscription_name               = "test-sub"
      + subscription_position           = "Latest"
      + tenant                          = "public"
      + timeout_ms                      = 6666
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value:

After enter “yes”, you should see the following:

pulsar_function.function-1: Creating...
pulsar_function.function-1: Creation complete after 1s [id=public/default/function1]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Replace the placeholder variables with the actual values that you can get when setting up client tools.

admin-url: the HTTP service URL of your Pulsar cluster.
privateKey: the path to the downloaded OAuth2 key file.
issuerUrl: the URL of the OAuth2 issuer.
audience: the Uniform Resource Name (URN), which is a combination of the urn:sn:pulsar, your organization name, and your Pulsar instance name.
- ${orgName}: the name of your organization.
- ${instanceName}: the name of your instance.

If you would like to create a function configuration using the REST API you can do so using CURL.

curl -X POST ${WEB_SERVICE_URL}/admin/v3/functions/public/default/${FUNCTION_NAME} \
  -H 'Authorization: Bearer ${TOKEN}' \
  -H "Content-Type: multipart/form-data" \
  -F 'functionConfig={"name": "${FUNCTION_NAME}", "tenant": "public", "namespace": "default", "runtime": "PYTHON", "py": "function://public/default/${PACKAGE_NAME}@${PACKAGE_VERSION}", "output": "public/default/output-test", "inputs": ["public/default/input"], "className": "exclamation", "customRuntimeOptions": "{\"genericKind\": \"wasm\"}"};type=application/json' \
  -F 'url=function://public/default/${PACKAGE_NAME}@${PACKAGE_VERSION}'

Since Pulsar doesn’t support WASM runtime, we need to use --py to specify the function file and specify the --custom-runtime-options '{"genericKind": "wasm"}' to make it work.

The function is assumed to be already uploaded at this point. If you have not uploaded the function, change the url parameter to be your local filepath. This will look something like the following.

-F 'url=file://$YOUR_LCOAL_FUNCTION_FILE'

You should see something like this:

Created successfully

Replace the placeholder variables with the actual values that you can get when setting up client tools.

WEB_SERVICE_URL: the HTTPS service URL of your Pulsar cluster.
TOKEN: a valid token to interact with your Pulsar cluster.
FUNCTION_NAME: the name of your function.
FUNCTION_VERSION: the version of your function you want to deploy e.g. 1.12.

For details about Pulsar function configurations, see Pulsar function configurations.

What’s next?

Learn how to manage functions.
Learn how to configure stateful functions.
Learn how to monitor functions.
Reference common configurations.

Get Started

Clusters

Data Streams

Process

Connect

Lakehouse

Governance

Pulsar Clients

MQTT Clients

Tools

Pulsar Changelogs

Develop Pulsar Functions in WASM(Private Preview)

Develop

Feature Matrix

Input Arguments

Output Arguments

Context

Other

Package

Deploy

(Optional) Upload your function file to Pulsar

Create

What’s next?

​Develop

​Feature Matrix

​Input Arguments

​Output Arguments

​Context

​Other

​Package

​Deploy

​(Optional) Upload your function file to Pulsar

​Create

​What’s next?

Develop

Feature Matrix

Input Arguments

Output Arguments

Context

Other

Package

Deploy

(Optional) Upload your function file to Pulsar

Create

What’s next?