Functions (Beta)

Apache Pulsar Functions are lightweight functions that consume messages from Pulsar topics, apply custom processing logic, and publish the results of the computation to other topics. With the initial release of this feature, you deploy your functions through the command line to StreamNative Cloud. You can also monitor your functions on the StreamNative Cloud Console, which visually displays all of your functions and their current state.

To learn more, watch the Pulsar Functions on StreamNative Cloud playlist.

Pulsar Functions are designed to perform lightweight stream processing. They are best for basic use cases that do not require the complexity of a full-stream processing engine.

• Simple per-message transformations for normalization, cleanup, or enriching with metadata.
• Simple aggregations like sums, counts, or averages for a single stream (over a short time period) and where duplicates are tolerated.
• Chained sequences of transformations on data in a single topic.

Pulsar Functions are computing infrastructure of Pulsar messaging system. With Pulsar Functions, you can create complex processing logic without deploying a separate neighboring system, such as Apache Storm, Apache Heron, or Apache Flink.

Pulsar Functions can be described as Lambda-style functions that are specifically designed to use Pulsar as a message bus.

For information about Pulsar Functions in general, see Pulsar Functions Overview.

Prerequisites

• Install and configure the snctl CLI tool.
• Install the pulsarctl CLI tool.
• Log in to StreamNative Cloud Console.
• Create a Pulsar cluster and connect to your Pulsar cluster using the pulsarctl CLI tool.
• Create a tenant and a namespace.

Prepare your environment

You need to prepare your environment before you submit a function to your Pulsar cluster.

Prepare service accounts

You need to create two service accounts. One is a normal service account granted with proper permissions (functions, produce, and consume). The other one must have the Super Admin access. The Super Admin service account is used to create a role binding for the normal service account.

To create a service account, follow these steps.

1. On the left navigation pane, click Service Accounts.

2. Click Create Service Account.

3. (Optional) Select Super Admin to grant the service account with Super admin access to a namespace or tenant.

4. Enter a name for the service account, and then click Confirm.

Get OAuth2 credential files of your service accounts

To get an OAuth2 credential file of a service account through the StreamNative Console, follow these steps.

1. On the left navigation pane, click Service Accounts.

2. In the row of the service account you want to use, in the Key File column, click the Download icon to download the OAuth2 credential file to your local directory.

   The OAuth2 credential file should be something like this:

   {
     "type": "SN_SERVICE_ACCOUNT",
     "client_id": "CLIENT_ID",
     "client_secret": "CLIENT_SECRET",
     "client_email": "[email protected]",
     "issuer_url": "https://auth.streamnative.cloud"
   }
   

Create a role binding for your normal service account

To grant the underlying infrastructure with access to the newly created service account's OAuth2 key file, you need to create a service account binding via UI.

Go to the Service Accounts tab and choose the service account you want to use for running the connector. Clicking on the right button and there willbe a Edit service account bindings option.

Click the Edit service account bindings, choose the desired pool member and confirm.

Now your connector is ready to use the service account in StreamNative environments.

Authorize your normal service account

To grant the service account permissions on the namespace level, follow these steps:

1. On the left navigation pane, in the Admin section, click Tenants/Namespaces.

2. On the Tenants page, select your tenant and namespace.

3. On your Namespace page, select the POLICY tab.

4. In the Authorization area, click ADD ROLE, and select the service account that you just created in the previous section.

5. On the drop-down menu below the service account, select the proper permissions to assign to the newly created service account. There are six permissions in total:

• consume: allow the service account to consume messages.
• produce: allow the service account to publish messages.
• functions: allow the service account to submit and manage functions.
• sinks: allow the service account to create and manage sink connectors.
• sources: allow the service account to create and manage source connectors.
• packages: allow the service account to upload and manage pulsar packages. If you want to submit a customized connector, then you will need to upload the connector’s JAR/NAR file first, which requires the packages permission.

Deploy a Pulsar function

Currently, you can deploy Pulsar Functions through the CLI tools. For detailed information on creating Pulsar Functions using the pulsarctl CLI tool, see work with Pulsar Functions using pulsarctl CLI tool.

View the function status on StreamNative Cloud Console

After you have successfully deployed a function, you can check its status, logs, and any exceptions through StreamNative Cloud Console.

1. Log in to StreamNative Cloud Console.

2. On the left navigation pane, in the Resources section, click Functions.

3. Click the status of the function you just deployed to view the details page. In the figure below, the status is Running.

   

4. On the functions detail page, click Logs to view a real-time log of the running function you selected. Click Exceptions to view any exceptions.