Elastic Beats Output to Apache Pulsar
Authored by
Support type
Apache License 2.0

Beat Output Pulsar

License FOSSA Status

This is an output implementation of elastic beats for support Filebeat, Metricbeat, Functionbeat, Winlogbeat, Journalbeat, Auditbeat to Apache Pulsar


This output is developed and tested using Apache Pulsar Client 2.4.0 and Beats 7.3.1

Download pulsar-beat-output

mkdir -p $GOPATH/src/github.com/streamnative/
cd $GOPATH/src/github.com/streamnative/
git clone https://github.com/streamnative/pulsar-beat-output
cd pulsar-beat-output


Build Filebeat

cd filebeat
go build .


Add following configuration to filebeat.yml

  url: "pulsar://localhost:6650"
  topic: my_topic
  name: test123

Start filebeat

./filebeat modules enable system
./filebeat modules list
./filebeat -c filebeat.yml -e

Build other beat

go build -o filebeat filebeat/filebeat.go
go build -o functionbeat functionbeat/functionbeat.go
go build -o journalbeat journalbeat/journalbeat.go
go build -o winlogbeat winlogbeat/winlogbeat.go
go build -o packetbeat packetbeat/packetbeat.go

Build and test with docker


Build Beat images

docker build -t pulsar-beat .

Create network

docker network create pulsar-beat

Start Pulsar service

docker run -d -it --network pulsar-beat -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-beat-standalone apachepulsar/pulsar:2.7.0 bin/pulsar standalone

Add following configuration to filebeat.yml

  url: "pulsar://pulsar-beat-standalone:6650"
  topic: my_topic
  name: test123

Start Filebeat

docker pull golang:1.17
docker run -it --network pulsar-beat --name filebeat golang:1.17 /bin/bash
git clone https://github.com/streamnative/pulsar-beat-output
cd pulsar-beat-output
go build -o filebeat filebeat/filebeat.go
chown -R root:root filebeat.yml test_module/modules.d/system.yml test_module/module/system
cp test_module/module/system/auth/test/test.log /var/log/messages.log
cp filebeat/filebeat filebeat.yml test_module
cd test_module
./filebeat modules enable system
./filebeat -c filebeat.yml -e

New open a window for consumer message

docker cp pulsar-client.py pulsar-beat-standalone:/pulsar
docker exec -it pulsar-beat-standalone /bin/bash
python pulsar-client.py

Now you can see the information collected from filebeat.



urlConfigure the service URL for the Pulsar servicepulsar://localhost:6650
certificate_pathpath of tls cert file""
private_key_pathpath of tls key file""
use_tlsWhether to turn on TLS, if to start, use protocol pulsar+sslfalse
tokenAccess token information of cluster""
token_file_pathThe file path where token is saved""
log_levelSetting the log level, available options(panic, fatal, error, warn, info, debug, trace)info
oauth2.enabledEnabled or disabled oauth2 authenticationfalse
oauth2.clientIdclient ID""
oauth2.issuerUrlURL of the authentication provider which allows the Pulsar client to obtain an access token""
oauth2.privateKeyURL of a JSON credentials file""
oauth2.audienceThe audience value is either the application (Client ID) for an ID Token or the API that is being called (API Identifier) for an Access Token""
oauth2.scopeScope is a mechanism in OAuth 2.0 to limit an application's access to a user's account""


topicSpecify the topic this producer will be publishing on.""
nameSpecify a name for the producer""
send_timeoutSet the send timeout30s
block_if_queue_fullSet whether the send and sendAsync operations should block when the outgoing message queue is full.false
batching_max_messagesmaximum number of messages in a batch1000
batching_max_publish_delaythe batch delay1ms
message_routing_modethe message routing mode, SinglePartition,RoundRobinPartition, CustomPartition(0,1,2)1


case-insensitive import collision: "github.com/datadog/zstd" and "github.com/DataDog/zstd"

/root/go/pkg/mod/github.com/apache/[email protected]/pulsar/internal/compression/zstd_cgo.go:27:2: case-insensitive import collision: "github.com/datadog/zstd" and "github.com/DataDog/zstd"

Replace zstd_cgo.go file

cp zstd_cgo.go /root/go/pkg/mod/github.com/apache/[email protected]/pulsar/internal/compression/zstd_cgo.go

Install Pulsar Go Client

Reference https://pulsar.apache.org/docs/en/client-libraries-go/ .

If you encounter problems with dynamic libraries´╝îplease reference:https://pulsar.apache.org/docs/en/client-libraries-cpp/.

Build Packetbeat

Reference https://github.com/elastic/beats/issues/11054.

Build journalbeat.go

systemd/sd-journal.h: No such file or directory
apt-get update
apt-get install libsystemd-dev

Build auditbeat.go

vendor/github.com/elastic/beats/x-pack/auditbeat/module/system/package/rpm_linux.go:23:24: fatal error: rpm/rpmlib.h: No such file or directory
aapt-get install librpm-dev

Start beat

Exiting: error loading config file: config file ("filebeat.yml") must be owned by the user identifier (uid=0) or root
chown -R root:root filebeat.yml


FOSSA Status