1. References
  2. StreamNative Pulsar Changelogs
  3. v2.11

StreamNative Weekly Release Notes v2.11.1.0

StreamNative Weekly Release Notes v2.11.1.0

Download

Distributions

  • https://github.com/streamnative/pulsar/releases/tag/v2.11.1.0

Packages

Images

General Changes

Apache Pulsar

[fix] [ml] Fix uncompleted future when remove cursor

[refactor][broker] Use AuthenticationParameters for rest producer

[fix] [cli] Fix Broker crashed by too much memory usage of pulsar tools

[fix][proxy] Fix connection read timeout handling in Pulsar Proxy

[improve][proxy] Implement graceful shutdown for Pulsar Proxy

[fix][build] Suppress Guava CVE-2020-8908 in OWASP dependency check

[fix][build] Client modules should be built with Java 8

[fix][broker] Only validate superuser access if authz enabled

[improve][txn] Cleanup how superusers abort txns

[refactor][fn] Use AuthorizationServer more in Function Worker API

[fix] [admin] fix incorrect state replication.connected on API partitioned-topic stat

[fix][broker] Return if AbstractDispatcherSingleActiveConsumer closed

[fix][client] Fix DeadLetterProducer creation callback blocking client io thread.

[fix] [admin] Make response code to 400 instead of 500 when delete topic fails due to enabled geo-replication

[fix][broker] Fix NPE when update topic policy.

[fix][meta] Follow up #19817, Fix race condition between ResourceLock update and invalidation

[fix][meta] Follow up #19817, Fix race condition between ResourceLock update and invalidation

[improve][broker] Follow up #19230 to tighten the validation scope (#19234)

[improve][broker] Authorize originalPrincipal when provided

[fix] [ml] make the result of delete cursor is success if cursor is deleted

[fix][meta] Fixed race condition between ResourceLock update and invalidation

[improve] Upgrade bookkeeper to 4.15.4

[fix][broker] Copy subscription properties during updating the topic partition number.

[fix][meta] Fix close borrowed executor

[fix][broker] Fix issue where msgRateExpired may not refresh forever

[fix][meta] Fix deadlock causes session notification not to work

[fix][broker] Fix potential exception cause the policy service init fail.

[fix] [broker] delete topic failed if disabled system topic

[fix][client] Fix topic list watcher fail log

[fix][broker] Fix index generator is not rollback after entries are failed added.

[improve][cli]Upgrade python to python3

[fix][cli] Fix Pulsar admin tool is ignoring tls-trust-cert path arg

[fix] [broker] Topic close failure leaves subscription in a permanent fence state

[fix][broker][branch-2.11] Fix geo-replication admin

[improve] Simplify enabling Broker, WS Proxy hostname verification

[fix][broker] Filter system topic when getting topic list by binary proto.

[feat][txn] Support for idempotent commit and abort, Solution1.

[fix][ci][branch-2.11] Fix broken CI

[fix][client] Broker address resolution wrong if connect through a multi-dns names proxy

[fix][test] ProxyWithAuthorizationTest remove SAN from test certs

[improve][broker][branch-2.11] Improve tls config on replication client and cluster cli

[fix] [client] fix memory leak if enabled pooled messages

[fix][txn]fix receive duplicated messages due to pendingAcks in PendingAckHandle

[fix][broker] Fixed history load not releasing

[fix][broker] Allow proxy to pass same role for authRole and originalRole

[fix][broker] Fix geo-replication admin

[fix] [test] Wrong mock-fail of the test ManagedLedgerErrorsTest.recoverLongTimeAfterMultipleWriteErrors

[fix][broker] Terminate the async call chain when the condition isn't met for resetCursor

[fix][proxy] Fix using wrong client version in pulsar proxy

[improve][broker] Use shrink map for trackerCache

[fix] [broker] Make the service name resolver cache of PulsarWebResource expire after access

[fix][authentication] Store the original authentication data

[fix][broker] Copy command fields and fix potential thread-safety in ServerCnx

[fix][client] Fix authentication not update after changing the serviceUrl

[fix][broker] Make ServerCnx#originalAuthData volatile

[fix][broker] Make authentication refresh threadsafe

[fix] [broker] Incorrect service name selection logic

[fix][proxy] Fix JKS TLS transport

[fix][client] Fix load the trust store file

[fix][fn] Fix k8s merge runtime opts bug

[improve][txn] Allow superusers to abort transactions

[improve][broker] Require authRole is proxyRole to set originalPrincipal

[fix][broker] Expect msgs after server initiated CloseProducer

[fix] [ml] topic load fail by ledger lost

[revert][misc] "modify check waitingForPingResponse with volatile (#12615)"

[improve][broker] Add test to verify authRole cannot change

[fix][ml] Reset individualDeletedMessagesSerializedSize after acked all messages.

[improve] Upgrade lombok to 1.8.26

[improve] Upgrade to zk 3.8.1

[fix][broker] Fix loadbalance score calculation problem

[improve][misc] Upgrade Netty to 4.1.87.Final

[fix][ml] Fix potential NPE cause future never complete.

[fix][authorization] Fix the return value of canConsumeAsync

[improve] [broker] Print warn log if compaction failure

[fix] [ml] Fix the incorrect total size if use ML interceptor

[fix][broker] Replace sync call in async call chain of AdminResource#internalCreatePartitionedTopic

[fix][broker] Release Netty buffer in finally block in ServerCnx#handleLastMessageIdFromCompactedLedger

[fix][client] Fix async completion in ConsumerImpl#processPossibleToDLQ

[fix][broker] Replace sync method in NamespacesBase#internalDeleteNamespaceBundleAsync

[fix][broker] Fix concurrency bug in PersistentTopicsBase#internalGetReplicatedSubscriptionStatus

[fix] Close TransactionBuffer when create persistent topic timeout

[fix][io] Update Elasticsearch sink idle cnx timeout to 30s

[fix] [ml] messagesConsumedCounter of NonDurableCursor was initialized incorrectly

[fix][client] Fix reader listener can't auto ack with pooled message.

[fix] [ml] The atomicity of multiple fields of ml is broken

[improve][broker] Added isActive in ManagedCursorImpl

[improve] Upgrade wildfly-elytron (used by debezium) to fix CVE-2022-3143

[fix][proxy] Only go to connecting state once

[fix][client] Set fields earlier for correct ClientCnx initialization

[improve][broker] ServerCnx: go to Failed state when auth fails

[improve][txn] Handle changeToReadyState failure correctly in TC client

[fix][broker] AbstractBatchedMetadataStore - use AlreadyClosedException instead of IllegalStateException

[fix] [ml] Topics stats shows msgBacklog but there reality no backlog

[cleanup][broker] Validate originalPrincipal earlier in ServerCnx

[fix][sec] Exclude log4j from openmldb

[fix][txn] Catch and log runtime exceptions in async operations

[fix] [broker] Counter of pending send messages in Replicator incorrect if schema future not complete

[improve][broker] Follow up #19230 to tighten the validation scope

[fix][broker] Support deleting partitioned topics with the keyword -partition-

[fix][broker] Copy subscription properties during updating the topic partition number.

[improve][websocket] Add ping support

[fix][doc] Remove lombok plugin and define FunctionRecordBuilder explicitly

[fix][txn] fix txn coordinator recover handle committing and aborting txn race condition.

[fix][broker] Fix race condition while updating partition number

[fix][client] Fix producer could send timeout when enable batching

[fix][broker] Pass subName for subscription operations in ServerCnx

[fix][build] Upgrade dependency-check-maven plugin to fix broken OWASP check

[improve][broker] Add ref count for sticky hash to optimize the performance of Key_Shared subscription

[fix][broker] Close transactionBuffer after MessageDeduplication#checkStatus failed

[fix][broker] Support zookeeper read-only config.

[fix][txn] Always send correct transaction id in end txn response

[improve][schema] Do not print error log with stacktrace for 404

[fix][broker] Topic could be in fenced state forever if deletion fails

[fix][broker] Fix consumer with schema cannot subscribe when AUTO_CONSUME consumers exist

[fix][build] Resolve OWASP Dependency Check false positives

[improve][sec] Suppress false positive OWASP reports

[fix][sec] Upgrade woodstox to 5.4.0

[fix][sec] Upgrade jettison to 1.5.3

[fix][client] Prevent DNS reverse lookup when physical address is an IP address

[improve][admin] Improve partitioned-topic condition evaluation

[improve][broker] Add logs for why namespace bundle been split

[fix][admin] Fix validatePersistencePolicies that Namespace/Topic persistent policies cannot set to < 0

[improve][broker] Omit making a copy of CommandAck when there are no broker interceptors

[fix][broker] Fix deadlock in PendingAckHandleImpl

[fix][broker] Copy proto command fields into final variables in ServerCnx

[fix][broker]fix multi invocation for ledger createComplete

[fix] [tx] [branch-2.11] Transaction buffer recover blocked by readNext

[improve] Add PulsarExceptionBase class that supports slf4j like parameterized string formatting

[fix][txn] transaction pending ack store future not completely problem

[fix][broker]Update interceptor handler exception

[improve][sec] Remove snakeyaml dependency to suppress CVE-2022-1471

[fix][broker] Check operstate when get physical NICs

[fix] [broker] getLastMessageId returns a wrong batch index of last message if enabled read compacted

[fix][fn] Typo in method name

[fix][test][branch-2.10] Remove testGetTopic method

[cherry-pick][branch-2.11] cherry-pick fixing can not delete namespace by force (#18307)

[fix][broker] Fix uncompleted future when getting the topic policies of a deleted topic

[fix][broker] Fix delete system topic clean topic policy

[fix][broker] Fix namespace deletion if __change_events topic has not been created yet

[fix][sql] Fix message without schema issue.

[cleanup][broker] Simplify extract entryMetadata code in filterEntriesForConsumer

[fix][broker] Fix duplicated schemas creation

[fix][broker] transactional producer created failed due to read snapshot blocked.

[fix][broker] Fix PulsarRegistrationClient and ZkRegistrationClient not aware rack info problem.

[improve][test] Add test brokerReachThreshold for ThresholdShedderTest

[improve][broker] Make Consumer#equals more effective

[fix][broker] Create replicated subscriptions for new partitions when needed

[fix][client] For exclusive subscriptions, if two consumers are created repeatedly, the second consumer will block

[fix][io] ElasticSearch sink: align null fields behaviour

[improve][broker] Adjust to sample the error log

[fix][broker] DnsResolverUtil.TTL should be greater than zero

[fix][client] Fix multi-topic consumer stuck after redeliver messages

[fix][client] Avoid redelivering duplicated messages when batching is enabled

[fix][client] Fix failover/exclusive consumer with batch cumulate ack issue.

[fix][broker] Correctly set byte and message out totals per subscription

[improve][client] Change the get lastMessageId to debug level

[fix][broker] Fix open cursor with null-initialPosition result with earliest position

[fix][broker]Fix failover/exclusive consumer cumulate ack not remove msg from UnAckedMessageTracker

[fix][client] Fix possible npe

[fix][client] Fixes batch_size not checked in MessageId#fromByteArrayWithTopic

[fix][client] Fix the Windows absolute path not recognized in auth param string

[fix][client] Set authentication when using loadConf in client and admin client

[fix][client] Fix exception when calling loadConf on a ConsumerBuilder that has a KeySharedPolicy

[fix][client] Support LocalDateTime Conversion

[fix][broker] Fix can not delete namespace by force

[fix][broker] In the trimDeletedEntries method, release the removed entry

[fix][schema] Fix creating lots of versions for the same schema

[fix][broker] fix delete_when_subscriptions_caught_up doesn't work while have active consumers

[fix][client] Fix IllegalThreadStateException when using newThread in ExecutorProvider.ExtendedThreadFactory

[fix] [pulsar-client] Fix pendingLookupRequestSemaphore leak when Ser…

[improve][broker] Add UncaughtExceptionHandler for every thread pool

[improve][broker] invalidate lock cache when Deleted NotificationType

[improve][broker] Remove locallyAcquiredLock when removeOwnership

[improve][broker] Support setting forceDeleteTenantAllowed dynamically

[improve][broker] Support setting ForceDeleteNamespaceAllowed dynamically

[fix][meta] fix getChildren in MemoryMetadataStore and EtcdMetadataStore

[fix][broker] AVAILABLE_PERMITS_UPDATER not updated, when writeAndFlush fails.

[fix][broker] PulsarRegistrationClient - implement getAllBookies and follow BookieServiceInfo updates

[fix][broker] Fix update authentication data

[fix][function] Fix invalid metric type "gauge "

[fix][fn] fix function failed to start if no typeClassName provided in FunctionDetails

[improve][client] Support MAX_ACK_GROUP_SIZE configurable

[improve][schema] Change update schema auth from tenant to produce

[fix][broker] Update the log print content of createSubscriptions

[improve][broker] Reduce unnecessary persistence of markdelete

[fix][broker] Fix the order of resource close in the InMemoryDelayedDeliveryTracker

[fix][broker] Fix incorrect bundle split count metric

[fix][broker]unify time unit at dropping the backlog on a topic

[improve][connector] JDBC sink: allow any jdbc driver

Make BookieId work with PulsarRegistrationDriver (second take)

[fix][broker]Fix mutex never released when trimming

Allow to configure and disable the size of lookahead for detecting fixed delays in messages

[fix][admin] returns 4xx error when pulsar-worker-service is disabled and trying to access it

[fix][loadbalance] Fix the wrong NIC speed rate unit.

[improve][broker]Improve PersistentMessageExpiryMonitor expire speed when ledger not existed

[fix][client] moving get sequenceId into the sync code segment

[refactor][java] Unify the acknowledge process for batch and non-batch message IDs

[fix][proxy] Fix refresh client auth

[fix][cli] Check numMessages after incrementing counter

[improve][java-client]Add init capacity for messages in BatchMessageContainerImpl

[fix][doc]fix comments for exposeManagedLedgerMetricsInPrometheus field

[Improve][auth]Update authentication failed metrics report

[fix][sec] Bump snakeyaml to 1.32 for CVE-2022-38752

[fix][cli] Quit PerformanceConsumer after receiving numMessages messages

[bugfix] ManagedLedger: move to FENCED state in case of BadVersionException

[fix][broker] Fix create ns

[fix][metrics]wrong metrics text generated when label_cluster specified

[fix][broker] Fix executeWithRetry result is null

[fix][cli] Fix mbeans to json

[fix][bookie] Correctly handle list configuration values

[fix][build] duplicate entry when merging services

[C++] Reset havePendingPingRequest flag for any data received from broker

[improve][pulsar-proxy] Update proxy lookup throw exception type

Issue 17588: Allow deletion of a namespace that was left in deleted status

[improve][test] Remove WhiteBox for ElasticSearchSinkTests

[improve][test] remove WhiteBox on MockZooKeeper

[improve][ci] Remove post commit builds and add manual workflows trigger

[ci] Move more tests to the flaky suite

[ci] Skip run flaky tests suite for cpp-only or doc-only changes

[ci] Extract FLAKY tests suite to a dedicated workflow

[ci] Reduce runners load for pulls that affect only doc or cpp changes

[fix][tiered-storage] Don't cleanup data when offload met Metastore exception

[improve][ci] Optimize "Pulsar Bot" workflow

[improve][ci] Remove "Cancel duplicate workflows" workflow

[fix][flaky-test]NamespaceServiceTest.flaky/testModularLoadManagerRemoveBundleAndLoad

[fix][license] Update the log4j version in the presto license file

[fix][sec] bump snakeyaml to 1.31 fix CVE-2022-25857

[improve][broker] Improve cursor.getNumberOfEntries if isUnackedRangesOpenCacheSetEnabled=true

[fix][client]Duplicate messages when use MultiTopicsConsumerImpl

[improve][cli] Using separate TLS config on the compactor

[improve][txn] Add getState in transaction for client API

[fix][broker] Remove timestamp from Prometheus metrics

[improve][broker] Using handle instead of handleAsync to avoid using common pool thread

[fix][broker] fix can not revoke permission after update topic partition

[improve][admin] PulsarAdminBuilderImpl overrides timeout properties passed through config map

[fix][broker]fix catching ConflictException when update topic partition

[fix][broker] Fix Npe thrown by splitBundle

[fix][flaky] Fix flakyness in testAutoSchemaFunction

[improve][broker]Only create extended partitions when updating partition number

[improve][schema]autoSkipNonRecoverableData work for schema Ledger

[improve][ci] Replace test reporting with less verbose solution

[improve][broker]Recover as much cursor data as possible, even if entry is invalid

[fix][broker]ManagedLedger metrics fail cause of zero period

[refactor][java] Improve docs and code quality about KeyValueSchema usages

[fix][broker]fail to update partition meta of topic due to ConflictException: subscription already exists for topic

[broker][fix]Fix update topic remove properties

[improve][build] Avoid building image multiple times

[fix][broker]fix arithmetic exception for LeastResourceUsageWithWeight strategy

[improve][docker] Switch to Temurin JDK

[fix][broker] Skip not connectable URL when get redirectionUrl

[improve][admin] Unset namespace policy to improve deleting namespace.

[fix][broker] Fix delivery the message that has been acknowledged

[fix][txn] Correct the prompt message

[improve][ci] Skip unnecessary tests when there are only cpp/python related changes

[Fix][flaky-test] Fix testConsumeTxnMessage

[Improve][broker]Reduce GetReplicatedSubscriptionStatus local REST call

[fix][ml] fix NPE

[improve] clean the empty topicAuthenticationMap in zk when revoke permission

[pulsar-io] KCA: handle kafka's logical schemas

[improve][broker] ServerCnx: log at warning level when topic not found

[fix][storage] Autorecovery default reppDnsResolverClass to ZkBookieRackAffinityMapping

[fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)

[refactor][broker] Arrange cleanup operation for topic creation

[monitor][txn] Add metrics for transaction

Issue 14583: [python client]: Improve garbage collection of producer/consumer/reader objects

[feature][txn] Fix individual ack batch message with transaction abort redevlier duplicate messages

[Improvement] Recycler should use io.netty.recycler.maxCapacityPerThread instead of io.netty.recycler.maxCapacity.default as maxCapacityPerThread configuration in Netty 4.1.x

[Issue 12844][pulsar-client-cpp] Excessive locking cause significant performance degradation

KoP

Increase timeout for KafkaListenerNameTest

[Schema] Fix wrong response when querying schema by subject and version

[bugfix] Fix regression with the upgrade to 2.8.x: Count corrently the number of partitions in ProduceRequest

[Schema Registry] Fix the delete operation not work

[Schema] Return 401 error when no HTTP authentication is configured

Return a correct metadata response when topic is not found

[test] Add kafka admin api unit test

Improve getting started guide

[fix][schema] Return JSON error message for failed request

pulsarctl

Fix vulnerabilities

Function Mesh Worker Service

fix ci in branch 2.11

fix build branch 2.11

adopt new api branch 2.10.3

Bump function-mesh to v0.13.0

support start\stop\restar function\sink\source

Previous
v2.11.0.5