StreamNative Weekly Release Notes v2.11.1.0

Download

Distributions

• https://github.com/streamnative/pulsar/releases/tag/v2.11.1.0

Packages

• Maven Central

Images

• sn-platform

General Changes

Apache Pulsar

(#20050) [fix] [ml] Fix uncompleted future when remove cursor

(#20046) [refactor][broker] Use AuthenticationParameters for rest producer

(#20031) [fix] [cli] Fix Broker crashed by too much memory usage of pulsar tools

(#20014) [fix][proxy] Fix connection read timeout handling in Pulsar Proxy

(#20011) [improve][proxy] Implement graceful shutdown for Pulsar Proxy

(#20005) [fix][build] Suppress Guava CVE-2020-8908 in OWASP dependency check

(#19991) [fix][build] Client modules should be built with Java 8

(#19989) [fix][broker] Only validate superuser access if authz enabled

(#19976) [improve][txn] Cleanup how superusers abort txns

(#19975) [refactor][fn] Use AuthorizationServer more in Function Worker API

(#19942) [fix] [admin] fix incorrect state replication.connected on API partitioned-topic stat

(#19934) [fix][broker] Return if AbstractDispatcherSingleActiveConsumer closed

(#19930) [fix][client] Fix DeadLetterProducer creation callback blocking client io thread.

(#19879) [fix] [admin] Make response code to 400 instead of 500 when delete topic fails due to enabled geo-replication

(#19875) [fix][broker] Fix NPE when update topic policy.

(#19859) [fix][meta] Follow up #19817, Fix race condition between ResourceLock update and invalidation

(#19844) [fix][meta] Follow up #19817, Fix race condition between ResourceLock update and invalidation

(#19839) [improve][broker] Follow up #19230 to tighten the validation scope (#19234)

(#19830) [improve][broker] Authorize originalPrincipal when provided

(#19825) [fix] [ml] make the result of delete cursor is success if cursor is deleted

(#19817) [fix][meta] Fixed race condition between ResourceLock update and invalidation

(#19812) [improve] Upgrade bookkeeper to 4.15.4

(#19800) [fix][broker] Copy subscription properties during updating the topic partition number.

(#19761) [fix][meta] Fix close borrowed executor

(#19759) [fix][broker] Fix issue where msgRateExpired may not refresh forever

(#19754) [fix][meta] Fix deadlock causes session notification not to work

(#19746) [fix][broker] Fix potential exception cause the policy service init fail.

(#19735) [fix] [broker] delete topic failed if disabled system topic

(#19733) [fix][client] Fix topic list watcher fail log

(#19727) [fix][broker] Fix index generator is not rollback after entries are failed added.

(#19700) [improve][cli]Upgrade python to python3

(#19696) [fix][cli] Fix Pulsar admin tool is ignoring tls-trust-cert path arg

(#19692) [fix] [broker] Topic close failure leaves subscription in a permanent fence state

(#19686) [fix][broker][branch-2.11] Fix geo-replication admin

(#19674) [improve] Simplify enabling Broker, WS Proxy hostname verification

(#19667) [fix][broker] Filter system topic when getting topic list by binary proto.

(#19662) [feat][txn] Support for idempotent commit and abort, Solution1.

(#19632) [fix][ci][branch-2.11] Fix broken CI

(#19597) [fix][client] Broker address resolution wrong if connect through a multi-dns names proxy

(#19594) [fix][test] ProxyWithAuthorizationTest remove SAN from test certs

(#19588) [improve][broker][branch-2.11] Improve tls config on replication client and cluster cli

(#19585) [fix] [client] fix memory leak if enabled pooled messages

(#19581) [fix][txn]fix receive duplicated messages due to pendingAcks in PendingAckHandle

(#19560) [fix][broker] Fixed history load not releasing

(#19557) [fix][broker] Allow proxy to pass same role for authRole and originalRole

(#19548) [fix][broker] Fix geo-replication admin

(#19545) [fix] [test] Wrong mock-fail of the test ManagedLedgerErrorsTest.recoverLongTimeAfterMultipleWriteErrors

(#19541) [fix][broker] Terminate the async call chain when the condition isn't met for resetCursor

(#19540) [fix][proxy] Fix using wrong client version in pulsar proxy

(#19534) [improve][broker] Use shrink map for trackerCache

(#19532) [fix] [broker] Make the service name resolver cache of PulsarWebResource expire after access

(#19519) [fix][authentication] Store the original authentication data

(#19517) [fix][broker] Copy command fields and fix potential thread-safety in ServerCnx

(#19510) [fix][client] Fix authentication not update after changing the serviceUrl

(#19507) [fix][broker] Make ServerCnx#originalAuthData volatile

(#19506) [fix][broker] Make authentication refresh threadsafe

(#19505) [fix] [broker] Incorrect service name selection logic

(#19485) [fix][proxy] Fix JKS TLS transport

(#19483) [fix][client] Fix load the trust store file

(#19481) [fix][fn] Fix k8s merge runtime opts bug

(#19467) [improve][txn] Allow superusers to abort transactions

(#19455) [improve][broker] Require authRole is proxyRole to set originalPrincipal

(#19446) [fix][broker] Expect msgs after server initiated CloseProducer

(#19444) [fix] [ml] topic load fail by ledger lost

(#19439) [revert][misc] "modify check waitingForPingResponse with volatile (#12615)"

(#19430) [improve][broker] Add test to verify authRole cannot change

(#19428) [fix][ml] Reset individualDeletedMessagesSerializedSize after acked all messages.

(#19426) [improve] Upgrade lombok to 1.8.26

(#19425) [improve] Upgrade to zk 3.8.1

(#19420) [fix][broker] Fix loadbalance score calculation problem

(#19417) [improve][misc] Upgrade Netty to 4.1.87.Final

(#19415) [fix][ml] Fix potential NPE cause future never complete.

(#19412) [fix][authorization] Fix the return value of canConsumeAsync

(#19405) [improve] [broker] Print warn log if compaction failure

(#19404) [fix] [ml] Fix the incorrect total size if use ML interceptor

(#19399) [fix][broker] Replace sync call in async call chain of AdminResource#internalCreatePartitionedTopic

(#19395) [fix][broker] Release Netty buffer in finally block in ServerCnx#handleLastMessageIdFromCompactedLedger

(#19392) [fix][client] Fix async completion in ConsumerImpl#processPossibleToDLQ

(#19391) [fix][broker] Replace sync method in NamespacesBase#internalDeleteNamespaceBundleAsync

(#19389) [fix][broker] Fix concurrency bug in PersistentTopicsBase#internalGetReplicatedSubscriptionStatus

(#19384) [fix] Close TransactionBuffer when create persistent topic timeout

(#19377) [fix][io] Update Elasticsearch sink idle cnx timeout to 30s

(#19355) [fix] [ml] messagesConsumedCounter of NonDurableCursor was initialized incorrectly

(#19354) [fix][client] Fix reader listener can't auto ack with pooled message.

(#19346) [fix] [ml] The atomicity of multiple fields of ml is broken

(#19341) [improve][broker] Added isActive in ManagedCursorImpl

(#19333) [improve] Upgrade wildfly-elytron (used by debezium) to fix CVE-2022-3143

(#19331) [fix][proxy] Only go to connecting state once

(#19327) [fix][client] Set fields earlier for correct ClientCnx initialization

(#19312) [improve][broker] ServerCnx: go to Failed state when auth fails

(#19308) [improve][txn] Handle changeToReadyState failure correctly in TC client

(#19284) [fix][broker] AbstractBatchedMetadataStore - use AlreadyClosedException instead of IllegalStateException

(#19275) [fix] [ml] Topics stats shows msgBacklog but there reality no backlog

(#19270) [cleanup][broker] Validate originalPrincipal earlier in ServerCnx

(#19266) [fix][sec] Exclude log4j from openmldb

(#19258) [fix][txn] Catch and log runtime exceptions in async operations

(#19242) [fix] [broker] Counter of pending send messages in Replicator incorrect if schema future not complete

(#19234) [improve][broker] Follow up #19230 to tighten the validation scope

(#19230) [fix][broker] Support deleting partitioned topics with the keyword -partition-

(#19223) [fix][broker] Copy subscription properties during updating the topic partition number.

(#19203) [improve][websocket] Add ping support

(#19202) [fix][doc] Remove lombok plugin and define FunctionRecordBuilder explicitly

(#19201) [fix][txn] fix txn coordinator recover handle committing and aborting txn race condition.

(#19199) [fix][broker] Fix race condition while updating partition number

(#19191) [fix][client] Fix producer could send timeout when enable batching

(#19184) [fix][broker] Pass subName for subscription operations in ServerCnx

(#19170) [fix][build] Upgrade dependency-check-maven plugin to fix broken OWASP check

(#19167) [improve][broker] Add ref count for sticky hash to optimize the performance of Key_Shared subscription

(#19157) [fix][broker] Close transactionBuffer after MessageDeduplication#checkStatus failed

(#19156) [fix][broker] Support zookeeper read-only config.

(#19137) [fix][txn] Always send correct transaction id in end txn response

(#19130) [improve][schema] Do not print error log with stacktrace for 404

(#19129) [fix][broker] Topic could be in fenced state forever if deletion fails

(#19128) [fix][broker] Fix consumer with schema cannot subscribe when AUTO_CONSUME consumers exist

(#19120) [fix][build] Resolve OWASP Dependency Check false positives

(#19105) [improve][sec] Suppress false positive OWASP reports

(#19041) [fix][sec] Upgrade woodstox to 5.4.0

(#19038) [fix][sec] Upgrade jettison to 1.5.3

(#19028) [fix][client] Prevent DNS reverse lookup when physical address is an IP address

(#19015) [improve][admin] Improve partitioned-topic condition evaluation

(#19003) [improve][broker] Add logs for why namespace bundle been split

(#18999) [fix][admin] Fix validatePersistencePolicies that Namespace/Topic persistent policies cannot set to < 0

(#18997) [improve][broker] Omit making a copy of CommandAck when there are no broker interceptors

(#18989) [fix][broker] Fix deadlock in PendingAckHandleImpl

(#18987) [fix][broker] Copy proto command fields into final variables in ServerCnx

(#18975) [fix][broker]fix multi invocation for ledger createComplete

(#18969) [fix] [tx] [branch-2.11] Transaction buffer recover blocked by readNext

(#18968) [improve] Add PulsarExceptionBase class that supports slf4j like parameterized string formatting

(#18943) [fix][txn] transaction pending ack store future not completely problem

(#18940) [fix][broker]Update interceptor handler exception

(#18922) [improve][sec] Remove snakeyaml dependency to suppress CVE-2022-1471

(#18883) [fix][broker] Check operstate when get physical NICs

(#18877) [fix] [broker] getLastMessageId returns a wrong batch index of last message if enabled read compacted

(#18844) [fix][fn] Typo in method name

(#18829) [fix][test][branch-2.10] Remove testGetTopic method

(#18826) [cherry-pick][branch-2.11] cherry-pick fixing can not delete namespace by force (#18307)

(#18824) [fix][broker] Fix uncompleted future when getting the topic policies of a deleted topic

(#18823) [fix][broker] Fix delete system topic clean topic policy

(#18804) [fix][broker] Fix namespace deletion if __change_events topic has not been created yet

(#18745) [fix][sql] Fix message without schema issue.

(#18729) [cleanup][broker] Simplify extract entryMetadata code in filterEntriesForConsumer

(#18701) [fix][broker] Fix duplicated schemas creation

(#18679) [fix][broker] transactional producer created failed due to read snapshot blocked.

(#18672) [fix][broker] Fix PulsarRegistrationClient and ZkRegistrationClient not aware rack info problem.

(#18664) [improve][test] Add test brokerReachThreshold for ThresholdShedderTest

(#18662) [improve][broker] Make Consumer#equals more effective

(#18659) [fix][broker] Create replicated subscriptions for new partitions when needed

(#18633) [fix][client] For exclusive subscriptions, if two consumers are created repeatedly, the second consumer will block

(#18577) [fix][io] ElasticSearch sink: align null fields behaviour

(#18576) [improve][broker] Adjust to sample the error log

(#18565) [fix][broker] DnsResolverUtil.TTL should be greater than zero

(#18491) [fix][client] Fix multi-topic consumer stuck after redeliver messages

(#18486) [fix][client] Avoid redelivering duplicated messages when batching is enabled

(#18454) [fix][client] Fix failover/exclusive consumer with batch cumulate ack issue.

(#18451) [fix][broker] Correctly set byte and message out totals per subscription

(#18421) [improve][client] Change the get lastMessageId to debug level

(#18416) [fix][broker] Fix open cursor with null-initialPosition result with earliest position

(#18407) [fix][broker]Fix failover/exclusive consumer cumulate ack not remove msg from UnAckedMessageTracker

(#18406) [fix][client] Fix possible npe

(#18405) [fix][client] Fixes batch_size not checked in MessageId#fromByteArrayWithTopic

(#18403) [fix][client] Fix the Windows absolute path not recognized in auth param string

(#18358) [fix][client] Set authentication when using loadConf in client and admin client

(#18345) [fix][client] Fix exception when calling loadConf on a ConsumerBuilder that has a KeySharedPolicy

(#18334) [fix][client] Support LocalDateTime Conversion

(#18307) [fix][broker] Fix can not delete namespace by force

(#18305) [fix][broker] In the trimDeletedEntries method, release the removed entry

(#18293) [fix][schema] Fix creating lots of versions for the same schema

(#18283) [fix][broker] fix delete_when_subscriptions_caught_up doesn't work while have active consumers

(#18268) [fix][client] Fix IllegalThreadStateException when using newThread in ExecutorProvider.ExtendedThreadFactory

(#18219) [fix] [pulsar-client] Fix pendingLookupRequestSemaphore leak when Ser…

(#18211) [improve][broker] Add UncaughtExceptionHandler for every thread pool

(#18199) [improve][broker] invalidate lock cache when Deleted NotificationType

(#18197) [improve][broker] Remove locallyAcquiredLock when removeOwnership

(#18192) [improve][broker] Support setting forceDeleteTenantAllowed dynamically

(#18181) [improve][broker] Support setting ForceDeleteNamespaceAllowed dynamically

(#18172) [fix][meta] fix getChildren in MemoryMetadataStore and EtcdMetadataStore

(#18164) [fix][broker] AVAILABLE_PERMITS_UPDATER not updated, when writeAndFlush fails.

(#18133) [fix][broker] PulsarRegistrationClient - implement getAllBookies and follow BookieServiceInfo updates

(#18130) [fix][broker] Fix update authentication data

(#18129) [fix][function] Fix invalid metric type "gauge "

(#18111) [fix][fn] fix function failed to start if no typeClassName provided in FunctionDetails

(#18107) [improve][client] Support MAX_ACK_GROUP_SIZE configurable

(#18074) [improve][schema] Change update schema auth from tenant to produce

(#18024) [fix][broker] Update the log print content of createSubscriptions

(#18006) [improve][broker] Reduce unnecessary persistence of markdelete

(#18000) [fix][broker] Fix the order of resource close in the InMemoryDelayedDeliveryTracker

(#17970) [fix][broker] Fix incorrect bundle split count metric

(#17957) [fix][broker]unify time unit at dropping the backlog on a topic

(#17951) [improve][connector] JDBC sink: allow any jdbc driver

(#17922) Make BookieId work with PulsarRegistrationDriver (second take)

(#17911) [fix][broker]Fix mutex never released when trimming

(#17907) Allow to configure and disable the size of lookahead for detecting fixed delays in messages

(#17901) [fix][admin] returns 4xx error when pulsar-worker-service is disabled and trying to access it

(#17890) [fix][loadbalance] Fix the wrong NIC speed rate unit.

(#17842) [improve][broker]Improve PersistentMessageExpiryMonitor expire speed when ledger not existed

(#17836) [fix][client] moving get sequenceId into the sync code segment

(#17833) [refactor][java] Unify the acknowledge process for batch and non-batch message IDs

(#17831) [fix][proxy] Fix refresh client auth

(#17826) [fix][cli] Check numMessages after incrementing counter

(#17822) [improve][java-client]Add init capacity for messages in BatchMessageContainerImpl

(#17792) [fix][doc]fix comments for exposeManagedLedgerMetricsInPrometheus field

(#17787) [Improve][auth]Update authentication failed metrics report

(#17779) [fix][sec] Bump snakeyaml to 1.32 for CVE-2022-38752

(#17750) [fix][cli] Quit PerformanceConsumer after receiving numMessages messages

(#17736) [bugfix] ManagedLedger: move to FENCED state in case of BadVersionException

(#17729) [fix][broker] Fix create ns

(#17704) [fix][metrics]wrong metrics text generated when label_cluster specified

(#17694) [fix][broker] Fix executeWithRetry result is null

(#17676) [fix][cli] Fix mbeans to json

(#17661) [fix][bookie] Correctly handle list configuration values

(#17659) [fix][build] duplicate entry when merging services

(#17658) [C++] Reset havePendingPingRequest flag for any data received from broker

(#17600) [improve][pulsar-proxy] Update proxy lookup throw exception type

(#17592) Issue 17588: Allow deletion of a namespace that was left in deleted status

(#17582) [improve][test] Remove WhiteBox for ElasticSearchSinkTests

(#17579) [improve][test] remove WhiteBox on MockZooKeeper

(#17569) [improve][ci] Remove post commit builds and add manual workflows trigger

(#17552) [ci] Move more tests to the flaky suite

(#17551) [ci] Skip run flaky tests suite for cpp-only or doc-only changes

(#17542) [ci] Extract FLAKY tests suite to a dedicated workflow

(#17541) [ci] Reduce runners load for pulls that affect only doc or cpp changes

(#17512) [fix][tiered-storage] Don't cleanup data when offload met Metastore exception

(#17491) [improve][ci] Optimize "Pulsar Bot" workflow

(#17490) [improve][ci] Remove "Cancel duplicate workflows" workflow

(#17487) [fix][flaky-test]NamespaceServiceTest.flaky/testModularLoadManagerRemoveBundleAndLoad

(#17462) [fix][license] Update the log4j version in the presto license file

(#17457) [fix][sec] bump snakeyaml to 1.31 fix CVE-2022-25857

(#17453) [improve][broker] Improve cursor.getNumberOfEntries if isUnackedRangesOpenCacheSetEnabled=true

(#17443) [fix][client]Duplicate messages when use MultiTopicsConsumerImpl

(#17426) [improve][cli] Using separate TLS config on the compactor

(#17423) [improve][txn] Add getState in transaction for client API

(#17419) [fix][broker] Remove timestamp from Prometheus metrics

(#17403) [improve][broker] Using handle instead of handleAsync to avoid using common pool thread

(#17393) [fix][broker] fix can not revoke permission after update topic partition

(#17375) [improve][admin] PulsarAdminBuilderImpl overrides timeout properties passed through config map

(#17374) [fix][broker]fix catching ConflictException when update topic partition

(#17370) [fix][broker] Fix Npe thrown by splitBundle

(#17353) [fix][flaky] Fix flakyness in testAutoSchemaFunction

(#17349) [improve][broker]Only create extended partitions when updating partition number

(#17322) [improve][schema]autoSkipNonRecoverableData work for schema Ledger

(#17319) [improve][ci] Replace test reporting with less verbose solution

(#17264) [improve][broker]Recover as much cursor data as possible, even if entry is invalid

(#17257) [fix][broker]ManagedLedger metrics fail cause of zero period

(#17256) [refactor][java] Improve docs and code quality about KeyValueSchema usages

(#17251) [fix][broker]fail to update partition meta of topic due to ConflictException: subscription already exists for topic

(#17231) [broker][fix]Fix update topic remove properties

(#17208) [improve][build] Avoid building image multiple times

(#17149) [fix][broker]fix arithmetic exception for LeastResourceUsageWithWeight strategy

(#17129) [improve][docker] Switch to Temurin JDK

(#17082) [fix][broker] Skip not connectable URL when get redirectionUrl

(#17033) [improve][admin] Unset namespace policy to improve deleting namespace.

(#17032) [fix][broker] Fix delivery the message that has been acknowledged

(#17009) [fix][txn] Correct the prompt message

(#16988) [improve][ci] Skip unnecessary tests when there are only cpp/python related changes

(#16981) [Fix][flaky-test] Fix testConsumeTxnMessage

(#16946) [Improve][broker]Reduce GetReplicatedSubscriptionStatus local REST call

(#16909) [fix][ml] fix NPE

(#16815) [improve] clean the empty topicAuthenticationMap in zk when revoke permission

(#16485) [pulsar-io] KCA: handle kafka's logical schemas

(#16225) [improve][broker] ServerCnx: log at warning level when topic not found

(#15640) [fix][storage] Autorecovery default reppDnsResolverClass to ZkBookieRackAffinityMapping

(#15558) [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)

(#15241) [refactor][broker] Arrange cleanup operation for topic creation

(#15140) [monitor][txn] Add metrics for transaction

(#14585) Issue 14583: [python client]: Improve garbage collection of producer/consumer/reader objects

(#14327) [feature][txn] Fix individual ack batch message with transaction abort redevlier duplicate messages

(#13563) [Improvement] Recycler should use io.netty.recycler.maxCapacityPerThread instead of io.netty.recycler.maxCapacity.default as maxCapacityPerThread configuration in Netty 4.1.x

(#13079) [Issue 12844][pulsar-client-cpp] Excessive locking cause significant performance degradation

KoP

(#1831) Increase timeout for KafkaListenerNameTest

(#1826) [Schema] Fix wrong response when querying schema by subject and version

(#1645) [bugfix] Fix regression with the upgrade to 2.8.x: Count corrently the number of partitions in ProduceRequest

(#1805) [Schema Registry] Fix the delete operation not work

(#1802) [Schema] Return 401 error when no HTTP authentication is configured

(#1801) Return a correct metadata response when topic is not found

(#1793) [test] Add kafka admin api unit test

(#1742) Improve getting started guide

(#1797) [fix][schema] Return JSON error message for failed request

pulsarctl

(#1038) Fix vulnerabilities

Function Mesh Worker Service

fix ci in branch 2.11

fix build branch 2.11

adopt new api branch 2.10.3

Bump function-mesh to v0.13.0

support start\stop\restar function\sink\source