StreamNative Weekly Release Notes v2.11.1.0
Download
Distributions
Packages
Images
General Changes
Apache Pulsar
[fix] [ml] Fix uncompleted future when remove cursor [refactor][broker] Use AuthenticationParameters for rest producer [fix] [cli] Fix Broker crashed by too much memory usage of pulsar tools [fix][proxy] Fix connection read timeout handling in Pulsar Proxy [improve][proxy] Implement graceful shutdown for Pulsar Proxy [fix][build] Suppress Guava CVE-2020-8908 in OWASP dependency check [fix][build] Client modules should be built with Java 8 [fix][broker] Only validate superuser access if authz enabled [improve][txn] Cleanup how superusers abort txns [refactor][fn] Use AuthorizationServer more in Function Worker API [fix] [admin] fix incorrect state replication.connected on API partitioned-topic stat [fix][broker] Return if AbstractDispatcherSingleActiveConsumer closed [fix][client] Fix DeadLetterProducer creation callback blocking client io thread. [fix] [admin] Make response code to 400 instead of 500 when delete topic fails due to enabled geo-replication [fix][broker] Fix NPE when update topic policy. [fix][meta] Follow up #19817, Fix race condition between ResourceLock update and invalidation [fix][meta] Follow up #19817, Fix race condition between ResourceLock update and invalidation [improve][broker] Follow up #19230 to tighten the validation scope (#19234) [improve][broker] Authorize originalPrincipal when provided [fix] [ml] make the result of delete cursor is success if cursor is deleted [fix][meta] Fixed race condition between ResourceLock update and invalidation [improve] Upgrade bookkeeper to 4.15.4 [fix][broker] Copy subscription properties during updating the topic partition number. [fix][meta] Fix close borrowed executor [fix][broker] Fix issue where msgRateExpired may not refresh forever [fix][meta] Fix deadlock causes session notification not to work [fix][broker] Fix potential exception cause the policy service init fail. [fix] [broker] delete topic failed if disabled system topic [fix][client] Fix topic list watcher fail log [fix][broker] Fix index generator is not rollback after entries are failed added. [improve][cli]Upgrade python to python3 [fix][cli] Fix Pulsar admin tool is ignoring tls-trust-cert path arg [fix] [broker] Topic close failure leaves subscription in a permanent fence state [fix][broker][branch-2.11] Fix geo-replication admin [improve] Simplify enabling Broker, WS Proxy hostname verification [fix][broker] Filter system topic when getting topic list by binary proto. [feat][txn] Support for idempotent commit and abort, Solution1. [fix][ci][branch-2.11] Fix broken CI [fix][client] Broker address resolution wrong if connect through a multi-dns names proxy [fix][test] ProxyWithAuthorizationTest remove SAN from test certs [improve][broker][branch-2.11] Improve tls config on replication client and cluster cli [fix] [client] fix memory leak if enabled pooled messages [fix][txn]fix receive duplicated messages due to pendingAcks in PendingAckHandle [fix][broker] Fixed history load not releasing [fix][broker] Allow proxy to pass same role for authRole and originalRole [fix][broker] Fix geo-replication admin [fix] [test] Wrong mock-fail of the test ManagedLedgerErrorsTest.recoverLongTimeAfterMultipleWriteErrors [fix][broker] Terminate the async call chain when the condition isn’t met for resetCursor [fix][proxy] Fix using wrong client version in pulsar proxy [improve][broker] Use shrink map for trackerCache [fix] [broker] Make the service name resolver cache of PulsarWebResource expire after access [fix][authentication] Store the original authentication data [fix][broker] Copy command fields and fix potential thread-safety in ServerCnx [fix][client] Fix authentication not update after changing the serviceUrl [fix][broker] Make ServerCnx#originalAuthData volatile [fix][broker] Make authentication refresh threadsafe [fix] [broker] Incorrect service name selection logic [fix][proxy] Fix JKS TLS transport [fix][client] Fix load the trust store file [fix][fn] Fix k8s merge runtime opts bug [improve][txn] Allow superusers to abort transactions [improve][broker] Require authRole is proxyRole to set originalPrincipal [fix][broker] Expect msgs after server initiated CloseProducer [fix] [ml] topic load fail by ledger lost [revert][misc] “modify check waitingForPingResponse with volatile (#12615)” [improve][broker] Add test to verify authRole cannot change [fix][ml] Reset individualDeletedMessagesSerializedSize after acked all messages. [improve] Upgrade lombok to 1.8.26 [improve] Upgrade to zk 3.8.1 [fix][broker] Fix loadbalance score calculation problem [improve][misc] Upgrade Netty to 4.1.87.Final [fix][ml] Fix potential NPE cause future never complete. [fix][authorization] Fix the return value of canConsumeAsync [improve] [broker] Print warn log if compaction failure [fix] [ml] Fix the incorrect total size if use ML interceptor [fix][broker] Replace sync call in async call chain of AdminResource#internalCreatePartitionedTopic [fix][broker] Release Netty buffer in finally block in ServerCnx#handleLastMessageIdFromCompactedLedger [fix][client] Fix async completion in ConsumerImpl#processPossibleToDLQ [fix][broker] Replace sync method in NamespacesBase#internalDeleteNamespaceBundleAsync [fix][broker] Fix concurrency bug in PersistentTopicsBase#internalGetReplicatedSubscriptionStatus [fix] Close TransactionBuffer when create persistent topic timeout [fix][io] Update Elasticsearch sink idle cnx timeout to 30s [fix] [ml] messagesConsumedCounter of NonDurableCursor was initialized incorrectly [fix][client] Fix reader listener can’t auto ack with pooled message. [fix] [ml] The atomicity of multiple fields of ml is broken [improve][broker] Added isActive in ManagedCursorImpl [improve] Upgrade wildfly-elytron (used by debezium) to fix CVE-2022-3143 [fix][proxy] Only go to connecting state once [fix][client] Set fields earlier for correct ClientCnx initialization [improve][broker] ServerCnx: go to Failed state when auth fails [improve][txn] Handle changeToReadyState failure correctly in TC client [fix][broker] AbstractBatchedMetadataStore - use AlreadyClosedException instead of IllegalStateException [fix] [ml] Topics stats shows msgBacklog but there reality no backlog [cleanup][broker] Validate originalPrincipal earlier in ServerCnx [fix][sec] Exclude log4j from openmldb [fix][txn] Catch and log runtime exceptions in async operations [fix] [broker] Counter of pending send messages in Replicator incorrect if schema future not complete [improve][broker] Follow up #19230 to tighten the validation scope [fix][broker] Support deleting partitioned topics with the keyword-partition-
[fix][broker] Copy subscription properties during updating the topic partition number.
[improve][websocket] Add ping support
[fix][doc] Remove lombok plugin and define FunctionRecordBuilder explicitly
[fix][txn] fix txn coordinator recover handle committing and aborting txn race condition.
[fix][broker] Fix race condition while updating partition number
[fix][client] Fix producer could send timeout when enable batching
[fix][broker] Pass subName for subscription operations in ServerCnx
[fix][build] Upgrade dependency-check-maven plugin to fix broken OWASP check
[improve][broker] Add ref count for sticky hash to optimize the performance of Key_Shared subscription
[fix][broker] Close transactionBuffer after MessageDeduplication#checkStatus failed
[fix][broker] Support zookeeper read-only config.
[fix][txn] Always send correct transaction id in end txn response
[improve][schema] Do not print error log with stacktrace for 404
[fix][broker] Topic could be in fenced state forever if deletion fails
[fix][broker] Fix consumer with schema cannot subscribe when AUTO_CONSUME consumers exist
[fix][build] Resolve OWASP Dependency Check false positives
[improve][sec] Suppress false positive OWASP reports
[fix][sec] Upgrade woodstox to 5.4.0
[fix][sec] Upgrade jettison to 1.5.3
[fix][client] Prevent DNS reverse lookup when physical address is an IP address
[improve][admin] Improve partitioned-topic condition evaluation
[improve][broker] Add logs for why namespace bundle been split
[fix][admin] Fix validatePersistencePolicies that Namespace/Topic persistent policies cannot set to < 0
[improve][broker] Omit making a copy of CommandAck when there are no broker interceptors
[fix][broker] Fix deadlock in PendingAckHandleImpl
[fix][broker] Copy proto command fields into final variables in ServerCnx
[fix][broker]fix multi invocation for ledger createComplete
[fix] [tx] [branch-2.11] Transaction buffer recover blocked by readNext
[improve] Add PulsarExceptionBase class that supports slf4j like parameterized string formatting
[fix][txn] transaction pending ack store future not completely problem
[fix][broker]Update interceptor handler exception
[improve][sec] Remove snakeyaml dependency to suppress CVE-2022-1471
[fix][broker] Check operstate when get physical NICs
[fix] [broker] getLastMessageId returns a wrong batch index of last message if enabled read compacted
[fix][fn] Typo in method name
[fix][test][branch-2.10] Remove testGetTopic method
[cherry-pick][branch-2.11] cherry-pick fixing can not delete namespace by force (#18307)
[fix][broker] Fix uncompleted future when getting the topic policies of a deleted topic
[fix][broker] Fix delete system topic clean topic policy
[fix][broker] Fix namespace deletion if __change_events topic has not been created yet
[fix][sql] Fix message without schema issue.
[cleanup][broker] Simplify extract entryMetadata code in filterEntriesForConsumer
[fix][broker] Fix duplicated schemas creation
[fix][broker] transactional producer created failed due to read snapshot blocked.
[fix][broker] Fix PulsarRegistrationClient and ZkRegistrationClient not aware rack info problem.
[improve][test] Add test brokerReachThreshold for ThresholdShedderTest
[improve][broker] Make Consumer#equals more effective
[fix][broker] Create replicated subscriptions for new partitions when needed
[fix][client] For exclusive subscriptions, if two consumers are created repeatedly, the second consumer will block
[fix][io] ElasticSearch sink: align null fields behaviour
[improve][broker] Adjust to sample the error log
[fix][broker] DnsResolverUtil.TTL should be greater than zero
[fix][client] Fix multi-topic consumer stuck after redeliver messages
[fix][client] Avoid redelivering duplicated messages when batching is enabled
[fix][client] Fix failover/exclusive consumer with batch cumulate ack issue.
[fix][broker] Correctly set byte and message out totals per subscription
[improve][client] Change the get lastMessageId to debug level
[fix][broker] Fix open cursor with null-initialPosition result with earliest position
[fix][broker]Fix failover/exclusive consumer cumulate ack not remove msg from UnAckedMessageTracker
[fix][client] Fix possible npe
[fix][client] Fixes batch_size not checked in MessageId#fromByteArrayWithTopic
[fix][client] Fix the Windows absolute path not recognized in auth param string
[fix][client] Set authentication when using loadConf in client and admin client
[fix][client] Fix exception when calling loadConf on a ConsumerBuilder that has a KeySharedPolicy
[fix][client] Support LocalDateTime Conversion
[fix][broker] Fix can not delete namespace by force
[fix][broker] In the trimDeletedEntries method, release the removed entry
[fix][schema] Fix creating lots of versions for the same schema
[fix][broker] fix delete_when_subscriptions_caught_up doesn’t work while have active consumers
[fix][client] Fix IllegalThreadStateException when using newThread in ExecutorProvider.ExtendedThreadFactory
[fix] [pulsar-client] Fix pendingLookupRequestSemaphore leak when Ser…
[improve][broker] Add UncaughtExceptionHandler for every thread pool
[improve][broker] invalidate lock cache when Deleted NotificationType
[improve][broker] Remove locallyAcquiredLock when removeOwnership
[improve][broker] Support setting forceDeleteTenantAllowed dynamically
[improve][broker] Support setting ForceDeleteNamespaceAllowed dynamically
[fix][meta] fix getChildren in MemoryMetadataStore and EtcdMetadataStore
[fix][broker] AVAILABLE_PERMITS_UPDATER not updated, when writeAndFlush fails.
[fix][broker] PulsarRegistrationClient - implement getAllBookies and follow BookieServiceInfo updates
[fix][broker] Fix update authentication data
[fix][function] Fix invalid metric type “gauge ”
[fix][fn] fix function failed to start if no typeClassName provided in FunctionDetails
[improve][client] Support MAX_ACK_GROUP_SIZE configurable
[improve][schema] Change update schema auth from tenant to produce
[fix][broker] Update the log print content of createSubscriptions
[improve][broker] Reduce unnecessary persistence of markdelete
[fix][broker] Fix the order of resource close in the InMemoryDelayedDeliveryTracker
[fix][broker] Fix incorrect bundle split count metric
[fix][broker]unify time unit at dropping the backlog on a topic
[improve][connector] JDBC sink: allow any jdbc driver
Make BookieId work with PulsarRegistrationDriver (second take)
[fix][broker]Fix mutex never released when trimming
Allow to configure and disable the size of lookahead for detecting fixed delays in messages
[fix][admin] returns 4xx error when pulsar-worker-service is disabled and trying to access it
[fix][loadbalance] Fix the wrong NIC speed rate unit.
[improve][broker]Improve PersistentMessageExpiryMonitor expire speed when ledger not existed
[fix][client] moving get sequenceId into the sync code segment
[refactor][java] Unify the acknowledge process for batch and non-batch message IDs
[fix][proxy] Fix refresh client auth
[fix][cli] Check numMessages after incrementing counter
[improve][java-client]Add init capacity for messages in BatchMessageContainerImpl
[fix][doc]fix comments for exposeManagedLedgerMetricsInPrometheus field
[Improve][auth]Update authentication failed metrics report
[fix][sec] Bump snakeyaml to 1.32 for CVE-2022-38752
[fix][cli] Quit PerformanceConsumer after receiving numMessages messages
[bugfix] ManagedLedger: move to FENCED state in case of BadVersionException
[fix][broker] Fix create ns
[fix][metrics]wrong metrics text generated when label_cluster specified
[fix][broker] Fix executeWithRetry result is null
[fix][cli] Fix mbeans to json
[fix][bookie] Correctly handle list configuration values
[fix][build] duplicate entry when merging services
[C++] Reset havePendingPingRequest flag for any data received from broker
[improve][pulsar-proxy] Update proxy lookup throw exception type
Issue 17588: Allow deletion of a namespace that was left in deleted status
[improve][test] Remove WhiteBox for ElasticSearchSinkTests
[improve][test] remove WhiteBox on MockZooKeeper
[improve][ci] Remove post commit builds and add manual workflows trigger
[ci] Move more tests to the flaky suite
[ci] Skip run flaky tests suite for cpp-only or doc-only changes
[ci] Extract FLAKY tests suite to a dedicated workflow
[ci] Reduce runners load for pulls that affect only doc or cpp changes
[fix][tiered-storage] Don’t cleanup data when offload met Metastore exception
[improve][ci] Optimize “Pulsar Bot” workflow
[improve][ci] Remove “Cancel duplicate workflows” workflow
[fix][flaky-test]NamespaceServiceTest.flaky/testModularLoadManagerRemoveBundleAndLoad
[fix][license] Update the log4j version in the presto license file
[fix][sec] bump snakeyaml to 1.31 fix CVE-2022-25857
[improve][broker] Improve cursor.getNumberOfEntries if isUnackedRangesOpenCacheSetEnabled=true
[fix][client]Duplicate messages when use MultiTopicsConsumerImpl
[improve][cli] Using separate TLS config on the compactor
[improve][txn] Add getState in transaction for client API
[fix][broker] Remove timestamp from Prometheus metrics
[improve][broker] Using handle instead of handleAsync to avoid using common pool thread
[fix][broker] fix can not revoke permission after update topic partition
[improve][admin] PulsarAdminBuilderImpl overrides timeout properties passed through config map
[fix][broker]fix catching ConflictException when update topic partition
[fix][broker] Fix Npe thrown by splitBundle
[fix][flaky] Fix flakyness in testAutoSchemaFunction
[improve][broker]Only create extended partitions when updating partition number
[improve][schema]autoSkipNonRecoverableData work for schema Ledger
[improve][ci] Replace test reporting with less verbose solution
[improve][broker]Recover as much cursor data as possible, even if entry is invalid
[fix][broker]ManagedLedger metrics fail cause of zero period
[refactor][java] Improve docs and code quality about KeyValueSchema usages
[fix][broker]fail to update partition meta of topic due to ConflictException: subscription already exists for topic
[broker][fix]Fix update topic remove properties
[improve][build] Avoid building image multiple times
[fix][broker]fix arithmetic exception for LeastResourceUsageWithWeight strategy
[improve][docker] Switch to Temurin JDK
[fix][broker] Skip not connectable URL when get redirectionUrl
[improve][admin] Unset namespace policy to improve deleting namespace.
[fix][broker] Fix delivery the message that has been acknowledged
[fix][txn] Correct the prompt message
[improve][ci] Skip unnecessary tests when there are only cpp/python related changes
[Fix][flaky-test] Fix testConsumeTxnMessage
[Improve][broker]Reduce GetReplicatedSubscriptionStatus local REST call
[fix][ml] fix NPE
[improve] clean the empty topicAuthenticationMap in zk when revoke permission
[pulsar-io] KCA: handle kafka’s logical schemas
[improve][broker] ServerCnx: log at warning level when topic not found
[fix][storage] Autorecovery default reppDnsResolverClass to ZkBookieRackAffinityMapping
[fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
[refactor][broker] Arrange cleanup operation for topic creation
[monitor][txn] Add metrics for transaction
Issue 14583: [python client]: Improve garbage collection of producer/consumer/reader objects
[feature][txn] Fix individual ack batch message with transaction abort redevlier duplicate messages
[Improvement] Recycler should use io.netty.recycler.maxCapacityPerThread instead of io.netty.recycler.maxCapacity.default as maxCapacityPerThread configuration in Netty 4.1.x
[Issue 12844][pulsar-client-cpp] Excessive locking cause significant performance degradation