StreamNative Weekly Release Notes v3.0.10.9
General Changes
Apache Pulsar
(#24960) [cleanup][broker][branch-3.0] Remove no-op configurations caused by cherry-picking (#23551) [fix][txn] fix concurrent error cause txn stuck in TransactionBufferHandlerImpl#endTxn (#24939) [fix][broker] Avoid recursive update in ConcurrentHashMap during policy cache cleanup (#24762) [fix][admin] Set local policies overwrites “number of bundles” passed during namespace creation (#24929) [fix][test] Stabilize testMsgDropStat by reliably triggering non-persistent publisher drop (#24934) [fix][broker] Fix stack overflow caused by race condition when closing a connection (#24898) [fix][broker] fix getMaxReadPosition in TransactionBufferDisable should return latest (#24794) [fix][client] Fix thread leak in reloadLookUp method which is used by ServiceUrlProvider (#24859) [fix][broker] Run ResourceGroup tasks only when tenants/namespaces registered (#24915) [fix][broker] BacklogMessageAge is not reset when cursor mdPosition is on an open ledger (#24917) [improve][ci] Move replication tests to new group Broker Group 5 in Pulsar CI (#24911) [improve][misc] Upgrade Netty to 4.1.128.Final (#24904) [fix][test] Fix flaky ReplicatorTest.testResumptionAfterBacklogRelaxed (#24885) [fix][broker] Fix totalAvailablePermits not reduced when removing consumer from non-persistent dispatcher (#24848) [improve][client]Add null check for Pulsar client clock configuration (#24880) [fix][broker] Stop to retry to read entries if the replicator has terminated (#24865) [fix][test] Fix flaky SubscriptionSeekTest.testSeekWillNotEncounteredFencedError by counting subscription is fenced only after seek (#24861) [fix][test] Stabilize SequenceIdWithErrorTest by fencing after first publish to avoid empty-ledger deletion and send timeout (#24852) [fix][ml] FixgetNumberOfEntries may point to deleted ledger
(#24841) [improve][ci] Upgrade GitHub Actions workflows to use ubuntu-24.04
(#24830) [fix][client] Fix getPendingQueueSize for PartitionedTopicProducerStatsRecorderImpl: avoid NPE and implement aggregation
(#24832) [fix] Fix mixed lookup/partition metadata requests causing reliability issues and incorrect responses
(#24822) [fix][client] Make auto partitions update work for old brokers without PIP-344
(#24812) [fix][build] Remove invalid profile in settings.xml that caused gpg signing to fail
(#24770) [fix][broker] Flaky-test: ExtensibleLoadManagerImplTest.testDisableBroker
(#24779) Bump org.apache.zookeeper:zookeeper from 3.9.3 to 3.9.4
(#24772) [fix][misc] Fix compareTo contract violation for NamespaceBundleStats, TimeAverageMessageData and ResourceUnitRanking
(#24769) [fix][test] Flaky-test: BrokerServiceTest.testShutDownWithMaxConcurrentUnload
(#24767) [fix][ci] Fix CI for Java 25 including upgrade of Gradle Develocity Maven extension
(#24763) [improve][build] Upgrade Lombok to 1.18.42 to fully support JDK25
(#23634) [improve][broker] If there is a deadlock in the service, the probe should return a failure because the service may be unavailable
(#24738) [fix][broker] First entry will be skipped if opening NonDurableCursor while trimmed ledger is adding first entry.
(#24753) [fix][ml]Fix EOFException after enabled topics offloading
(#24741) [fix][broker] Prevent unexpected recycle failure in dispatcher’s read callback
(#20522) [improve][broker] Choose random thread for consumerFlow in PersistentDispatcherSingleActiveConsumer
(#24758) [fix][broker][branch-3.0] Prevent NPE in ownedBundlesCountPerNamespace on first bundle load
(#24752) [fix][client] rollback TopicListWatcher retry behavior
(#24698) [fix][client]TopicListWatcher not closed when calling PatternMultiTopicsConsumerImpl.closeAsync() method
(#24634) Dispatcher did unnecessary sort for recentlyJoinedConsumers and printed noisy error logs
(#24730) [fix][broker] Ensure KeyShared sticky mode consumer respects assigned ranges
(#24743) [fix][client] Fix receiver queue auto-scale without memory limit
(#24731) [fix][broker] Fix cannot shutdown broker gracefully by admin api
(#24654) [fix][io] Improve Kafka Connect source offset flushing logic
(#24725) [fix][client] Avoid recycling the same ConcurrentBitSetRecyclable among different threads
(#24719) [fix][broker] Fix memory leak when metrics are updated in a thread other than FastThreadLocalThread
(#24594) [improve][build] Disable javadoc build failure
(#24580) [fix][broker]Fix never recovered metadata store bad version issue if received a large response from ZK
(#23336) [fix][client] Fix ArrayIndexOutOfBoundsException when using SameAuthParamsLookupAutoClusterFailover
(#23977) [fix][broker] Invalid regex in PulsarLedgerManager causes zk data notification to be ignored
(#24518) (#24671) [fix][broker][branch-3.0] Fix wrong backlog age metrics when the mark delete position point to a deleted ledger
(#24663) [fix][client] Skip schema validation when sending messages to DLQ to avoid infinite loop when schema validation fails on an incoming message
(#24669) [improve][io] Support specifying Kinesis KPL native binary path with 1.0 version specific path
(#24668) [improve][build] Use org.apache.nifi:nifi-nar-maven-plugin:2.1.0 with skipDocGeneration=true
(#24661) [improve][io] Upgrade AWS SDK v1 & v2, Kinesis KPL and KPC versions
(#24639) [fix][broker] Fix race condition in MetadataStoreCacheLoader causing inconsistent availableBroker list caching
(#24666) [improve][build] Increase maven resolver’s sync context timeout
(#24662) [fix][client] fix ArrayIndexOutOfBoundsException in SameAuthParamsLookupAutoClusterFailover
(#24649) [fix][offload] Exclude unnecessary dependencies from tiered storage provider / offloader nar files
(#24643) [fix][broker] Add double-check for non-durable cursor creation
(#24633) [fix][test]fix flaky ZeroQueueSizeTest.testZeroQueueGetExceptionWhenReceiveBatchMessage
(#24632) [fix][test] Fix ConcurrentModificationException in Ipv4Proxy
(#24630) [fix][test]fix flaky ZeroQueueSizeTest.testZeroQueueGetExceptionWhenReceiveBatchMessage
(#24626) [fix][proxy] Fix TooLongFrameException with Pulsar Proxy
(#24621) [fix][broker] Fix duplicate watcher registration after SessionReestablished
(#24610) [fix][client]Prevent ZeroQueueConsumer from receiving batch messages when using MessagePayloadProcessor
(#24604) [improve][io] Add dependency file name information to error message when .nar file validation fails with ZipException
(#21361) [improve][broker] Optimize and clean up aggregation of topic stats
(#24601) [improve][doc] Improve the JavaDocs of sendAsync to avoid improper use
(#24599) [fix][client] Retry for unknown exceptions when creating a producer or consumer
(#24450) [fix][broker] Fix REST API to produce messages to single-partitioned topics
(#24595) [fix][ci] Fix code coverage metrics in Pulsar CI
(#24582) [improve][client] Support load RSA PKCS#8 private key
(#24535) [improve][test] Add test for dead letter topic with max unacked messages blocking
(#24532) [fix][misc] Upgrade dependencies to fix critical security vulnerabilities
(#24514) [improve][build] Bump org.apache.commons:commons-lang3 from 3.17.0 to 3.18.0
(#24586) [improve][test] Refactor the way way pulsar-io-debezium-oracle nar file is patched when building the test image
(#24590) [fix][broker] Fix flaky testReplicatorsInflightTaskListIsEmptyAfterReplicationFinished
(#24542))) Revert “[fix][broker]Fix thread safety issues in BucketDelayedDeliveryTracker with StampedLock optimistic reads
(#24554) (#24571) [fix][client][branch-4.0] Partitioned topics are unexpectedly created by client after deletion
(#24576) [fix][test] fix flaky GrowableArrayBlockingQueueTest.testPollBlockingThreadsTermination
(#24569) [fix][broker] Fix ManagedCursor state management race conditions and lifecycle issues
(#24550) [improve][client] Terminate consumer.receive() when consumer is closed
(#24473) [improve][build] replace org.apache.commons.lang to org.apache.commons.lang3
(#24560) [fix][broker] Fix maxTopicsPerNamespace might report a false failure
(#24505) [fix][test]fix flaky test BrokerServiceAutoTopicCreationTest.testDynamicConfigurationTopicAutoCreationPartitioned
(#24472) [fix] Prevent IllegalStateException: Field ‘message’ is not set
(#24542) [fix][broker]Fix thread safety issues in BucketDelayedDeliveryTracker with StampedLock optimistic reads
(#24551) [fix][broker] Fix Broker OOM due to too many waiting cursors and reuse a recycled OpReadEntry incorrectly
(#24511) [fix][broker] Fix deduplication replay might never complete for exceptions
(#24522) [fix][ml] Fix the possibility of message loss or disorder when ML PayloadProcessor processing fails
(#24451) [fix][test][branch-3.0] Correct topic policy loading logic and improve related tests
(#24552) [improve][test] Remove EntryCacheCreator from ManagedLedgerFactoryImpl
(#24544) [improve] Upgrade pulsar-client-python to 3.8.0 in Docker image
(#24516) [fix][broker] Fix exclusive producer creation when last shared producer closes
(#24506) [fix][broker] Fix duplicate increment of ADD_OP_COUNT_UPDATER in OpAddEntry
(#24543) [fix][broker] Fix matching of topicsPattern for topic names which contain non-ascii characters
(#24539) [fix][client] Close orphan producer or consumer when the creation is interrupted
(#24517) [fix][client] Fix ClientCnx handleSendError NPE
(#24515) [fix][ml] Fix asyncReadEntries might never complete if empty entries are read from BK
(#24525) [improve][misc] Optimize topic list hashing so that potentially large String allocation is avoided
(#24528) [fix][client] Fix issue in auto releasing of idle connection with topics pattern consumer
(#24529) [fix][proxy] Fix default value of connectionMaxIdleSeconds in Pulsar Proxy
(#24476) [fix][client] NPE in MultiTopicsConsumerImpl.negativeAcknowledge
(#24465) [fix][proxy] Fix proxy OOM by replacing TopicName with a simple conversion method
(#22495) [fix][test] Move ExtensibleLoadManagerImplTest to flaky tests
(#21642) [fix][test] Fix flaky test SimpleProducerConsumerStatTest#testPartitionTopicStats
(#24453) [fix][broker] replication does not work due to the mixed and repetitive sending of user messages and replication markers
(#24424) [fix][broker] Fix the non-persistenttopic’s replicator always get error “Producer send queue is full” if set a small value of the config replicationProducerQueueSize
(#24189) [fix][broker]excessive replication speed leads to error: Producer send queue is full
(#22674) [Fix][broker] Limit replication rate based on bytes
(#20931) [fix][broker] Fix ack hole in cursor for geo-replication
(#24443) [fix][txn] Fix negative unacknowledged messages in transactions by ensuring that the batch size is added into CommandAck
(#24421) [fix][build] Add missing <name> to submodules
(#24441) [fix][ml] Enhance OpFindNewest to support skip non-recoverable data
(#24459) [improve][broker] change to warn log level for ack validation error
(#24434) [improve][broker] Improve the log when namespace bundle is not available
(#24432) [fix][ml]Still got BK ledger, even though it has been deleted after offloaded
(#21467) [fix][test] Cleanup resources if starting PulsarService fails in PulsarTestContext
(#24351) [improve][broker] Deny removing local cluster from topic level replicated cluster policy
(#24419) [fix][broker] Once the cluster is configured incorrectly, the broker maintains the incorrect cluster configuration even if you removed it
(#24404) [fix][client] Prevent NPE when seeking with null topic in TopicMessageId
(#24405) [fix][ml]Received more than once callback when calling cursor.delete
(#24406) [fix][ml] Cursor ignores the position that has an empty ack-set if disabled deletionAtBatchIndexLevelEnabled
(#24401) [fix][txn] Fix deadlock when loading transaction buffer snapshot
(#24402) [fix][client] Fix some potential resource leak
(#20629) [improve][test] Fix flaky test SimpleProducerConsumerStatTest#testMsgRateExpired
(#21629) [fix][build] Fix potential insufficient protostuff-related configs
(#23594) [fix] [broker] No longer allow creating subscription that contains slash
(#24366) [fix][broker]Fix deadlock when compaction and topic deletion execute concurrently
(#24350) [fix][broker] Fix issue that topic policies was deleted after a sub topic deleted, even if the partitioned topic still exists
(#24384) [fix][ml]Revert a behavior change of releasing idle offloaded ledger handle: only release idle BlobStoreBackedReadHandle
(#24397) [improve][misc] Upgrade Netty to 4.1.122.Final and tcnative to 2.0.72.Final
(#24391) [improve][broker] Add managedCursor/LedgerInfoCompressionType settings to broker.conf
(#24392) [improve][broker] Make maxBatchDeletedIndexToPersist configurable and document other related configs
(#24386) [improve][broker] Added synchronized for sendMessages in Non-Persistent message dispatchers
(#24381) [improve][ml]Release idle offloaded read handle only the ref count is 0
(#19783) [improve][offloaders] Automatically evict Offloaded Ledgers from memory
(#24360) [fix][broker] expose consumer name for partitioned topic stats
(#24359) [improve][broker]Improve the log when encountered in-flight read limitation
(#24354) [fix][io] Acknowledge RabbitMQ message after processing the message successfully
(#24352) [fix][broker] Ignore metadata changes when broker is not in the Started state
(#24190) [fix][broker] Resolve the issue of frequent updates in message expiration deletion rate
(#24338) [fix][ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent issue
(#24331) [fix][offload] Complete the future outside of the reading loop in BlobStoreBackedReadHandleImplV2.readAsync
(#24324) [fix][test] Fix flaky AutoScaledReceiverQueueSizeTest.testNegativeClientMemory
(#24316) [fix][io] Fix kinesis avro bytes handling
(#24344) [improve][ml] Offload ledgers without check ledger length
(#24286) [fix][broker]Non-global topic policies and global topic policies overwrite each other
(#24279) [fix][broker]Global topic policies do not affect after unloading topic and persistence global topic policies never affect
(#24349) [fix][io][branch-3.0] Backport Kinesis Sink custom native executable support #23762
(#24317) [fix][io][branch-3.0]Pulsar-SQL: Fix classcast ex when decode decimal value
(#24313) [fix][broker] Fix potential deadlock when creating partitioned topic
(#24293) [fix][broker] fix wrong method name checkTopicExists.
(#24307) [fix][build] Ensure that buildtools is Java 8 compatible and fix remaining compatibility issue
(#24304) [fix][test] Simplify BetweenTestClassesListenerAdapter and fix issue with BeforeTest/AfterTest annotations
(#24289) [improve][io] Add configuration parameter for disabling aggregation for Kinesis Producers
(#24302) [improve] Upgrade pulsar-client-python to 3.7.0 in Docker image
(#24299) [fix][test] Fix more Netty ByteBuf leaks in tests
(#24297) [fix][io] Fix SyntaxWarning in Pulsar Python functions
(#24282) [fix][client] Fix producer publishing getting stuck after message with incompatible schema is discarded
(#24283) [cleanup][test] Remove unused parameter from deleteNamespaceWithRetry method in MockedPulsarServiceBaseTest
(#24263) [improve][build] Upgrade zstd version from 1.5.2-3 to 1.5.7-3
(#24281) [fix][test] Fix multiple ByteBuf leaks in tests
(#24275) [fix][broker] Fix HashedWheelTimer leak in PulsarService by stopping it in shutdown
(#24274) [fix][misc] Fix ByteBuf leak in SchemaUtils
(#24254) [fix][broker]Fix incorrect priority between topic policies and global topic policies
(#24266) [improve][ci] Disable detailed console logging for integration tests in CI
(#24261) [fix][test] Fix flaky ManagedCursorTest.testLastActiveAfterResetCursor and disable failing SchemaTest
(#24244) [fix][test] Fix flaky ManagedCursorTest.testSkipEntriesWithIndividualDeletedMessages
(#24248) [improve][io][kca] support fully-qualified topic names in source records
(#24260) [improve][build] Upgrade Gradle Develocity Maven Extension dependencies
(#24258) [fix][test] Fix TestNG BetweenTestClassesListenerAdapter listener
(#24257) [fix][broker] Unregister non-static metrics collectors registered in Prometheus default registry
(#24178) [fix][broker]fix memory leak, messages lost, incorrect replication state if using multiple schema versions(auto_produce)
(#24219) [improve][broker]Improve the feature “Optimize subscription seek (cursor reset) by timestamp”: search less entries
(#23919) [fix][broker] Fix seeking by timestamp can be reset the cursor position to earliest
(#22792) [improve][broker] Optimize subscription seek (cursor reset) by timestamp
(#24243) [improve][build] Upgrade SpotBugs to 4.9.x
(#24240) [improve][build] Upgrade to jacoco 0.8.13
(#24237) [improve][build] Upgrade Lombok to 1.18.38 to support JDK 24
(#24221) [improve][io] support kafka connect transforms and predicates
(#24230) [improve][client]Improve transaction log when a TXN command timeout
(#24223) [fix][broker] Orphan schema after disabled a cluster for a namespace
(#24228) [fix][broker] Fix ByteBuf memory leak in REST API for publishing messages
(#24184) [fix][client] Fix incorrect producer.getPendingQueueSize due to incomplete queue implementation
(#24214) [improve] Upgrade Netty to 4.1.121.Final
(#24212) [fix][test] Fix flaky BatchMessageWithBatchIndexLevelTest.testBatchMessageAck
(#24218) [fix][test] Fix multiple resource leaks in tests
(#24187) [improve][client] validate ClientConfigurationData earlier to avoid resource leaks
(#24216) [fix][broker] Fix HealthChecker deadlock in shutdown
(#24209) [fix][broker] Fix tenant creation and update with null value
(#24192) [fix][admin] Backlog quota’s policy is null which causes a NPE
(#24210) [fix][broker] Fix broker shutdown delay by resolving hanging health checks
(#24207) [fix][broker] Fix compaction service log’s wrong condition
(#24204) [fix][test] Fix resource leaks in ProxyTest and fix invalid tests
(#24201) [improve][io] Upgrade Kafka client and compatible Confluent platform version
(#24118))) Revert “[fix][broker] Add topic consistency check
(#24154))) Revert “[fix][broker] Directly query single topic existence when the topic is partitioned
(#24032) [fix][broker] Fix missing validation when setting retention policy on topic level
(#24098) [fix][ml] Skip deleting cursor if it was already deleted before calling unsubscribe
(#24181) [fix][proxy] Fix incorrect client error when calling get topic metadata
(#24158) [fix][proxy] Propagate client connection feature flags through Pulsar Proxy to Broker
(#24103) [fix][schema] Reject unsupported Avro schema types during schema registration
(#24091) [fix][broker] Fix some problems in calculate totalAvailableBookies in method getExcludedBookiesWithIsolationGroups when some bookies belongs to multiple isolation groups.
(#21320) [fix][bk] Fix the var name for IsolationGroups
(#24171) [improve][test] Use configured session timeout for MockZooKeeper and TestZKServer in PulsarTestContext
(#24172) [fix][test] Improve reliability of IncrementPartitionsTest
(#24170) [fix][test]flaky-test:ManagedLedgerInterceptorImplTest.testManagedLedgerPayloadInputProcessorFailure
(#23980) [fix][broker] Consumer stuck when delete subscription __compaction failed
(#24167) [fix][ml] Fix ML thread blocking issue in internalGetPartitionedStats API
(#24166) [fix][test] Fix invalid test CompactionTest.testDeleteCompactedLedgerWithSlowAck
(#24150) [fix][broker] The feature brokerDeleteInactivePartitionedTopicMetadataEnabled leaves orphan topic policies and topic schemas
(#24154) [fix][broker] Directly query single topic existence when the topic is partitioned
(#24118) [fix][broker] Add topic consistency check
(#24056) [fix][test] Update partitioned topic subscription assertions in IncrementPartitionsTest
(#24033) [cleanup][misc] Add override annotation
(#24161) [fix][test] Fix flaky BrokerServiceChaosTest.testFetchPartitionedTopicMetadataWithCacheRefresh
(#24162) [fix][test] Fix flaky BrokerServiceChaosTest
(#24097) [fix] [broker] topics infinitely failed to delete after remove cluster from replicated clusters modifying when using partitioned system topic
(#22261) [fix] Bump google.golang.org/protobuf from 1.32.0 to 1.33.0 in /pulsar-function-go
(#24132) [fix][io] Fix KinesisSink json flattening for AVRO’s SchemaType.BYTES
(#20984) [fix][broker] Fix get outdated compactedTopicContext after compactionHorizon has been updated
(#20697) [improve][broker] Improve CompactedTopicImpl lock
(#24131) [fix][ml] Return 1 when bytes size is 0 or negative for entry count estimation
(#24128) [improve][io] Enhance Kafka connector logging with focused bootstrap server information
(#24125) [fix][ml] Don’t estimate number of entries when ledgers are empty, return 1 instead
(#24123) [improve][client] Prevent NullPointException when closing ClientCredentialsFlow
(#24124) [improve][io] Remove sleep when sourceTask.poll of kafka return null
(#24116) [improve][broker] Change topic exists log to warn
(#24104) [fix][client] Pattern subscription regression when broker-side evaluation is disabled
(#24100) [fix][client] Fix consumer leak when thread is interrupted before subscribe completes
(#24089) [fix][ml] Fix issues in estimateEntryCountBySize
(#24073) [improve][broker] Optimize message expiration rate repeated update issues
(#24087) [fix][broker] Avoid IllegalStateException when marker_type field is not set in publishing
(#24083) [fix][ci] Bump dependency-check to 12.1.0 to fix OWASP Dependency Check job
(#24082) [clean][client] Clean code for the construction of retry/dead letter topic name
(#24079) [fix][broker] Fix NPE while publishing Metadata-Event with not init producer
(#24080) [fix][broker] Fix Metadata event synchronizer should not fail with bad version
(#24081) [fix][broker] Fix Metadata Event Synchronizer producer creation retry so that the producer gets created eventually
(#24048) [fix][broker] Fix UnsupportedOperationException while setting subscription level dispatch rate policy
(#24054) [fix][ml] Corrected pulsar_storage_size metric to not multiply offloaded storage by the write quorum
(#24067) [fix][broker] http metric endpoint get compaction latency stats always be 0
(#24064) [improve][broker] Optimize ThresholdShedder with improved boundary checks and parameter reuse
(#24055) [fix] Avoid negative estimated entry count
(#24060) [improve][monitor] Add version=0.0.4 to /metrics content type for Prometheus 3.x compatibility
(#24059) [fix][client] Copy eventTime to retry letter topic and DLQ messages
(#24061) [fix][client] Fix building broken batched message when publishing
(#24063) [fix][broker]Fix failed consumption after loaded up a terminated topic
(#24072) [fix][broker] Pattern subscription doesn’t work when the pattern excludes the topic domain.
(#24049) [improve] Upgrade Netty to 4.1.119.Final
(#23975) [fix][broker] Add expire check for replicator
(#24023) [fix][doc] fix doc related to chunk message feature.
(#23962) [improve][ml] Use lock-free queue in InflightReadsLimiter since there’s no concurrent access
(#23978) [improve][cli] Support additional msg metadata for V1 topic on peek message cmd
(#24014) [fix][broker] Fix BucketDelayedDeliveryTracker thread safety
(#24019) [fix][test]Fix flaky test V1_ProducerConsumerTest.testConcurrentConsumerReceiveWhileReconnect
(#24011) [fix][test] Fix flaky test OneWayReplicatorUsingGlobalZKTest.testConfigReplicationStartAt
(#23931) [improve] [broker] Make the estimated entry size more accurate
(#24004) [improve][ci] Upgrade Gradle Develocity Maven Extension to 1.23.1
(#23697) [fix][broker] Geo Replication lost messages or frequently fails due to Deduplication is not appropriate for Geo-Replication
(#24006) [fix][broker] fix broker identifying incorrect stuck topic
(#23286) [improve][broker] Fix non-persistent system topic schema compatibility
(#23881) [improve][fn] Set default tenant and namespace for ListFunctions cmd
(#23730) [fix][admin] Verify is policies read only before revoke permissions on topic
(#24003) [improve][test] Upgrade Testcontainers to 1.20.4 and docker-java to 3.4.0