StreamNative Weekly Release Notes v3.0.10.8
General Changes
Apache Pulsar
(#24772) [fix][misc] Fix compareTo contract violation for NamespaceBundleStats, TimeAverageMessageData and ResourceUnitRanking (#24769) [fix][test] Flaky-test: BrokerServiceTest.testShutDownWithMaxConcurrentUnload (#24767) [fix][ci] Fix CI for Java 25 including upgrade of Gradle Develocity Maven extension (#24763) [improve][build] Upgrade Lombok to 1.18.42 to fully support JDK25 (#23634) [improve][broker] If there is a deadlock in the service, the probe should return a failure because the service may be unavailable (#24738) [fix][broker] First entry will be skipped if opening NonDurableCursor while trimmed ledger is adding first entry. (#24753) [fix][ml]Fix EOFException after enabled topics offloading (#24741) [fix][broker] Prevent unexpected recycle failure in dispatcher’s read callback (#20522) [improve][broker] Choose random thread for consumerFlow in PersistentDispatcherSingleActiveConsumer (#24758) [fix][broker][branch-3.0] Prevent NPE in ownedBundlesCountPerNamespace on first bundle load (#24752) [fix][client] rollback TopicListWatcher retry behavior (#24698) [fix][client]TopicListWatcher not closed when calling PatternMultiTopicsConsumerImpl.closeAsync() method (#24634) Dispatcher did unnecessary sort for recentlyJoinedConsumers and printed noisy error logs (#24730) [fix][broker] Ensure KeyShared sticky mode consumer respects assigned ranges (#24743) [fix][client] Fix receiver queue auto-scale without memory limit (#24731) [fix][broker] Fix cannot shutdown broker gracefully by admin api (#24654) [fix][io] Improve Kafka Connect source offset flushing logic (#24725) [fix][client] Avoid recycling the same ConcurrentBitSetRecyclable among different threads (#24719) [fix][broker] Fix memory leak when metrics are updated in a thread other than FastThreadLocalThread (#24594) [improve][build] Disable javadoc build failure (#24580) [fix][broker]Fix never recovered metadata store bad version issue if received a large response from ZK (#23336) [fix][client] Fix ArrayIndexOutOfBoundsException when using SameAuthParamsLookupAutoClusterFailover (#23977) [fix][broker] Invalid regex in PulsarLedgerManager causes zk data notification to be ignored (#24518) (#24671) [fix][broker][branch-3.0] Fix wrong backlog age metrics when the mark delete position point to a deleted ledger (#24663) [fix][client] Skip schema validation when sending messages to DLQ to avoid infinite loop when schema validation fails on an incoming message (#24669) [improve][io] Support specifying Kinesis KPL native binary path with 1.0 version specific path (#24668) [improve][build] Use org.apache.nifi:nifi-nar-maven-plugin:2.1.0 with skipDocGeneration=true (#24661) [improve][io] Upgrade AWS SDK v1 & v2, Kinesis KPL and KPC versions (#24639) [fix][broker] Fix race condition in MetadataStoreCacheLoader causing inconsistent availableBroker list caching (#24666) [improve][build] Increase maven resolver’s sync context timeout (#24662) [fix][client] fix ArrayIndexOutOfBoundsException in SameAuthParamsLookupAutoClusterFailover (#24649) [fix][offload] Exclude unnecessary dependencies from tiered storage provider / offloader nar files (#24643) [fix][broker] Add double-check for non-durable cursor creation (#24633) [fix][test]fix flaky ZeroQueueSizeTest.testZeroQueueGetExceptionWhenReceiveBatchMessage (#24632) [fix][test] Fix ConcurrentModificationException in Ipv4Proxy (#24630) [fix][test]fix flaky ZeroQueueSizeTest.testZeroQueueGetExceptionWhenReceiveBatchMessage (#24626) [fix][proxy] Fix TooLongFrameException with Pulsar Proxy (#24621) [fix][broker] Fix duplicate watcher registration after SessionReestablished (#24610) [fix][client]Prevent ZeroQueueConsumer from receiving batch messages when using MessagePayloadProcessor (#24604) [improve][io] Add dependency file name information to error message when .nar file validation fails with ZipException (#21361) [improve][broker] Optimize and clean up aggregation of topic stats (#24601) [improve][doc] Improve the JavaDocs of sendAsync to avoid improper use (#24599) [fix][client] Retry for unknown exceptions when creating a producer or consumer (#24450) [fix][broker] Fix REST API to produce messages to single-partitioned topics (#24595) [fix][ci] Fix code coverage metrics in Pulsar CI (#24582) [improve][client] Support load RSA PKCS#8 private key (#24535) [improve][test] Add test for dead letter topic with max unacked messages blocking (#24532) [fix][misc] Upgrade dependencies to fix critical security vulnerabilities (#24514) [improve][build] Bump org.apache.commons:commons-lang3 from 3.17.0 to 3.18.0 (#24586) [improve][test] Refactor the way way pulsar-io-debezium-oracle nar file is patched when building the test image (#24590) [fix][broker] Fix flaky testReplicatorsInflightTaskListIsEmptyAfterReplicationFinished (#24542))) Revert “[fix][broker]Fix thread safety issues in BucketDelayedDeliveryTracker with StampedLock optimistic reads (#24554) (#24571) [fix][client][branch-4.0] Partitioned topics are unexpectedly created by client after deletion (#24576) [fix][test] fix flaky GrowableArrayBlockingQueueTest.testPollBlockingThreadsTermination (#24569) [fix][broker] Fix ManagedCursor state management race conditions and lifecycle issues (#24550) [improve][client] Terminate consumer.receive() when consumer is closed (#24473) [improve][build] replace org.apache.commons.lang to org.apache.commons.lang3 (#24560) [fix][broker] Fix maxTopicsPerNamespace might report a false failure (#24505) [fix][test]fix flaky test BrokerServiceAutoTopicCreationTest.testDynamicConfigurationTopicAutoCreationPartitioned (#24472) [fix] Prevent IllegalStateException: Field ‘message’ is not set (#24542) [fix][broker]Fix thread safety issues in BucketDelayedDeliveryTracker with StampedLock optimistic reads (#24551) [fix][broker] Fix Broker OOM due to too many waiting cursors and reuse a recycled OpReadEntry incorrectly (#24511) [fix][broker] Fix deduplication replay might never complete for exceptions (#24522) [fix][ml] Fix the possibility of message loss or disorder when ML PayloadProcessor processing fails (#24451) [fix][test][branch-3.0] Correct topic policy loading logic and improve related tests (#24552) [improve][test] Remove EntryCacheCreator from ManagedLedgerFactoryImpl (#24544) [improve] Upgrade pulsar-client-python to 3.8.0 in Docker image (#24516) [fix][broker] Fix exclusive producer creation when last shared producer closes (#24506) [fix][broker] Fix duplicate increment of ADD_OP_COUNT_UPDATER in OpAddEntry (#24543) [fix][broker] Fix matching of topicsPattern for topic names which contain non-ascii characters (#24539) [fix][client] Close orphan producer or consumer when the creation is interrupted (#24517) [fix][client] Fix ClientCnx handleSendError NPE (#24515) [fix][ml] Fix asyncReadEntries might never complete if empty entries are read from BK (#24525) [improve][misc] Optimize topic list hashing so that potentially large String allocation is avoided (#24528) [fix][client] Fix issue in auto releasing of idle connection with topics pattern consumer (#24529) [fix][proxy] Fix default value of connectionMaxIdleSeconds in Pulsar Proxy (#24476) [fix][client] NPE in MultiTopicsConsumerImpl.negativeAcknowledge (#24465) [fix][proxy] Fix proxy OOM by replacing TopicName with a simple conversion method (#22495) [fix][test] Move ExtensibleLoadManagerImplTest to flaky tests (#21642) [fix][test] Fix flaky test SimpleProducerConsumerStatTest#testPartitionTopicStats (#24453) [fix][broker] replication does not work due to the mixed and repetitive sending of user messages and replication markers (#24424) [fix][broker] Fix the non-persistenttopic’s replicator always get error “Producer send queue is full” if set a small value of the config replicationProducerQueueSize (#24189) [fix][broker]excessive replication speed leads to error: Producer send queue is full (#22674) [Fix][broker] Limit replication rate based on bytes (#20931) [fix][broker] Fix ack hole in cursor for geo-replication (#24443) [fix][txn] Fix negative unacknowledged messages in transactions by ensuring that the batch size is added into CommandAck (#24421) [fix][build] Add missing<name> to submodules
(#24441) [fix][ml] Enhance OpFindNewest to support skip non-recoverable data
(#24459) [improve][broker] change to warn log level for ack validation error
(#24434) [improve][broker] Improve the log when namespace bundle is not available
(#24432) [fix][ml]Still got BK ledger, even though it has been deleted after offloaded
(#21467) [fix][test] Cleanup resources if starting PulsarService fails in PulsarTestContext
(#24351) [improve][broker] Deny removing local cluster from topic level replicated cluster policy
(#24419) [fix][broker] Once the cluster is configured incorrectly, the broker maintains the incorrect cluster configuration even if you removed it
(#24404) [fix][client] Prevent NPE when seeking with null topic in TopicMessageId
(#24405) [fix][ml]Received more than once callback when calling cursor.delete
(#24406) [fix][ml] Cursor ignores the position that has an empty ack-set if disabled deletionAtBatchIndexLevelEnabled
(#24401) [fix][txn] Fix deadlock when loading transaction buffer snapshot
(#24402) [fix][client] Fix some potential resource leak
(#20629) [improve][test] Fix flaky test SimpleProducerConsumerStatTest#testMsgRateExpired
(#21629) [fix][build] Fix potential insufficient protostuff-related configs
(#23594) [fix] [broker] No longer allow creating subscription that contains slash
(#24366) [fix][broker]Fix deadlock when compaction and topic deletion execute concurrently
(#24350) [fix][broker] Fix issue that topic policies was deleted after a sub topic deleted, even if the partitioned topic still exists
(#24384) [fix][ml]Revert a behavior change of releasing idle offloaded ledger handle: only release idle BlobStoreBackedReadHandle
(#24397) [improve][misc] Upgrade Netty to 4.1.122.Final and tcnative to 2.0.72.Final
(#24391) [improve][broker] Add managedCursor/LedgerInfoCompressionType settings to broker.conf
(#24392) [improve][broker] Make maxBatchDeletedIndexToPersist configurable and document other related configs
(#24386) [improve][broker] Added synchronized for sendMessages in Non-Persistent message dispatchers
(#24381) [improve][ml]Release idle offloaded read handle only the ref count is 0
(#19783) [improve][offloaders] Automatically evict Offloaded Ledgers from memory
(#24360) [fix][broker] expose consumer name for partitioned topic stats
(#24359) [improve][broker]Improve the log when encountered in-flight read limitation
(#24354) [fix][io] Acknowledge RabbitMQ message after processing the message successfully
(#24352) [fix][broker] Ignore metadata changes when broker is not in the Started state
(#24190) [fix][broker] Resolve the issue of frequent updates in message expiration deletion rate
(#24338) [fix][ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent issue
(#24331) [fix][offload] Complete the future outside of the reading loop in BlobStoreBackedReadHandleImplV2.readAsync
(#24324) [fix][test] Fix flaky AutoScaledReceiverQueueSizeTest.testNegativeClientMemory
(#24316) [fix][io] Fix kinesis avro bytes handling
(#24344) [improve][ml] Offload ledgers without check ledger length
(#24286) [fix][broker]Non-global topic policies and global topic policies overwrite each other
(#24279) [fix][broker]Global topic policies do not affect after unloading topic and persistence global topic policies never affect
(#24349) [fix][io][branch-3.0] Backport Kinesis Sink custom native executable support #23762
(#24317) [fix][io][branch-3.0]Pulsar-SQL: Fix classcast ex when decode decimal value
(#24313) [fix][broker] Fix potential deadlock when creating partitioned topic
(#24293) [fix][broker] fix wrong method name checkTopicExists.
(#24307) [fix][build] Ensure that buildtools is Java 8 compatible and fix remaining compatibility issue
(#24304) [fix][test] Simplify BetweenTestClassesListenerAdapter and fix issue with BeforeTest/AfterTest annotations
(#24289) [improve][io] Add configuration parameter for disabling aggregation for Kinesis Producers
(#24302) [improve] Upgrade pulsar-client-python to 3.7.0 in Docker image
(#24299) [fix][test] Fix more Netty ByteBuf leaks in tests
(#24297) [fix][io] Fix SyntaxWarning in Pulsar Python functions
(#24282) [fix][client] Fix producer publishing getting stuck after message with incompatible schema is discarded
(#24283) [cleanup][test] Remove unused parameter from deleteNamespaceWithRetry method in MockedPulsarServiceBaseTest
(#24263) [improve][build] Upgrade zstd version from 1.5.2-3 to 1.5.7-3
(#24281) [fix][test] Fix multiple ByteBuf leaks in tests
(#24275) [fix][broker] Fix HashedWheelTimer leak in PulsarService by stopping it in shutdown
(#24274) [fix][misc] Fix ByteBuf leak in SchemaUtils
(#24254) [fix][broker]Fix incorrect priority between topic policies and global topic policies
(#24266) [improve][ci] Disable detailed console logging for integration tests in CI
(#24261) [fix][test] Fix flaky ManagedCursorTest.testLastActiveAfterResetCursor and disable failing SchemaTest
(#24244) [fix][test] Fix flaky ManagedCursorTest.testSkipEntriesWithIndividualDeletedMessages
(#24248) [improve][io][kca] support fully-qualified topic names in source records
(#24260) [improve][build] Upgrade Gradle Develocity Maven Extension dependencies
(#24258) [fix][test] Fix TestNG BetweenTestClassesListenerAdapter listener
(#24257) [fix][broker] Unregister non-static metrics collectors registered in Prometheus default registry
(#24178) [fix][broker]fix memory leak, messages lost, incorrect replication state if using multiple schema versions(auto_produce)
(#24219) [improve][broker]Improve the feature “Optimize subscription seek (cursor reset) by timestamp”: search less entries
(#23919) [fix][broker] Fix seeking by timestamp can be reset the cursor position to earliest
(#22792) [improve][broker] Optimize subscription seek (cursor reset) by timestamp
(#24243) [improve][build] Upgrade SpotBugs to 4.9.x
(#24240) [improve][build] Upgrade to jacoco 0.8.13
(#24237) [improve][build] Upgrade Lombok to 1.18.38 to support JDK 24
(#24221) [improve][io] support kafka connect transforms and predicates
(#24230) [improve][client]Improve transaction log when a TXN command timeout
(#24223) [fix][broker] Orphan schema after disabled a cluster for a namespace
(#24228) [fix][broker] Fix ByteBuf memory leak in REST API for publishing messages
(#24184) [fix][client] Fix incorrect producer.getPendingQueueSize due to incomplete queue implementation
(#24214) [improve] Upgrade Netty to 4.1.121.Final
(#24212) [fix][test] Fix flaky BatchMessageWithBatchIndexLevelTest.testBatchMessageAck
(#24218) [fix][test] Fix multiple resource leaks in tests
(#24187) [improve][client] validate ClientConfigurationData earlier to avoid resource leaks
(#24216) [fix][broker] Fix HealthChecker deadlock in shutdown
(#24209) [fix][broker] Fix tenant creation and update with null value
(#24192) [fix][admin] Backlog quota’s policy is null which causes a NPE
(#24210) [fix][broker] Fix broker shutdown delay by resolving hanging health checks
(#24207) [fix][broker] Fix compaction service log’s wrong condition
(#24204) [fix][test] Fix resource leaks in ProxyTest and fix invalid tests
(#24201) [improve][io] Upgrade Kafka client and compatible Confluent platform version
(#24118))) Revert “[fix][broker] Add topic consistency check
(#24154))) Revert “[fix][broker] Directly query single topic existence when the topic is partitioned
(#24032) [fix][broker] Fix missing validation when setting retention policy on topic level
(#24098) [fix][ml] Skip deleting cursor if it was already deleted before calling unsubscribe
(#24181) [fix][proxy] Fix incorrect client error when calling get topic metadata
(#24158) [fix][proxy] Propagate client connection feature flags through Pulsar Proxy to Broker
(#24103) [fix][schema] Reject unsupported Avro schema types during schema registration
(#24091) [fix][broker] Fix some problems in calculate totalAvailableBookies in method getExcludedBookiesWithIsolationGroups when some bookies belongs to multiple isolation groups.
(#21320) [fix][bk] Fix the var name for IsolationGroups
(#24171) [improve][test] Use configured session timeout for MockZooKeeper and TestZKServer in PulsarTestContext
(#24172) [fix][test] Improve reliability of IncrementPartitionsTest
(#24170) [fix][test]flaky-test:ManagedLedgerInterceptorImplTest.testManagedLedgerPayloadInputProcessorFailure
(#23980) [fix][broker] Consumer stuck when delete subscription __compaction failed
(#24167) [fix][ml] Fix ML thread blocking issue in internalGetPartitionedStats API
(#24166) [fix][test] Fix invalid test CompactionTest.testDeleteCompactedLedgerWithSlowAck
(#24150) [fix][broker] The feature brokerDeleteInactivePartitionedTopicMetadataEnabled leaves orphan topic policies and topic schemas
(#24154) [fix][broker] Directly query single topic existence when the topic is partitioned
(#24118) [fix][broker] Add topic consistency check
(#24056) [fix][test] Update partitioned topic subscription assertions in IncrementPartitionsTest
(#24033) [cleanup][misc] Add override annotation
(#24161) [fix][test] Fix flaky BrokerServiceChaosTest.testFetchPartitionedTopicMetadataWithCacheRefresh
(#24162) [fix][test] Fix flaky BrokerServiceChaosTest
(#24097) [fix] [broker] topics infinitely failed to delete after remove cluster from replicated clusters modifying when using partitioned system topic
(#22261) [fix] Bump google.golang.org/protobuf from 1.32.0 to 1.33.0 in /pulsar-function-go
(#24132) [fix][io] Fix KinesisSink json flattening for AVRO’s SchemaType.BYTES
(#20984) [fix][broker] Fix get outdated compactedTopicContext after compactionHorizon has been updated
(#20697) [improve][broker] Improve CompactedTopicImpl lock
(#24131) [fix][ml] Return 1 when bytes size is 0 or negative for entry count estimation
(#24128) [improve][io] Enhance Kafka connector logging with focused bootstrap server information
(#24125) [fix][ml] Don’t estimate number of entries when ledgers are empty, return 1 instead
(#24123) [improve][client] Prevent NullPointException when closing ClientCredentialsFlow
(#24124) [improve][io] Remove sleep when sourceTask.poll of kafka return null
(#24116) [improve][broker] Change topic exists log to warn
(#24104) [fix][client] Pattern subscription regression when broker-side evaluation is disabled
(#24100) [fix][client] Fix consumer leak when thread is interrupted before subscribe completes
(#24089) [fix][ml] Fix issues in estimateEntryCountBySize
(#24073) [improve][broker] Optimize message expiration rate repeated update issues
(#24087) [fix][broker] Avoid IllegalStateException when marker_type field is not set in publishing
(#24083) [fix][ci] Bump dependency-check to 12.1.0 to fix OWASP Dependency Check job
(#24082) [clean][client] Clean code for the construction of retry/dead letter topic name
(#24079) [fix][broker] Fix NPE while publishing Metadata-Event with not init producer
(#24080) [fix][broker] Fix Metadata event synchronizer should not fail with bad version
(#24081) [fix][broker] Fix Metadata Event Synchronizer producer creation retry so that the producer gets created eventually
(#24048) [fix][broker] Fix UnsupportedOperationException while setting subscription level dispatch rate policy
(#24054) [fix][ml] Corrected pulsar_storage_size metric to not multiply offloaded storage by the write quorum
(#24067) [fix][broker] http metric endpoint get compaction latency stats always be 0
(#24064) [improve][broker] Optimize ThresholdShedder with improved boundary checks and parameter reuse
(#24055) [fix] Avoid negative estimated entry count
(#24060) [improve][monitor] Add version=0.0.4 to /metrics content type for Prometheus 3.x compatibility
(#24059) [fix][client] Copy eventTime to retry letter topic and DLQ messages
(#24061) [fix][client] Fix building broken batched message when publishing
(#24063) [fix][broker]Fix failed consumption after loaded up a terminated topic
(#24072) [fix][broker] Pattern subscription doesn’t work when the pattern excludes the topic domain.
(#24049) [improve] Upgrade Netty to 4.1.119.Final
(#23975) [fix][broker] Add expire check for replicator
(#24023) [fix][doc] fix doc related to chunk message feature.
(#23962) [improve][ml] Use lock-free queue in InflightReadsLimiter since there’s no concurrent access
(#23978) [improve][cli] Support additional msg metadata for V1 topic on peek message cmd
(#24014) [fix][broker] Fix BucketDelayedDeliveryTracker thread safety
(#24019) [fix][test]Fix flaky test V1_ProducerConsumerTest.testConcurrentConsumerReceiveWhileReconnect
(#24011) [fix][test] Fix flaky test OneWayReplicatorUsingGlobalZKTest.testConfigReplicationStartAt
(#23931) [improve] [broker] Make the estimated entry size more accurate
(#24004) [improve][ci] Upgrade Gradle Develocity Maven Extension to 1.23.1
(#23697) [fix][broker] Geo Replication lost messages or frequently fails due to Deduplication is not appropriate for Geo-Replication
(#24006) [fix][broker] fix broker identifying incorrect stuck topic
(#23286) [improve][broker] Fix non-persistent system topic schema compatibility
(#23881) [improve][fn] Set default tenant and namespace for ListFunctions cmd
(#23730) [fix][admin] Verify is policies read only before revoke permissions on topic
(#24003) [improve][test] Upgrade Testcontainers to 1.20.4 and docker-java to 3.4.0