V3.0
V3.0.10.7
StreamNative Weekly Release Notes v3.0.10.7
General Changes
Apache Pulsar
(#24552) [improve][test] Remove EntryCacheCreator from ManagedLedgerFactoryImpl (#24544) [improve] Upgrade pulsar-client-python to 3.8.0 in Docker image (#24516) [fix][broker] Fix exclusive producer creation when last shared producer closes (#24506) [fix][broker] Fix duplicate increment of ADD_OP_COUNT_UPDATER in OpAddEntry (#24543) [fix][broker] Fix matching of topicsPattern for topic names which contain non-ascii characters (#24539) [fix][client] Close orphan producer or consumer when the creation is interrupted (#24517) [fix][client] Fix ClientCnx handleSendError NPE (#24515) [fix][ml] Fix asyncReadEntries might never complete if empty entries are read from BK (#24525) [improve][misc] Optimize topic list hashing so that potentially large String allocation is avoided (#24528) [fix][client] Fix issue in auto releasing of idle connection with topics pattern consumer (#24529) [fix][proxy] Fix default value of connectionMaxIdleSeconds in Pulsar Proxy (#24476) [fix][client] NPE in MultiTopicsConsumerImpl.negativeAcknowledge (#24465) [fix][proxy] Fix proxy OOM by replacing TopicName with a simple conversion method (#22495) [fix][test] Move ExtensibleLoadManagerImplTest to flaky tests (#21642) [fix][test] Fix flaky test SimpleProducerConsumerStatTest#testPartitionTopicStats (#24453) [fix][broker] replication does not work due to the mixed and repetitive sending of user messages and replication markers (#24424) [fix][broker] Fix the non-persistenttopic’s replicator always get error “Producer send queue is full” if set a small value of the config replicationProducerQueueSize (#24189) [fix][broker]excessive replication speed leads to error: Producer send queue is full (#22674) [Fix][broker] Limit replication rate based on bytes (#20931) [fix][broker] Fix ack hole in cursor for geo-replication (#24443) [fix][txn] Fix negative unacknowledged messages in transactions by ensuring that the batch size is added into CommandAck (#24421) [fix][build] Add missing<name> to submodules
(#24441) [fix][ml] Enhance OpFindNewest to support skip non-recoverable data
(#24459) [improve][broker] change to warn log level for ack validation error
(#24434) [improve][broker] Improve the log when namespace bundle is not available
(#24432) [fix][ml]Still got BK ledger, even though it has been deleted after offloaded
(#21467) [fix][test] Cleanup resources if starting PulsarService fails in PulsarTestContext
(#24351) [improve][broker] Deny removing local cluster from topic level replicated cluster policy
(#24419) [fix][broker] Once the cluster is configured incorrectly, the broker maintains the incorrect cluster configuration even if you removed it
(#24404) [fix][client] Prevent NPE when seeking with null topic in TopicMessageId
(#24405) [fix][ml]Received more than once callback when calling cursor.delete
(#24406) [fix][ml] Cursor ignores the position that has an empty ack-set if disabled deletionAtBatchIndexLevelEnabled
(#24401) [fix][txn] Fix deadlock when loading transaction buffer snapshot
(#24402) [fix][client] Fix some potential resource leak
(#20629) [improve][test] Fix flaky test SimpleProducerConsumerStatTest#testMsgRateExpired
(#21629) [fix][build] Fix potential insufficient protostuff-related configs
(#23594) [fix] [broker] No longer allow creating subscription that contains slash
(#24366) [fix][broker]Fix deadlock when compaction and topic deletion execute concurrently
(#24350) [fix][broker] Fix issue that topic policies was deleted after a sub topic deleted, even if the partitioned topic still exists
(#24384) [fix][ml]Revert a behavior change of releasing idle offloaded ledger handle: only release idle BlobStoreBackedReadHandle
(#24397) [improve][misc] Upgrade Netty to 4.1.122.Final and tcnative to 2.0.72.Final
(#24391) [improve][broker] Add managedCursor/LedgerInfoCompressionType settings to broker.conf
(#24392) [improve][broker] Make maxBatchDeletedIndexToPersist configurable and document other related configs
(#24386) [improve][broker] Added synchronized for sendMessages in Non-Persistent message dispatchers
(#24381) [improve][ml]Release idle offloaded read handle only the ref count is 0
(#19783) [improve][offloaders] Automatically evict Offloaded Ledgers from memory
(#24360) [fix][broker] expose consumer name for partitioned topic stats
(#24359) [improve][broker]Improve the log when encountered in-flight read limitation
(#24354) [fix][io] Acknowledge RabbitMQ message after processing the message successfully
(#24352) [fix][broker] Ignore metadata changes when broker is not in the Started state
(#24190) [fix][broker] Resolve the issue of frequent updates in message expiration deletion rate
(#24338) [fix][ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent issue
(#24331) [fix][offload] Complete the future outside of the reading loop in BlobStoreBackedReadHandleImplV2.readAsync
(#24324) [fix][test] Fix flaky AutoScaledReceiverQueueSizeTest.testNegativeClientMemory
(#24316) [fix][io] Fix kinesis avro bytes handling
(#24344) [improve][ml] Offload ledgers without check ledger length
(#24286) [fix][broker]Non-global topic policies and global topic policies overwrite each other
(#24279) [fix][broker]Global topic policies do not affect after unloading topic and persistence global topic policies never affect
(#24349) [fix][io][branch-3.0] Backport Kinesis Sink custom native executable support #23762
(#24317) [fix][io][branch-3.0]Pulsar-SQL: Fix classcast ex when decode decimal value
(#24313) [fix][broker] Fix potential deadlock when creating partitioned topic
(#24293) [fix][broker] fix wrong method name checkTopicExists.
(#24307) [fix][build] Ensure that buildtools is Java 8 compatible and fix remaining compatibility issue
(#24304) [fix][test] Simplify BetweenTestClassesListenerAdapter and fix issue with BeforeTest/AfterTest annotations
(#24289) [improve][io] Add configuration parameter for disabling aggregation for Kinesis Producers
(#24302) [improve] Upgrade pulsar-client-python to 3.7.0 in Docker image
(#24299) [fix][test] Fix more Netty ByteBuf leaks in tests
(#24297) [fix][io] Fix SyntaxWarning in Pulsar Python functions
(#24282) [fix][client] Fix producer publishing getting stuck after message with incompatible schema is discarded
(#24283) [cleanup][test] Remove unused parameter from deleteNamespaceWithRetry method in MockedPulsarServiceBaseTest
(#24263) [improve][build] Upgrade zstd version from 1.5.2-3 to 1.5.7-3
(#24281) [fix][test] Fix multiple ByteBuf leaks in tests
(#24275) [fix][broker] Fix HashedWheelTimer leak in PulsarService by stopping it in shutdown
(#24274) [fix][misc] Fix ByteBuf leak in SchemaUtils
(#24254) [fix][broker]Fix incorrect priority between topic policies and global topic policies
(#24266) [improve][ci] Disable detailed console logging for integration tests in CI
(#24261) [fix][test] Fix flaky ManagedCursorTest.testLastActiveAfterResetCursor and disable failing SchemaTest
(#24244) [fix][test] Fix flaky ManagedCursorTest.testSkipEntriesWithIndividualDeletedMessages
(#24248) [improve][io][kca] support fully-qualified topic names in source records
(#24260) [improve][build] Upgrade Gradle Develocity Maven Extension dependencies
(#24258) [fix][test] Fix TestNG BetweenTestClassesListenerAdapter listener
(#24257) [fix][broker] Unregister non-static metrics collectors registered in Prometheus default registry
(#24178) [fix][broker]fix memory leak, messages lost, incorrect replication state if using multiple schema versions(auto_produce)
(#24219) [improve][broker]Improve the feature “Optimize subscription seek (cursor reset) by timestamp”: search less entries
(#23919) [fix][broker] Fix seeking by timestamp can be reset the cursor position to earliest
(#22792) [improve][broker] Optimize subscription seek (cursor reset) by timestamp
(#24243) [improve][build] Upgrade SpotBugs to 4.9.x
(#24240) [improve][build] Upgrade to jacoco 0.8.13
(#24237) [improve][build] Upgrade Lombok to 1.18.38 to support JDK 24
(#24221) [improve][io] support kafka connect transforms and predicates
(#24230) [improve][client]Improve transaction log when a TXN command timeout
(#24223) [fix][broker] Orphan schema after disabled a cluster for a namespace
(#24228) [fix][broker] Fix ByteBuf memory leak in REST API for publishing messages
(#24184) [fix][client] Fix incorrect producer.getPendingQueueSize due to incomplete queue implementation
(#24214) [improve] Upgrade Netty to 4.1.121.Final
(#24212) [fix][test] Fix flaky BatchMessageWithBatchIndexLevelTest.testBatchMessageAck
(#24218) [fix][test] Fix multiple resource leaks in tests
(#24187) [improve][client] validate ClientConfigurationData earlier to avoid resource leaks
(#24216) [fix][broker] Fix HealthChecker deadlock in shutdown
(#24209) [fix][broker] Fix tenant creation and update with null value
(#24192) [fix][admin] Backlog quota’s policy is null which causes a NPE
(#24210) [fix][broker] Fix broker shutdown delay by resolving hanging health checks
(#24207) [fix][broker] Fix compaction service log’s wrong condition
(#24204) [fix][test] Fix resource leaks in ProxyTest and fix invalid tests
(#24201) [improve][io] Upgrade Kafka client and compatible Confluent platform version
(#24118))) Revert “[fix][broker] Add topic consistency check
(#24154))) Revert “[fix][broker] Directly query single topic existence when the topic is partitioned
(#24032) [fix][broker] Fix missing validation when setting retention policy on topic level
(#24098) [fix][ml] Skip deleting cursor if it was already deleted before calling unsubscribe
(#24181) [fix][proxy] Fix incorrect client error when calling get topic metadata
(#24158) [fix][proxy] Propagate client connection feature flags through Pulsar Proxy to Broker
(#24103) [fix][schema] Reject unsupported Avro schema types during schema registration
(#24091) [fix][broker] Fix some problems in calculate totalAvailableBookies in method getExcludedBookiesWithIsolationGroups when some bookies belongs to multiple isolation groups.
(#21320) [fix][bk] Fix the var name for IsolationGroups
(#24171) [improve][test] Use configured session timeout for MockZooKeeper and TestZKServer in PulsarTestContext
(#24172) [fix][test] Improve reliability of IncrementPartitionsTest
(#24170) [fix][test]flaky-test:ManagedLedgerInterceptorImplTest.testManagedLedgerPayloadInputProcessorFailure
(#23980) [fix][broker] Consumer stuck when delete subscription __compaction failed
(#24167) [fix][ml] Fix ML thread blocking issue in internalGetPartitionedStats API
(#24166) [fix][test] Fix invalid test CompactionTest.testDeleteCompactedLedgerWithSlowAck
(#24150) [fix][broker] The feature brokerDeleteInactivePartitionedTopicMetadataEnabled leaves orphan topic policies and topic schemas
(#24154) [fix][broker] Directly query single topic existence when the topic is partitioned
(#24118) [fix][broker] Add topic consistency check
(#24056) [fix][test] Update partitioned topic subscription assertions in IncrementPartitionsTest
(#24033) [cleanup][misc] Add override annotation
(#24161) [fix][test] Fix flaky BrokerServiceChaosTest.testFetchPartitionedTopicMetadataWithCacheRefresh
(#24162) [fix][test] Fix flaky BrokerServiceChaosTest
(#24097) [fix] [broker] topics infinitely failed to delete after remove cluster from replicated clusters modifying when using partitioned system topic
(#22261) [fix] Bump google.golang.org/protobuf from 1.32.0 to 1.33.0 in /pulsar-function-go
(#24132) [fix][io] Fix KinesisSink json flattening for AVRO’s SchemaType.BYTES
(#20984) [fix][broker] Fix get outdated compactedTopicContext after compactionHorizon has been updated
(#20697) [improve][broker] Improve CompactedTopicImpl lock
(#24131) [fix][ml] Return 1 when bytes size is 0 or negative for entry count estimation
(#24128) [improve][io] Enhance Kafka connector logging with focused bootstrap server information
(#24125) [fix][ml] Don’t estimate number of entries when ledgers are empty, return 1 instead
(#24123) [improve][client] Prevent NullPointException when closing ClientCredentialsFlow
(#24124) [improve][io] Remove sleep when sourceTask.poll of kafka return null
(#24116) [improve][broker] Change topic exists log to warn
(#24104) [fix][client] Pattern subscription regression when broker-side evaluation is disabled
(#24100) [fix][client] Fix consumer leak when thread is interrupted before subscribe completes
(#24089) [fix][ml] Fix issues in estimateEntryCountBySize
(#24073) [improve][broker] Optimize message expiration rate repeated update issues
(#24087) [fix][broker] Avoid IllegalStateException when marker_type field is not set in publishing
(#24083) [fix][ci] Bump dependency-check to 12.1.0 to fix OWASP Dependency Check job
(#24082) [clean][client] Clean code for the construction of retry/dead letter topic name
(#24079) [fix][broker] Fix NPE while publishing Metadata-Event with not init producer
(#24080) [fix][broker] Fix Metadata event synchronizer should not fail with bad version
(#24081) [fix][broker] Fix Metadata Event Synchronizer producer creation retry so that the producer gets created eventually
(#24048) [fix][broker] Fix UnsupportedOperationException while setting subscription level dispatch rate policy
(#24054) [fix][ml] Corrected pulsar_storage_size metric to not multiply offloaded storage by the write quorum
(#24067) [fix][broker] http metric endpoint get compaction latency stats always be 0
(#24064) [improve][broker] Optimize ThresholdShedder with improved boundary checks and parameter reuse
(#24055) [fix] Avoid negative estimated entry count
(#24060) [improve][monitor] Add version=0.0.4 to /metrics content type for Prometheus 3.x compatibility
(#24059) [fix][client] Copy eventTime to retry letter topic and DLQ messages
(#24061) [fix][client] Fix building broken batched message when publishing
(#24063) [fix][broker]Fix failed consumption after loaded up a terminated topic
(#24072) [fix][broker] Pattern subscription doesn’t work when the pattern excludes the topic domain.
(#24049) [improve] Upgrade Netty to 4.1.119.Final
(#23975) [fix][broker] Add expire check for replicator
(#24023) [fix][doc] fix doc related to chunk message feature.
(#23962) [improve][ml] Use lock-free queue in InflightReadsLimiter since there’s no concurrent access
(#23978) [improve][cli] Support additional msg metadata for V1 topic on peek message cmd
(#24014) [fix][broker] Fix BucketDelayedDeliveryTracker thread safety
(#24019) [fix][test]Fix flaky test V1_ProducerConsumerTest.testConcurrentConsumerReceiveWhileReconnect
(#24011) [fix][test] Fix flaky test OneWayReplicatorUsingGlobalZKTest.testConfigReplicationStartAt
(#23931) [improve] [broker] Make the estimated entry size more accurate
(#24004) [improve][ci] Upgrade Gradle Develocity Maven Extension to 1.23.1
(#23697) [fix][broker] Geo Replication lost messages or frequently fails due to Deduplication is not appropriate for Geo-Replication
(#24006) [fix][broker] fix broker identifying incorrect stuck topic
(#23286) [improve][broker] Fix non-persistent system topic schema compatibility
(#23881) [improve][fn] Set default tenant and namespace for ListFunctions cmd
(#23730) [fix][admin] Verify is policies read only before revoke permissions on topic
(#24003) [improve][test] Upgrade Testcontainers to 1.20.4 and docker-java to 3.4.0