Skip to main contentStreamNative Weekly Release Notes v3.0.10.6
General Changes
Apache Pulsar
(#23594) [fix] [broker] No longer allow creating subscription that contains slash
232f0ef492 remove unnecessary codes
(#24366) [fix][broker]Fix deadlock when compaction and topic deletion execute concurrently
(#24350) [fix][broker] Fix issue that topic policies was deleted after a sub topic deleted, even if the partitioned topic still exists
(#24384) [fix][ml]Revert a behavior change of releasing idle offloaded ledger handle: only release idle BlobStoreBackedReadHandle
(#24397) [improve][misc] Upgrade Netty to 4.1.122.Final and tcnative to 2.0.72.Final
(#24391) [improve][broker] Add managedCursor/LedgerInfoCompressionType settings to broker.conf
(#24392) [improve][broker] Make maxBatchDeletedIndexToPersist configurable and document other related configs
(#24386) [improve][broker] Added synchronized for sendMessages in Non-Persistent message dispatchers
(#24381) [improve][ml]Release idle offloaded read handle only the ref count is 0
(#19783) [improve][offloaders] Automatically evict Offloaded Ledgers from memory
(#24360) [fix][broker] expose consumer name for partitioned topic stats
(#24359) [improve][broker]Improve the log when encountered in-flight read limitation
(#24354) [fix][io] Acknowledge RabbitMQ message after processing the message successfully
(#24352) [fix][broker] Ignore metadata changes when broker is not in the Started state
(#24190) [fix][broker] Resolve the issue of frequent updates in message expiration deletion rate
(#24338) [fix][ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent issue
(#24331) [fix][offload] Complete the future outside of the reading loop in BlobStoreBackedReadHandleImplV2.readAsync
(#24324) [fix][test] Fix flaky AutoScaledReceiverQueueSizeTest.testNegativeClientMemory
(#24316) [fix][io] Fix kinesis avro bytes handling
a75d16fa6e Fix checkstyle errors after previous cherry-picks
(#24344) [improve][ml] Offload ledgers without check ledger length
(#24286) [fix][broker]Non-global topic policies and global topic policies overwrite each other
(#24279) [fix][broker]Global topic policies do not affect after unloading topic and persistence global topic policies never affect
(#24349) [fix][io][branch-3.0] Backport Kinesis Sink custom native executable support #23762
(#24317) [fix][io][branch-3.0]Pulsar-SQL: Fix classcast ex when decode decimal value
(#24313) [fix][broker] Fix potential deadlock when creating partitioned topic
(#24293) [fix][broker] fix wrong method name checkTopicExists.
(#24307) [fix][build] Ensure that buildtools is Java 8 compatible and fix remaining compatibility issue
(#24304) [fix][test] Simplify BetweenTestClassesListenerAdapter and fix issue with BeforeTest/AfterTest annotations
(#24289) [improve][io] Add configuration parameter for disabling aggregation for Kinesis Producers
(#24302) [improve] Upgrade pulsar-client-python to 3.7.0 in Docker image
(#24299) [fix][test] Fix more Netty ByteBuf leaks in tests
(#24297) [fix][io] Fix SyntaxWarning in Pulsar Python functions
(#24282) [fix][client] Fix producer publishing getting stuck after message with incompatible schema is discarded
(#24283) [cleanup][test] Remove unused parameter from deleteNamespaceWithRetry method in MockedPulsarServiceBaseTest
(#24263) [improve][build] Upgrade zstd version from 1.5.2-3 to 1.5.7-3
(#24281) [fix][test] Fix multiple ByteBuf leaks in tests
(#24275) [fix][broker] Fix HashedWheelTimer leak in PulsarService by stopping it in shutdown
(#24274) [fix][misc] Fix ByteBuf leak in SchemaUtils
(#24254) [fix][broker]Fix incorrect priority between topic policies and global topic policies
(#24266) [improve][ci] Disable detailed console logging for integration tests in CI
(#24261) [fix][test] Fix flaky ManagedCursorTest.testLastActiveAfterResetCursor and disable failing SchemaTest
(#24244) [fix][test] Fix flaky ManagedCursorTest.testSkipEntriesWithIndividualDeletedMessages
(#24248) [improve][io][kca] support fully-qualified topic names in source records
(#24260) [improve][build] Upgrade Gradle Develocity Maven Extension dependencies
(#24258) [fix][test] Fix TestNG BetweenTestClassesListenerAdapter listener
(#24257) [fix][broker] Unregister non-static metrics collectors registered in Prometheus default registry
bebc3b0d6a Fix checkstyle issue in previous cherry-pick c2d33cc
(#24178) [fix][broker]fix memory leak, messages lost, incorrect replication state if using multiple schema versions(auto_produce)
(#24219) [improve][broker]Improve the feature “Optimize subscription seek (cursor reset) by timestamp”: search less entries
(#23919) [fix][broker] Fix seeking by timestamp can be reset the cursor position to earliest
(#22792) [improve][broker] Optimize subscription seek (cursor reset) by timestamp
(#24243) [improve][build] Upgrade SpotBugs to 4.9.x
(#24240) [improve][build] Upgrade to jacoco 0.8.13
(#24237) [improve][build] Upgrade Lombok to 1.18.38 to support JDK 24
(#24221) [improve][io] support kafka connect transforms and predicates
(#24230) [improve][client]Improve transaction log when a TXN command timeout
(#24223) [fix][broker] Orphan schema after disabled a cluster for a namespace
(#24228) [fix][broker] Fix ByteBuf memory leak in REST API for publishing messages
b4e1c93d2c Fix presto-distribution/LICENSE
(#24184) [fix][client] Fix incorrect producer.getPendingQueueSize due to incomplete queue implementation
(#24214) [improve] Upgrade Netty to 4.1.121.Final
(#24212) [fix][test] Fix flaky BatchMessageWithBatchIndexLevelTest.testBatchMessageAck
(#24218) [fix][test] Fix multiple resource leaks in tests
(#24187) [improve][client] validate ClientConfigurationData earlier to avoid resource leaks
(#24216) [fix][broker] Fix HealthChecker deadlock in shutdown
(#24209) [fix][broker] Fix tenant creation and update with null value
(#24192) [fix][admin] Backlog quota’s policy is null which causes a NPE
(#24210) [fix][broker] Fix broker shutdown delay by resolving hanging health checks
(#24207) [fix][broker] Fix compaction service log’s wrong condition
(#24204) [fix][test] Fix resource leaks in ProxyTest and fix invalid tests
(#24201) [improve][io] Upgrade Kafka client and compatible Confluent platform version
(#24118))) Revert “[fix][broker] Add topic consistency check
(#24154))) Revert “[fix][broker] Directly query single topic existence when the topic is partitioned
(#24032) [fix][broker] Fix missing validation when setting retention policy on topic level
(#24098) [fix][ml] Skip deleting cursor if it was already deleted before calling unsubscribe
(#24181) [fix][proxy] Fix incorrect client error when calling get topic metadata
(#24158) [fix][proxy] Propagate client connection feature flags through Pulsar Proxy to Broker
(#24103) [fix][schema] Reject unsupported Avro schema types during schema registration
(#24091) [fix][broker] Fix some problems in calculate totalAvailableBookies in method getExcludedBookiesWithIsolationGroups when some bookies belongs to multiple isolation groups.
(#21320) [fix][bk] Fix the var name for IsolationGroups
(#24171) [improve][test] Use configured session timeout for MockZooKeeper and TestZKServer in PulsarTestContext
(#24172) [fix][test] Improve reliability of IncrementPartitionsTest
(#24170) [fix][test]flaky-test:ManagedLedgerInterceptorImplTest.testManagedLedgerPayloadInputProcessorFailure
(#23980) [fix][broker] Consumer stuck when delete subscription __compaction failed
(#24167) [fix][ml] Fix ML thread blocking issue in internalGetPartitionedStats API
(#24166) [fix][test] Fix invalid test CompactionTest.testDeleteCompactedLedgerWithSlowAck
(#24150) [fix][broker] The feature brokerDeleteInactivePartitionedTopicMetadataEnabled leaves orphan topic policies and topic schemas
(#24154) [fix][broker] Directly query single topic existence when the topic is partitioned
(#24118) [fix][broker] Add topic consistency check
(#24056) [fix][test] Update partitioned topic subscription assertions in IncrementPartitionsTest
(#24033) [cleanup][misc] Add override annotation
(#24161) [fix][test] Fix flaky BrokerServiceChaosTest.testFetchPartitionedTopicMetadataWithCacheRefresh
(#24162) [fix][test] Fix flaky BrokerServiceChaosTest
1035accffd Bump version to next snapshot version
(#24097) [fix] [broker] topics infinitely failed to delete after remove cluster from replicated clusters modifying when using partitioned system topic
(#22261) [fix] Bump google.golang.org/protobuf from 1.32.0 to 1.33.0 in /pulsar-function-go
(#24132) [fix][io] Fix KinesisSink json flattening for AVRO’s SchemaType.BYTES
(#20984) [fix][broker] Fix get outdated compactedTopicContext after compactionHorizon has been updated
(#20697) [improve][broker] Improve CompactedTopicImpl lock
(#24131) [fix][ml] Return 1 when bytes size is 0 or negative for entry count estimation
(#24128) [improve][io] Enhance Kafka connector logging with focused bootstrap server information
(#24125) [fix][ml] Don’t estimate number of entries when ledgers are empty, return 1 instead
(#24123) [improve][client] Prevent NullPointException when closing ClientCredentialsFlow
(#24124) [improve][io] Remove sleep when sourceTask.poll of kafka return null
(#24116) [improve][broker] Change topic exists log to warn
(#24104) [fix][client] Pattern subscription regression when broker-side evaluation is disabled
(#24100) [fix][client] Fix consumer leak when thread is interrupted before subscribe completes
(#24089) [fix][ml] Fix issues in estimateEntryCountBySize
(#24073) [improve][broker] Optimize message expiration rate repeated update issues
(#24087) [fix][broker] Avoid IllegalStateException when marker_type field is not set in publishing
(#24083) [fix][ci] Bump dependency-check to 12.1.0 to fix OWASP Dependency Check job
(#24082) [clean][client] Clean code for the construction of retry/dead letter topic name
(#24079) [fix][broker] Fix NPE while publishing Metadata-Event with not init producer
(#24080) [fix][broker] Fix Metadata event synchronizer should not fail with bad version
(#24081) [fix][broker] Fix Metadata Event Synchronizer producer creation retry so that the producer gets created eventually
(#24048) [fix][broker] Fix UnsupportedOperationException while setting subscription level dispatch rate policy
(#24054) [fix][ml] Corrected pulsar_storage_size metric to not multiply offloaded storage by the write quorum
(#24067) [fix][broker] http metric endpoint get compaction latency stats always be 0
(#24064) [improve][broker] Optimize ThresholdShedder with improved boundary checks and parameter reuse
(#24055) [fix] Avoid negative estimated entry count
(#24060) [improve][monitor] Add version=0.0.4 to /metrics content type for Prometheus 3.x compatibility
(#24059) [fix][client] Copy eventTime to retry letter topic and DLQ messages
(#24061) [fix][client] Fix building broken batched message when publishing
(#24063) [fix][broker]Fix failed consumption after loaded up a terminated topic
(#24072) [fix][broker] Pattern subscription doesn’t work when the pattern excludes the topic domain.
ebce3b07ed Fix presto LICENSE after Netty 4.1.119.Final upgrade
(#24049) [improve] Upgrade Netty to 4.1.119.Final
(#23975) [fix][broker] Add expire check for replicator
(#24023) [fix][doc] fix doc related to chunk message feature.
8437af98eb Bump version to next snapshot version
(#23962) [improve][ml] Use lock-free queue in InflightReadsLimiter since there’s no concurrent access
(#23978) [improve][cli] Support additional msg metadata for V1 topic on peek message cmd
(#24014) [fix][broker] Fix BucketDelayedDeliveryTracker thread safety
(#24019) [fix][test]Fix flaky test V1_ProducerConsumerTest.testConcurrentConsumerReceiveWhileReconnect
(#24011) [fix][test] Fix flaky test OneWayReplicatorUsingGlobalZKTest.testConfigReplicationStartAt
(#23931) [improve] [broker] Make the estimated entry size more accurate
(#24004) [improve][ci] Upgrade Gradle Develocity Maven Extension to 1.23.1
(#23697) [fix][broker] Geo Replication lost messages or frequently fails due to Deduplication is not appropriate for Geo-Replication
(#24006) [fix][broker] fix broker identifying incorrect stuck topic
(#23286) [improve][broker] Fix non-persistent system topic schema compatibility
(#23881) [improve][fn] Set default tenant and namespace for ListFunctions cmd
(#23730) [fix][admin] Verify is policies read only before revoke permissions on topic
(#24003) [improve][test] Upgrade Testcontainers to 1.20.4 and docker-java to 3.4.0
KoP
[refactor] Create PartitionLog only after ProducerStateManager#recover is done
StreamNative Pulsar Plugins
Add test to verify the sts module
Fix the backup tool can not use sts to authenticate
pulsarctl
e7d5e82 Use snstage docker image
(#1778) Fix jwt cve
(#1711) fix code check
(#1704) feat: Subscription get message by id json output
(#1699) Update subscription get message by id typo lederId to ledgerId
(#1585) fix: upgrade golang version to fix CVE
(#1587) Setup go version to 1.22 fix cve
(#1549) update pulsar-client-go to master latest commit
2af1258 fix ci
(#1537) Bump the pulsar-client-go to the master version
6f25051 Fix TestDeleteNonExistPartitionedTopic
(#1509) Fix json marshal error for Secrets and UserConfigs when creating/updating functions
(#1455) Support create token with headers
(#1478) Upgrade the dependency version to fix vulnerabilities
(#1447) Add trivy scan workflow to avoid vulnerabilities
(#1451) [fix] Upgrade go version to 1.21 to fix CVE-2023-24538
(#1362) fix source test typo
(#1360) fix source test
(#1419) Auth SN docker hub
(#1405) Support no auth context
(#1393) fix token
(#1402) Add docker hub login
(#1398) Auth SN docker hub
5cb0593 Disable bk unit test and fix it later ---
(#1257) Add method to mark bookie readonly
(#1328) Build arm64 linux executable binary artifact
(#1305) Update jose2go to fix GHSA-mhpq-9638-x6pw
(#1268) Update golang.org/x/net
(#1205) Replace apache pulsar client go repo on 3.0 branch
(#1084) Fixed remove auth plugin suffix
(#1072) Removed error char
(#1065) Bump pulsar version to 3.0.0.1
(#1067) fix: Fix TestUpdateTopicNotExist and TestUpdateNonPartitionedTopic
Function Mesh Worker Service
Support invalid name 
